Plenary template adjustment to fix the template library support #87

jouvin · 2018-05-23T20:01:52Z

For host, cluster and metacluster plenary templates, include a preface template, called archetype/declarations.pan, intended mainly to define the loadpath for the rest of the profile. This happens only if the broker config option object_declarations_template is true (false by default).

Broker option include_pan is also removed (addition reverted), as it is superseded by the previous change

This PR is critical for enable the use of the template library in Aquilon

Fixes #86

jouvin · 2018-05-24T12:50:24Z

@urbonegi I just updated the PR based on our discussion. I added another change I had in mind (I mentioned it during our discussion but it was may not clear) to allow to define that a variable is equal to another variable and I used this to define LOADPATH so that it is more clear that the initial value is the archetype.

Shaeli · 2018-05-24T16:24:35Z

Hi, just noticed that all the commits in this PR have different change-id. Can you please change this to have the same change id? Thanks

jouvin · 2018-05-24T16:38:24Z

@Shaeli Done! Sorry it was not clear for me if there should be one Change-Id for all the commits in the PR or one per commit...

jrha · 2018-05-24T16:40:08Z

Please ignore the test failures, these are an artifact of the move from travisci.org to travisci.com

jouvin · 2018-05-24T17:57:30Z

@Shaeli I just fixed a typo in the second commit message... (missing s)

jrha · 2018-05-24T22:57:29Z

I have just re-enabled branch protection for upstream and master, please let me know if it causes any problems pushing to the repo.

urbonegi · 2018-05-25T14:52:25Z

lib/aquilon/aqdb/model/building.py

@@ -34,8 +34,4 @@ class Building(Location):

    address = Column(String(255), nullable=False)

-    next_rackid = Column(Integer, nullable=False, default=2)


Added by accident? Probably you need to rebase on latest upstream.

This is a little bit strange, I thought I just reverted the previous commit which had something related to building.py. I started from the latest upstream if I'm right (latest=1 week ago!). May be the easiest is that I remove this line (or building.py) from the revert commit if this is the only change?

jouvin · 2018-05-25T15:12:38Z

@urbonegi @Shaeli Revert commit amended to remove building.py from it (was really part of the original commit reverted...).

urbonegi · 2018-05-25T18:28:46Z

@jouvin so we do not understand fully why the archetype/base does not indeed become the first one if one would skip loading the pan/unis and functions? Please see my comment here:#86

Another this is that, I advised @Shaeli that we do need same 'change-id' for all the commits in the PR - while it is not true. I got confused sorry. The change-id needs to be different for each commit. It is there so that gerrit can cope with rebase, when commit-id will change - change-id will not :) sorry for confusion!

jouvin · 2018-05-25T23:22:13Z

@urbonegi I hope that #86 (comment) answers your question! It seems that MS has some concerns (that I don't really understand frankly) about these changes, because they are not useful for MS who does not rely on the template library. To move forward with this PR asasp, as it is critical for allowing the use of Aquilon with the template library without hacking Aquilon, I propose to add a new config option, template_library_configured, that will be false by default and true in conf.noms. If this option is false, the generated profile will be unchanged, If it is true, the 2 changes implemented by this PR will be present in the profile. Would it help to have MS agreeing on this PR and get it through quickly?

About Gerrit Change-ID, don't worry. This is easy to generate again a separate one for each commit!

jouvin · 2018-05-26T07:26:55Z

@urbonegi I implemented what I suggested in my previous comment, with even more elementary commits (support for variable definition based on a variable and addition of the new variable ARCHETYPE splitted in 2 commits).
New host.py successfully tested at LAL (we don't have cluster or metacluster objects but the change is trivial enough that I'm confident it is good!).

jouvin · 2018-05-27T17:15:29Z

After the very long and intense discussion with @ned21 in #86 (😄), I remove from this PR the 2 commits not related to the template library support but to the addition of an ARCHETYPE variable. In the last version, the archetype/loadpath template must exist if the template_library_configured option is true, as per @ned21 suggestion.
2 minor details need to be agreed:

Template name: @ned21 would prefer declaration.pan rather than loadpath.pan to make it clear that it should be a declaration template, as this is not possible to impose it in the object template. I tend to prefer a name that reflects the content expected and let to the documentation the task to explain that it should be a declaration template rather than a standard template (that would work anyway).
Config option name: the idea was that this template is really required when you want to rely on templates that are imported from somewhere else and cannot be found in default include paths (plenary root, template-king repo...). This is typically the case with the template library (see https://github.com/quattor/release/tree/master/src/scripts/plenary_template_library), thus the name. The idea was to have an option that can be used for any object template variant that may be required by the template library use, if any other...

jrha · 2018-05-29T12:28:24Z

For the record, I have no strong preferences for the name, as long as it is clear what it is for.

Ours currently looks like this:

declaration template archetype/preface;

final variable ARCHETYPE = "ral-tier1";

final variable QUATTOR_RELEASE = '17.12.0';

variable LOADPATH = prepend(SELF, format('template-library/%s/core', QUATTOR_RELEASE));
variable LOADPATH = prepend(SELF, format('template-library/%s/standard', QUATTOR_RELEASE));
variable LOADPATH = prepend(SELF, 'ral-tier1/core');
variable LOADPATH = prepend(SELF, 'ral-tier1/standard');

urbonegi · 2018-05-29T15:16:10Z

etc/aqd.conf.defaults

@@ -345,8 +345,6 @@ gzip_output = false
 # Assume the webserver will decompress transparently as needed.
 # only used if gzip_output = true
 transparent_gzip = true
-template_extension = .tpl


why this is here?

A mistake of mine! Thanks for noticing it.

urbonegi · 2018-05-29T15:16:22Z

Ok, this been on-going for some time now. We are happy to merge this one nearly as it is. We suggest to have code like this in hosts, clusters and metaclusters before the base/archetype temaple:

# Allow settings such as loadpath to be modified by the archetype before anything else happens
# Included only if template_library_configured option is true
# This template is optional: included only if it exists
if self.config.getboolean("panc", "include_declarations_template"):
    # Also contain other declarations such as pan/units and pan/functions
    pan_include(lines, "archetype/declarations")
    lines.append("")
else:
    pan_include(lines, ["pan/units", "pan/functions"])
    lines.append("")
# Okay, here's the real content

This new 'archetype/declarations' template should accommodate @jrha suggestion above + could contain pan/units and pan/functions. MS later could also move to it too, but for now it could be controlled with an option.

jouvin · 2018-05-29T18:19:55Z

@urbonegi I think @ned21 mentioned that but also agreed later that it should be discussed separately. I don't see any reason to couple them and at this stage, I am not convinced this is a good move as it requires the user to do more when for me, the objective is to have the user doing only the things that the broker cannot do in a sensible manner (or without too much complexitiy).

jouvin · 2018-05-29T18:24:08Z

I updated the PR taking into account most of @urbonegi remarks. 2 points still need to be agreed:

The preface template name. As said by @jrha, I think it must reflects what the template is intended for. For me, declaration doesn't qualify as it is more about a type of template than the purpose of the template. I agree with @ned21 that preface is too vague. And I'm convinced we should insist that this template should not be used for anything else than adjusting the LOADPATH so that the templates until base can be found. Presently, I kept loadpath as the name but this is easy to change when we come to an agreement.
The aqd.conf option name. I went for object_preface_template to reflect the option purpose, with the idea (opposite to my initial one) that it should be used only for this purpose.

ned21 · 2018-05-29T20:13:34Z

re: The name My view is that we cannot stop people making changes other than LOADPATH manipulations so we should not name it something that me become laughable false in a few years as people find other clever uses for it. Hence "declarations" is exactly what it says it is: you can use it for the same things that PAN defines as as declaration, anything else and your template will be invalid. And that's enforced in the compiler.

The config option name: object_preface_template: that's a good name, although we might rename preface to whatever we finally decide to call it.

Removal of pan/units,functions: I suggested this because those lines are now superfluous and can be more accurately managed per-archetype in the preface template. In particular for us it means that for those those archetypes where we do not use the definitions in those files, we can remove the lines altogether which we regard as a nicely improvement. It also makes the object template much cleaner by removing a hardcoded dependency on files from the TL that are not relevant to many of our archetypes.

jouvin · 2018-05-29T20:41:49Z

@ned21 we are not in strong disagreement on the reasoning, despite we have not yet converged on the conclusion 😄

template name: I agree that we cannot stop people from using it for something else but I think we should do our best to avoid this template to become another base. Thus my proposal to have a name reflecting this. If my analysis is wrong that there is nothing else that is site/archetype dependent and cannot wait base, I agree that this could become a problem in the future. I missed the s in your proposal that makes it a little bit better than declaration. But I also recognize that the exact usage of this template is influenced by your other proposal about pan/... templates.
Config option name: I'm fine with updating my last proposal with the actual name choosen, despite I chose this name because I thought it was abstract enough to be usable with whatever name we decide. It is an easy change to do, I'm happy to update the PR if others prefer.
Removal of pan/... templates: I understand that you don't want the object template to include things that are not useful for any object... But as soon as you use Pan, you probably need at least pan/functions which defines push function for example. These particular templates were designed to be the kind of thing you may need as long as you want to use Pan, even for something else than a host profile. Anyway, I accept that this is discussable. My main point here is that we should really make an effort to stick to what you requested at some point: 1 PR = 1 goal. This is like the archetype discussion (1.12.58: the broker should pass the archetype to the object template #88): it was a mistake of mine to have started both things in the same PR. I don't see an urgency to decide on this (where there is one for the preface template as without it we cannot expect any TL site to adopt Aquilon) and my proposal was to handle it separately. I recognize that my loadpath name proposal is a little bit a conclusion for this point and I'm ready to accept declarations to ensure that we don't mix 2 different issues here and concentrate first on all the issues that are showstoppers for new sites (or may require to document hacks in the documentation which is very bad IMO).

urbonegi · 2018-05-30T12:58:57Z

@ned21 if you agree, i think we could go one step at the time. For now just allow to add the extra initial template if newly defined config option is set to true. Then later we might move the functions and units there. However, calling that new template 'archetype/loadpath' is to explicit. It locks the usage of this template for one usecase - define the loadpath var. However, lets not limit ourselves like that. Please change the name to 'archetype/declarations' or 'archetype/preface' as suggested above. the config option then should be 'object_declarations_template' or 'object_preface_template'. I vote for 'declarations'. @jouvin there are 2 small other issues in the PR -will add comments. Hopefully we will have an agreement with this - it been ongoing for too long ;)

urbonegi · 2018-05-30T12:59:51Z

etc/aqd.conf.defaults

@@ -345,6 +345,9 @@ gzip_output = false
 # Assume the webserver will decompress transparently as needed.
 # only used if gzip_output = true
 transparent_gzip = true
+template_extension = .tpl


it would make sense to fix the 'Revert' commit to not delete this rather than re-add it in here.

You mean that this line disappeared because of the revert commit? This is stange... That means that the original commit did some bad things! I'll double check and fix the revert commit in this case...

urbonegi · 2018-05-30T13:00:18Z

lib/aquilon/worker/templates/cluster.py

+        # Included only if template_library_configured option is true
+        # This template is optional: included only if it exists
+        if self.config.getboolean("panc", "object_preface_template"):
+            pan_include(lines, "archetype/loadpath")


I think you missed: lines.append("") after pan_include

I thought you asked me to delete it in a previous comment.. I'll readd it!

urbonegi · 2018-05-30T13:00:32Z

lib/aquilon/worker/templates/host.py

+        # Included only if template_library_configured option is true
+        # This template is optional: included only if it exists
+        if self.config.getboolean("panc", "object_preface_template"):
+            pan_include(lines, "archetype/loadpath")


I think you missed: lines.append("") after pan_include

urbonegi · 2018-05-30T13:00:39Z

lib/aquilon/worker/templates/metacluster.py

+        # Included only if template_library_configured option is true
+        # This template is optional: included only if it exists
+        if self.config.getboolean("panc", "object_preface_template"):
+            pan_include(lines, "archetype/loadpath")


I think you missed: lines.append("") after pan_include

- Except lib/aquilon/aqdb/model/building.py that should not have been part of the initial commit This reverts commit 1ada34b. Change-Id: I32e0681660f53db630cbcddf0163cb4e694d9ade

jouvin · 2018-05-30T13:20:37Z

@urbonegi hopefully the last version of the commits should fix all your remaining remarks. Sorry for overlooking that the aqd.conf.defaults problem was introduced by the revert commit. I fixed it.

ned21 · 2018-05-30T15:01:33Z

if you agree, i think we could go one step at the time.

Sure. Where does that leave the code we added last year as an unsuccessful attempt to fix the same problem? If no one is using that, can we repurpose it to remove those options as a separately configurable item? Am I correct in thinking that at present the option replaces the include with an include { if_exists } ?

jouvin · 2018-05-30T15:08:29Z

@ned21 yes the commit adding the include_pan option was removed (in fact the commit adding it was reverted, this is the first commit in this PR). It was a request by @urbonegi who said that this option was added only for the non MS users and thus had no longer any use. It is not a problem to readd it, in the view of the possiblity to have profiles without pan/... templates included in the object profile. If you confirm that the previous option was not used at MS, I'd suggest that we readd it after renaming it include_pan_core_templates as I find include_pan very misleading...

ned21 · 2018-05-30T16:08:46Z

OK, so if it's reverted that's good. Re-adding it as include_pan_core_templates and have it not include them at all (not even with if_exists) is fine with me but I shall defer final judgement @urbonegi since she made a very good point about separation of concerns and tackling one thing in each PR.

jouvin · 2018-05-30T16:36:46Z

So I think we all agree eventually! It was me who suggested first to separate both issues so I agree with @urbonegi remark on this point 😄 It'd be good if you (@ned21 ) could open another issue to tackle the pan core templates include. If @urbonegi agrees with the current state of this PR, you have the green light from my side to merge it after the internal review.

urbonegi · 2018-05-31T15:31:24Z

lib/aquilon/worker/templates/cluster.py

-            lines.append("")
-
-        # Okay, here's the real content
+        pan_include(lines, ["pan/units", "pan/functions"])


I am really confused to see this happened: hosts.py have the lines.append("") after the pan_include where cluster.py and metacluster.py do not...

@urbonegi the easiest is that you tell me what is your expectation for blank lines. My original PR was one after the include of declarations.pan and one after the include of pan core templates but you commented, if I understood properly, that it was too much. Is what is currently in host.py what you want everywhere?

I see now that the host.py was inconsistent in blank lines initially (1ada34b) - so that is fine. Revert here without changing is fine.

urbonegi · 2018-05-31T15:32:46Z

etc/aqd.conf.lal

@@ -0,0 +1,150 @@
+# This config file is for running the broker outside of MS where no afs available.


What is this :) ?

Another mistake of mine, should not be part of the commit....

- Only if object_declarations_template option is true: default=false (true in aqd.conf.noms) - Template must be archetype/declarations (intended to define LOADPATH mainly) - It is the first executed template in the plenary object - Implemented for host, cluster and metacluster Fixes quattor#86 Change-Id: I7e57401fc82bfbc5af21d3d07efe543e999c23e6

jouvin · 2018-05-31T20:09:51Z

@urbonegi I had another look to the commit, saw that the comment was still refering to template_library_enabled so I fixed it. I am not clear about your expecation regarding blank lines in the profile, as #87 (review) and #87 (review) seems a bit contradictory... I decided to put one blank line after the include of declarations and one after the include of pan core templates. Let me know if you prefer something different. At least it should be consistent now.

urbonegi · 2018-06-01T13:12:41Z

@Shaeli it looks good now - ready to be fetched for review/internal CI. If tests succeed we can release this one next week.

Shaeli · 2018-06-01T19:34:00Z

@urbonegi @jouvin Fetched for review and tests succeeded :)

jouvin · 2018-06-01T20:30:53Z

Great!

urbonegi · 2018-07-03T09:33:10Z

merged: 3ccb231

…merge/master/by_topic/serviceaddr_to_sn to master * commit '2431e18e14bcffb83e2ff380f320a653e251aef8': Allow service address PTR to point at shared name Refactor add_address_alias into dbwrappers/dns.py Fix minor formatting bug in notification error

jouvin requested a review from urbonegi May 23, 2018 20:01

jouvin force-pushed the plenary_template branch from 73fabe7 to 24fb191 Compare May 24, 2018 12:44

jouvin force-pushed the plenary_template branch from 24fb191 to b5e6a3e Compare May 24, 2018 16:36

jrha changed the title ~~Plenary template adjustement to fix the template library support~~ Plenary template adjustment to fix the template library support May 24, 2018

jouvin force-pushed the plenary_template branch from b5e6a3e to 904c6af Compare May 24, 2018 17:55

urbonegi reviewed May 25, 2018

View reviewed changes

jouvin force-pushed the plenary_template branch from 904c6af to 2640be5 Compare May 25, 2018 15:11

jouvin mentioned this pull request May 25, 2018

Plenary templates readme fixes and improvements quattor/release#324

Merged

jouvin mentioned this pull request May 25, 2018

1.12.58: profile prolog template needed to define the LOADPATH #86

Closed

jouvin force-pushed the plenary_template branch from 2640be5 to e0d63db Compare May 26, 2018 06:46

jouvin mentioned this pull request May 26, 2018

Aquilon documentation improvements quattor/quattor.github.com#244

Merged

jouvin force-pushed the plenary_template branch 4 times, most recently from f19fbc0 to d88cf72 Compare May 27, 2018 17:08

urbonegi mentioned this pull request May 29, 2018

aqd.conf.noms: update inappropriate defaults #79

Closed

urbonegi reviewed May 29, 2018

View reviewed changes

urbonegi reviewed May 30, 2018

View reviewed changes

jouvin force-pushed the plenary_template branch from 058cbab to 6d5feef Compare May 30, 2018 13:12

Revert "Add support for changing template LOADPATH"

591131b

- Except lib/aquilon/aqdb/model/building.py that should not have been part of the initial commit This reverts commit 1ada34b. Change-Id: I32e0681660f53db630cbcddf0163cb4e694d9ade

jouvin force-pushed the plenary_template branch from 6d5feef to 46495bb Compare May 30, 2018 13:18

jouvin force-pushed the plenary_template branch from 46495bb to 1d15fa1 Compare May 30, 2018 20:12

urbonegi reviewed May 31, 2018

View reviewed changes

jouvin force-pushed the plenary_template branch from 1d15fa1 to 1cb4bf0 Compare May 31, 2018 16:34

jouvin force-pushed the plenary_template branch from 1cb4bf0 to 129cf74 Compare May 31, 2018 20:05

urbonegi closed this Jul 3, 2018

		@@ -34,8 +34,4 @@ class Building(Location):

		address = Column(String(255), nullable=False)

		next_rackid = Column(Integer, nullable=False, default=2)

		@@ -0,0 +1,150 @@
		# This config file is for running the broker outside of MS where no afs available.

Plenary template adjustment to fix the template library support #87

Plenary template adjustment to fix the template library support #87

Conversation

jouvin commented May 23, 2018 • edited Loading

jouvin commented May 24, 2018

Shaeli commented May 24, 2018

jouvin commented May 24, 2018

jrha commented May 24, 2018

jouvin commented May 24, 2018

jrha commented May 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jouvin commented May 25, 2018

urbonegi commented May 25, 2018

jouvin commented May 25, 2018

jouvin commented May 26, 2018

jouvin commented May 27, 2018

jrha commented May 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

urbonegi commented May 29, 2018 • edited Loading

jouvin commented May 29, 2018

jouvin commented May 29, 2018

ned21 commented May 29, 2018

jouvin commented May 29, 2018

urbonegi commented May 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jouvin commented May 30, 2018

ned21 commented May 30, 2018

jouvin commented May 30, 2018

ned21 commented May 30, 2018

jouvin commented May 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jouvin May 31, 2018 • edited Loading

Choose a reason for hiding this comment

jouvin commented May 31, 2018

urbonegi commented Jun 1, 2018

Shaeli commented Jun 1, 2018 • edited Loading

jouvin commented Jun 1, 2018

urbonegi commented Jul 3, 2018

jouvin commented May 23, 2018 •

edited

Loading

urbonegi commented May 29, 2018 •

edited

Loading

jouvin May 31, 2018 •

edited

Loading

Shaeli commented Jun 1, 2018 •

edited

Loading