-
Notifications
You must be signed in to change notification settings - Fork 372
Expanded configuration mechanisms #147
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I haven't yet sat down and pondered how I want ye olde config override hierarchy to work here, but this is definitely part of it. Then there's also the possibility of env vars (#22) and config files, possibly multiple layers of those as well (system/global, per user, local to project, etc). I'm already semi unhappy with how the existing config mechanism can be unintuitive and hard to reason about, so I'd prefer to really hash this out sometime with pros/cons instead of iterating. Would welcome more ideas if you have them, of course. |
Generally speaking.. (leaving aside for a moment the namespace stuff in inv) in unix land the hierarchy goes something like this (lowest priority to highest priority):
Of course, some programs swap the config files with one of the environment variables - because strong consistency is not a unixy concept. So I think in the large, that would be a good idea to go with as a start. Now, about configs files, and how they override each other, and so on - for an extreme case of what could happen I like to refer to vim's runtimepath. It does (lowest precedence to highest):
(it also then has "after/" stuff for which it does in the reverse order, with all sorts of other exceptions and other rules (in vim do - :he runtimepath for the gory details). I think that's not the goal here :)) So about invoke... The way collection namespacing and config overriding works now is pretty nice (once the multiple paths to a collection issue is figured out that is). It's basically the same as the ones mentioned above...
The common thing about all of these is they basically order precedence on a concept of "locality" or "immediacy". Effectively the, the precedence is given to the option that was most likely to have been changed most recently. In the invoke case this is "i imported already written code, then called configure on a collection that had the imported stuff"... it's most likely I won't import a library that isn't written yet. All that is a really long-winded way of suggesting this type of "how to set variables" precedence order:
[1]With the option of having ~/.invoke or /etc/invoke.conf specify a search path for relevant config files. Also note - then of course naming config files conf.py is not a good idea anymore. Anyway that turned out longer than expected. hah! |
Hah, no, that's perfect - overall it obviously fits what I or any other longtime Unix user would have come up with if we'd spent a few minutes thinking about it. Thanks! The only parts I have any quarrel with:
|
+1 for this - to be able to override configuration settings and for the use of JSON and/or (JSON-subset style) YAML for configuration files. |
Not sure why I put config.py instead of invoke.conf - I'm just going to blame "Friday afternoonism". I agree that they should all be the same format. (And no preference on what the actual format is). As for the use case of task local invoke.conf vs having it built into the tasks.py: for me it comes down to separation of concerns - I just like having config details different than the code that executes them. It's not a strong technical argument, just a preference - in the end either way would be fine. |
Overriding this ticket to be a general config overhaul/expansion one. I need to start poking at this to support Fabric 2 development, where things like "storing the default SSH port and local username, then allow overriding in various ways" come into play. I skimmed tickets on Fabric's tracker and most PRs or issues mentioning the topic, and specifically JSON/YAML support (which I think is absolutely required) just tie such file loads into Fab 1's single level I googled for Python libs dealing with this to see if we can avoid reinventing the wheel (tho as usual I worry that we'd have the usual fork/collaborate/fuckit problem trying to hone the inevitable 10% of functionality that we need to change). I'll edit this post to add more links, but so far I've found:
|
+1 for etcaetera but all of them look pretty good. (note: my vote is based only on random details in my first impression and is totally subjective. I have not arguments for or against any of them) |
Cool, thanks @sophacles. Digging deeper into etcaetera now (tho partly just as it's first). For any of these, our specific needs seem to be:
|
Notes from etcaetera:
So far it fits all of my previous bullet point needs except possibly the "inject our own CLI flag values" - I need to see if we can stick those in the middle or if they can only be added via the 'defaults' adapter. Optimistic tho. EDIT: looks like the only builtin way to do that is the Overrides adapter which always comes last in the hierarchy, but that may be appropriate for CLI flags anyway. Gotta doublecheck against the above notes. We could also probably make a custom adapter, have to look into that. |
There are no explicit docs on custom adapters, but it's pretty basic, the AdapterSet accepts any number of adapter objects between the defaults (always forced first in the deque) and overrides (always last) and as long as one implements a few basic "here's my settings" methods, should be easy to subclass. And again that's if Overrides don't work for us. |
Taking the above hierarchy list, here's how it could work with an etcaetera based Config object + adapters. Note: it might replace the existing namespace driven hierarchy, or it might augment it, I need to revisit the corner cases of that design to see if it works ok in what etc gives us.
|
Re: file formats, random thoughts:
|
If it's decided that invoke should handle all the formats, then I agree there should be a strict preference ordering. That is: /etc/invoke.yml always wins, if it isn't present, then /etc/invoke.json always wins, and so on. (note: the actual ordering choice is irrelevant to me, the one I presented was example only). When wearing my devops and/or sysadmin hats, I really really want this to be deterministic and simple. This should be true any time invoke finds a file rather than being explicitly told "use this file". Additionally, whichever format gets the "always wins" choice, should be the format distributed with the pypi and github packages for things like default values and primary example scripts. Secondary examples in the other formats can be provided in a contrib area, and maybe in one documented example each. I can see the strong ordering being a bit confusing at first, and there being a bit of pushback and FAQ answering, but in the long run, I think it's far simpler than trying to deal with >1 "winning" config files and all the weird merging cases.
|
Deeper on:
|
To clarify, do you mean that if there's both a I had been imagining we'd load anything that exists but allow If you did mean the 'winning' should be more exclusive, I'm torn, I could see users being confused with either implementation. Leaning towards the exclusive option though - it is simpler to reason about as you said.
Excellent point, I agree. I've always (lately anyway) been of the opinion one should 'bless' a single option in any situation when there's multiple equally-valid options. Makes it easier for folks who have no strong pre-existing preference, and for us.
Random response thoughts:
|
Oh man, so I made a bunch of notes a couple weeks back, which I thought I had fleshed out into a comment here. Since there is no evidence of such a comment, I'll do it here. Hopefully that will help clarify all of the above. I'll also try and address the things specifically mentioned above. (Warning - this will probably be long). The way I'm using invoke is this:
Ok - so thats some background on how we currently use invoke. There are a few major pain points:
With those pain points in mind, there are a few things I actually see getting worse by adding configuration files as an option:
So in order to mitigate these I propose that OK - that covers most of the things I think are implicit to my comments from yesterday (whew). To address your specific points (in a slightly different than asked order, because I feel they build on each other a bit):
About config file format and naming:
Then this happens within invoke:
The excepion to the above is when there is an explicit command line arg stating which invoke file to use, at the I hope this giant wall of text makes my thoughts a bit clearer for you. |
Oh boy oh boy reply time~~ The tl;dr I hear for your setup is:
That strikes me as a reasonable enough 'big site deployment' of invoke. It mirrors how I perform code reuse in Invoke to a degree - the What I hear for your pain points is really one thing: it's difficult to introspect the overall state of the configuration. What are the defaults, and where; how are they overridden, and by whom; what is the final state? I agree that exposing that info would be useful, IMHO as a separate flag (bundling into A sub-issue there is your mention of knowing "what settings are available" - which really means "what settings are being consumed anywhere in the code". Dunno if there's any good way to check on this that isn't static analysis? Unless we were to set up a declaration style mechanism where you must declare settings at a high level (which could be framed as "you must always have some default value for every setting"?) Finally, re: task kwargs, I totally flaked and forgot that we already do that, Re: config file loading, I agree with most of your points re: selecting files when there are conflicts, that's all what was in my head too. Your points re: conf files vs in-python structures, I think you may have misinterpreted something, my intention here was to add config files as another option for configuration. In-Python dicts would not be removed in any way; for folks who find conf files (or env vars or etc) an easier alternative (or who want the 'extra levels' of merging) they'd simply gain the option whereas right now it's not possible w/o ugly manual hacks. |
re: conf files vs in-python-structures - yeah, I was confusing myself - rereading this thread seems to have put me on the same page with what you're saying. re: re: introspection things and showing the configuration options and what's available in contexts.
I didn't intend that - bad wording on my part. I really just meant introspection of what a context would be given the "current" tree. Assuming that there's an argument
This would require some merge tracking, but the way overrides work in the "etc" package would facilitate this. If I use a variable |
OK, I get all that. We've definitely words'd enough and I think we're mostly on the same page now too, so here's what I think needs doing for this (so I can get down to brass tacks): EDIT: moved to description, derp. |
As per the just-closed #101 we need to make sure that defaults for things like |
Type-casting down. |
Bunch of hierarchy tests already pass, first fail was unexpected, env vars are not overriding the per-project conf file (only thing that should override env vars is the runtime conf file). Wondering if the "double-load" setup isn't working the way I expect it to and env isn't getting applied - though it was working fine in the simpler tests. |
Interesting - when I introspect the AdapterSet at the fail point, the NestedEnv adapter is ending up 2nd in the list, after Defaults, instead of at the end where I expected it. Need to dig and figure out why, perhaps some sort of "where's the next spot to insert an adapter" pointer is getting reset on that first EDIT: Looks like a bug in EDIT 2: The rotate is https://docs.python.org/2.6/library/collections.html#collections.deque.rotate |
OK, fixed boog, submitted oleiade/etcaetera#19 |
Now I am at the expected failure where env/runtime conflict:
|
OK, entire config test module is green, time to add a handful of CLI module integration style tests (and probably something to actual integration suite, heh) and this might be like...done. For real. EDIT: except I forgot the debuggery stuff in the checklist, and attempted real world use to eg override the Realized that super low level debugging might mean even more invasive changes to etcaetera, hopefully not the case (and I might skip it if so, given how long this has taken). |
Wow, I think I got all the way here only to figure out that |
Yup |
Have that generally working, but now running into other issues, I suspect because it's not performing tilde expansion within At this point, really starting to suspect it's time to cut losses and just remove etcaetera from the loop as I'm spending more time working around it / modifying it, than actually using it :( Will see though, if i can just get this tiny bit farther, leaving it in would work for now. |
Have even more tiny baby branches ready to submit upstream, tho I am going to ask oleiade what he thinks of both my existing PRs and these new ones, and whether it's worth even submitting stuff or if I should just keep using my fork-as is (and probably wean myself back off of etcaetera if I find teh time). Also realized while poking at the debuggery - I somehow misplaced the tests that proved "YAML prevents loading of JSON or Python, etc"!! Poking at that now :( EDIT: tests in, naturally implementation requires Yet More Etcaetera Munging because I can't really do it as I am now w/ all 3 explicit file paths. So, a File subclass (or something) it is. |
While writing out the docstring for this, it does make me wonder if it's worth revisiting our logic here, now that we have an existing implementation that makes it much easier to have all 3 file suffixes in play. Rereading our arguments above, though, it does still feel like the option best for users as well - nobody is likely to have two types of this file in a single prefix/location and this does make things easier to reason about when troubleshooting. Will soldier on. |
Welp also realized while debugging that I never implemented the INVOKE_ prefix the docs claim we look for, for env vars. Hooray! Easy enough to make an option which I can disable for testing though, adding it to every test string feels kinda gross. |
Continuing to discover various new exciting things that were either forgotten, or unknowable until implementation time. Latest is that if we want to allow configuration of the This is now the third or fourth instance of having to bend over backwards to meet etcaetera's view of config loading :( I think I am going to keep digging this hole deeper & add another 'pre-load load()' but at this point it's clear I'd have spent much less effort just writing something from scratch. |
Now running into the 4th or 5th instance: dealing with how etc's AdapterSet is a deque and makes reasoning about the hierarchy and inserting/overwriting things a big PITA. Time to branch and try ripping it out. |
A ways into that effort (been doing other things lately too). Realizing my original idea of a truly lazy config object is too much magic/code/etc and is really not necessary, so I'm tweaking things to be a little simpler. |
Wow. The tests pass. And then they pass under Python 3. And so do the integration tests. And the docs build. Is this nightmare over?! Need to revisit checklist up top. Eg debugging is still possibly hairy - by having debug output in every config-merge, and then having to run merging many times during a single CLI session (due to reloading for env vars, re-merging after cloning, etc) it becomes very verbose, and this isn't even with per-variable logging, which might be useful in some deep cases. But really, I've spent so much God damn time on this I can't see myself waiting much longer to merge & move onwards. It's been literally months of spare time down the drain. |
W00t. For the record... It's not down the drain. We've been building our scripts aroarouthe eventual aavailability of this feature and are super excited for it! |
Yea it was a poor choice of words, but still, there's just no way this feature - however complex - should have taken literally months >_< at least I learned some hard lessons about using third party code. (Which is sad, usually I'm super aggro about people not reusing libraries.) |
I'm currently back looking at my nascent Fabric 2 branch to make sure that this new API works for its needs, since that was the entire reason I started poking this :) assuming it looks passable I'll merge to master and put out a new Invoke release. |
W00t!! |
I haven't actually finished the Fab 2 integration but I figured any changes would be minimal and was not worth having this mega huge branch floating outside master :) whee. |
Hi, what is the conclusion regarding
I am struggling to find what is the current way of setting global arguments that are valid for all tasks. |
@pieterlukasse I believe this is being followed on #153 |
@sophacles are there any linked issues for this? |
Hi @LecrisUT - I don't think there are. I've forgotten most of the context (pun not intended) around this at this point, and any issues I would have linked to it went away when the startup I was working for at the time closed down. |
Maintainer edits
Overall TODO generated by huge arse discussion thread.
.yaml
, not.yml
.Probably also needs an update to the CLI docs section re: flags and env vars (new subsection)punting on user-driven config updates that aren't env vars; env vars have their own section in config so no need to duplicateRe: env vars, need to figure out best way for user-driven config settings to work too, eventuallypuntingBrief mention of concepts doc in the tutorialActually don't see an elegant way to work it in - feels best just as a standalone regular concepts doc for nowobj['foo.bar']
should turn intoobj['foo']['bar']
or ideally with the option ofobj.foo.bar
.concepts/contexts
to adapt, possibly paring it way down or removing entirely, tl;dr it's just about explaining how contexts perform shared state.etcaetera
Config
object in place ofContext
's existing inner dicts, get tests passing again referencing it (it's dict-like).-f
or whatnot for "specific config file")cli flags+ in-module configs/etc/invoke.yaml
and/etc/invoke.json
./etc/invok.yaml
, or fatfingered the arg toinvoke -f <path>
- it should be clear when running with-d
which config files it attempted to load and which ones failed to load.Context
objects (& subclasses like Fabric2'sConnection
) need a clean API for this configuration stuff - eitherContext
directly grows all the config related methods (perhaps becoming aetcaetera.Config
subclass itself?) or it accepts some config object (again perhapsetc.Config
) as an instance member.Connection
tests.Original ticket description from sophacles
It would be nice to be able to pass context variables on the command line to invoke scripts. Something like:
And perhaps even allow targeting variables to specific namespaces, ala:
for those cases where
target2
should use the default for var1, but ns.target1 needs the overridden value.There are of course concerns about how command line set options override things in a call to
Collection.configure
... take for instance this script:Which is the right output:
option 1:
(note - because generally it's considered cli options override anything in-program)
option 2:
(note because of the way overrides work in invoke already)
I think everyone would agree that:
is correct.
The text was updated successfully, but these errors were encountered: