Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

invoke.yaml ignored for a tasks module. #467

Closed
lukeorland opened this issue Aug 3, 2017 · 5 comments
Closed

invoke.yaml ignored for a tasks module. #467

lukeorland opened this issue Aug 3, 2017 · 5 comments

Comments

@lukeorland
Copy link

@lukeorland lukeorland commented Aug 3, 2017

If I use a module called tasks, as a directory named tasks/, containing an __init__.py, instead of tasks.py, where should the project-level invoke.yaml configuration file live? It is ignored if it's in the same directory as the tasks/ directory and also ignored if it's in the tasks/ directory.

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Aug 4, 2017

It should be picked up from the same directory of wherever import tasks picked up your code (so, same directory as tasks/ in this case.) So perhaps there's a bug.

Can you post the output of INVOKE_DEBUG=1 inv --list (or any other inv command, but --list will limit how much other extra debuggery shows up)? It will include some info on where it looked for task and config files & what it loaded up.

Loading

@lukeorland
Copy link
Author

@lukeorland lukeorland commented Aug 4, 2017

$ tree
.
├── invoke.yaml
└── tasks
    └── __init__.py
$ cat invoke.yaml
tasks:
    auto_dash_names: false
$ cat tasks/__init__.py
from invoke import task


@task
def my_awesome_task(c):
    print("Awesome!")
$ INVOKE_DEBUG=1 inv --list
invoke.config._load_file: Didn't see any /etc/invoke.yaml, skipping.
invoke.config._load_file: Didn't see any /etc/invoke.yml, skipping.
invoke.config._load_file: Didn't see any /etc/invoke.json, skipping.
invoke.config._load_file: Didn't see any /Users/lukeorland/.invoke.yaml, skipping.
invoke.config._load_file: Didn't see any /Users/lukeorland/.invoke.yml, skipping.
invoke.config._load_file: Didn't see any /Users/lukeorland/.invoke.json, skipping.
invoke.config.merge: Merging config sources in order onto new empty _config...
invoke.config.merge: Defaults: {'run': {'shell': '/bin/bash', 'hide': None, 'env': {}, 'replace_env': False, 'in_stream': None, 'encoding': None, 'echo': False, 'warn': False, 'echo_stdin': None, 'watchers': [], 'pty': False, 'out_stream': None, 'err_stream': None, 'fallback': True}, 'tasks': {'search_root': None, 'collection_name': 'tasks', 'dedupe': True, 'auto_dash_names': True}, 'sudo': {'password': None, 'prompt': '[sudo] password: ', 'user': None}, 'runners': {'local': <class 'invoke.runners.Local'>}}
invoke.config.merge: Collection-driven: {}
invoke.config._merge_file: System-wide config file (/etc/invoke.py): {}
invoke.config._merge_file: Per-user config file (/Users/lukeorland/.invoke.py): {}
invoke.config._merge_file: Per-project config file has not been loaded yet, skipping
invoke.config.merge: Environment variable config: {}
invoke.config._merge_file: Runtime config file has not been loaded yet, skipping
invoke.config.merge: Overrides: {}
invoke.config.merge: Modifications: {}
invoke.config.merge: Deletions: {}
invoke.program._parse: argv given to Program.run: None
invoke.program.normalize_argv: argv was None; using sys.argv: ['/Users/lukeorland/workspace/venv/bin/inv', '--list']
invoke.program.parse_core_args: Parsing initial context (core args)
invoke.parser.__init__: Initialized with context: <parser/Context: {u'hide': <Argument: hide>, u'help': <Argument: help (h) ?>, u'config': <Argument: config (f)>, u'write-pyc': <Argument: write-pyc [bool]>, u'list': <Argument: list (l) [bool]>, u'collection': <Argument: collection (c)>, u'echo': <Argument: echo (e) [bool]>, u'version': <Argument: version (V) [bool]>, u'warn-only': <Argument: warn-only (w) [bool]>, u'debug': <Argument: debug (d) [bool]>, u'pty': <Argument: pty (p) [bool]>, u'root': <Argument: root (r)>, u'no-dedupe': <Argument: no-dedupe [bool]>, u'complete': <Argument: complete [bool]>}>
invoke.parser.__init__: Available contexts: {}
invoke.parser.complete_context: Wrapping up context None
invoke.parser.parse_argv: Starting argv: ['--list']
invoke.parser.handle: Handling token: '--list'
invoke.parser.handle: Saw flag '--list'
invoke.parser.switch_to_flag: Moving to flag <Argument: list (l) [bool]>
invoke.parser.switch_to_flag: Marking seen flag <Argument: list (l) [bool]> as True
invoke.parser.changing_state: ParseMachine: 'context' => 'end'
invoke.parser.complete_context: Wrapping up context None
invoke.program.parse_core_args: Core-args parse result: [<parser/Context: {u'hide': <Argument: hide>, u'help': <Argument: help (h) ?>, u'config': <Argument: config (f)>, u'write-pyc': <Argument: write-pyc [bool]>, u'list': <Argument: list (l) [bool]>, u'collection': <Argument: collection (c)>, u'echo': <Argument: echo (e) [bool]>, u'version': <Argument: version (V) [bool]>, u'warn-only': <Argument: warn-only (w) [bool]>, u'debug': <Argument: debug (d) [bool]>, u'pty': <Argument: pty (p) [bool]>, u'root': <Argument: root (r)>, u'no-dedupe': <Argument: no-dedupe [bool]>, u'complete': <Argument: complete [bool]>}>] & unparsed: []
invoke.program._parse: Finished parsing core args
invoke.program._parse: No default namespace provided, trying to load one from disk
invoke.loader.find: FilesystemLoader find starting at '/Users/lukeorland/workspace/invoke-yaml'
invoke.loader.find: Found module: '/Users/lukeorland/workspace/invoke-yaml/tasks'
invoke.parser.__init__: Adding <parser/Context 'my-awesome-task'>
invoke.program.parse_tasks: Parsing tasks against <Collection 'tasks': my-awesome-task>
invoke.parser.__init__: Initialized with context: <parser/Context: {u'hide': <Argument: hide>, u'help': <Argument: help (h) ?>, u'config': <Argument: config (f)>, u'write-pyc': <Argument: write-pyc [bool]>, u'list': <Argument: list (l) [bool]>, u'collection': <Argument: collection (c)>, u'echo': <Argument: echo (e) [bool]>, u'version': <Argument: version (V) [bool]>, u'warn-only': <Argument: warn-only (w) [bool]>, u'debug': <Argument: debug (d) [bool]>, u'pty': <Argument: pty (p) [bool]>, u'root': <Argument: root (r)>, u'no-dedupe': <Argument: no-dedupe [bool]>, u'complete': <Argument: complete [bool]>}>
invoke.parser.__init__: Available contexts: {'my-awesome-task': <parser/Context 'my-awesome-task'>}
invoke.parser.complete_context: Wrapping up context None
invoke.parser.parse_argv: Starting argv: []
invoke.parser.changing_state: ParseMachine: 'context' => 'end'
invoke.parser.complete_context: Wrapping up context None
invoke.program.parse_tasks: Resulting task contexts: []
invoke.program.run: Received a possibly-skippable exception: Exit()
Available tasks:

  my-awesome-task

Notable line: invoke.config._merge_file: Per-project config file has not been loaded yet, skipping

$ pip freeze | grep invoke
invoke==0.20.2

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Aug 4, 2017

Ah, yea. This is because you're not actually getting to the task execution step; project conf files are loaded at task execution time. Note the following from the auto dash docs:

If you’d prefer the underscores to remain instead, you can update your configuration to set tasks.auto_dash_names to False in one of the loaded-by-default config files – typically, either the system or user location.

That said, I found myself surprised in retrospect, and suspect the current setup can be improved. Quick background (mostly rubber-ducking to myself here, but if you grok it, even better):

  • Prior to 0.20, all config files were loaded "late", right before executing tasks.
  • In 0.20 (due to #310 and #329) we broke that up and tried loading some parts of the config "early", namely the system/user files the docs mention above. However, the rest - including project conf files - continue to be loaded on the later side.
  • Project conf files must be loaded at least semi-late - specifically, after the tasks collection is found/loaded. This is why they occur in the later step and not the earlier.
  • That later config-update step occurs after the 'parse' step, which is the crux of the problem, since it parses core args, loads collection (in case one used --collection), then parses tasks/task args.
  • Your problem is that the third substep there, parsing tasks, is where things currently bail out, expecting my_awesome_task but only seeing my-awesome-task, as project conf isn't loaded yet (as above.)
  • The doc snippet I linked technically "fixes" this by encouraging you to put auto_dash_names: false into system or user locations. If there were some other reason that project confs had to come very late in the process, this is where we'd stop, and I'd ask you how to improve visibility of the docs.
  • However, I'm realizing now that when I implemented all this, the order-of-operations focus was mostly on #310, "make sure ppl can configure the task loading stuff in user/system conf files", and I didn't take things to their logical extreme.
  • Which in this case is, "can't we make the parse step even more granular, and inject project conf loading as early as possible - namely immediately after the task collection is loaded?"
  • And I think that's possible, so I'll see if I can prove it...check back later.

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Aug 4, 2017

Ah yea, there is still a wrinkle - creating the Collection is when we tell it whether or not to auto-dash names. So if we wait until the collection is completely loaded to grab the project config, it's still too late to affect that side of things.

I think we can still shove project loading earlier but it will mean forcing the loader to learn more about the config system (i.e. violating separation of concerns). Not ideal but might still be doable, and I don't think there will be any drawbacks unless one has valid Invoke conf file names lying around between the CWD and the actual tasks file. Even then, it should just be log spam; we can always unload such a config if there turned out to be no collection there.

EDIT: Though this could mean the loader shouldn't be the one making the collection, only returning the info required to make one. That feels better?

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Aug 4, 2017

Yea that feels cleaner now, at the mild expense of making a test module do marginally more work on its end.

Pushing commits now; barring any issues caught by my pre-push hook or Travis, this should get released as 0.20.3 soon. Thanks again & sorry this particular change had so many annoying roadblocks in the way of disabling it!

Loading

bitprophet added a commit that referenced this issue Aug 4, 2017
bitprophet added a commit that referenced this issue Aug 4, 2017
Includes nuking some older tests that were outdated in recent changes
@bitprophet bitprophet closed this in e3eda3b Aug 4, 2017
bitprophet added a commit that referenced this issue Aug 4, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants