Delayed target regex

[felixfontein]

Allows to specify regexes for delayed task loaders, which allows to specify targets of delayed tasks with doit run.

If there is more than one regex matching a target, creates a task which has all potential tasks as a dependency and adds it to the list of selected tasks. This allows to specify any target created by a DelayedTask as long as it provides a regex which matches the target name.

There's an option to doit run called --auto-delayed-regex which takes '.*' as a regex for every delayed loaded task for which no regex is specified explicitely.

fixes #20

[felixfontein]

I think the remaining failed test isn't my fault. The problem seems to be that the test task has no action. Replacing Task("t_name", None) with Task("t_name", ['']) seems to make that test pass.

[schettino72]

Actually there are several issues with this implementation. I had not noticed you used task_dep, so of course it will build too much. I will fix this myself, just give me some time... :D

[felixfontein]

Yeah, task_dep is not a good idea at all. But it was the only one which I actually got working after spending many hours of trying other approaches.

My original plan is as follows:

• for every task (supposed to build), we keep track of a list of targets we hope it might lead to, or a special marker in case we know that this task has to be processed;
• when a target is 'created' by a task (because it shows up in its targets list), the lists are checked: if any other task is ran because it might lead to this target, the target is removed from its list, and it its list is now empty, it will not be build.

That should minimize the number of targets which have to be built until the target is found (and at the end we know whether all requested targets have been built, and if not, an error could be emitted).

This (probably) requires some deeper changes at least in control.py, runner.py and loader.py. I've spend some time trying to implement this, but unfortunately didn't got it working in the end.

[schettino72]

I merged this, please check if it works as expected for you. Although I already merged there are 2 pending issues:

• needs documentation
• if you specify a target on command line that is not created by any dealyed task you get a pretty unhelpful error message like

DependencyException: Dependent file 'zzz' does not exist.

It should be:

ERROR: cmd `run` invalid parameter: "zzz". Must be a task, or a target.
Type "doit list" to see available tasks

[felixfontein]

I also get that exception if I the file is created by a delayed task: assume you have two delayed tasks, A and B. Both regexes match a file f, and B is the one who builds it. Now if you run doit build f, two tasks are created: _regex_target_f:A which loads the tasks for A, and _regex_target_f:B which loads the tasks for B. (We also have that B depends on A.) Now while executing, first _regex_target_f:A is processed. Its tasks are loaded, the generated tasks are executed, and then doit wants to process _regex_target_f:A. But the file dependency of that task, f, is not known yet -- so doit will fail, before actually loading the tasks for B.

I had the problem during building a blog, which creates of course a way more complex set of tasks, but I think it boils down to this problem. I can also re-create this with the nikola demo site and nikola's earlytask_impl branch: just run nikola build once, then delete output/robots.txt, and run nikola build output/robots.txt. I got:

Scanning posts....done!
########################################
DependencyError - taskid:_regex_target_output/robots.txt:render_pages
ERROR: Task '_regex_target_output/robots.txt:render_pages' checking dependencies
Traceback (most recent call last):
  File "~/doit/doit/dependency.py", line 626, in get_status
    file_stat = os.stat(dep)
FileNotFoundError: [Errno 2] No such file or directory: 'output/robots.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/doit/doit/runner.py", line 125, in select_task
    node.run_status = self.dep_manager.get_status(task, tasks_dict)
  File "~/doit/doit/dependency.py", line 629, in get_status
    .format(dep))
doit.dependency.DependencyException: Dependent file 'output/robots.txt' does not exist.

[felixfontein]

(Actually, that was why I used task_dep in the first place. It resulted in way too many things built, but it didn't stop with this problem :) )

[schettino72]

The problem you described should be fixed by a075e02

Please check and let me know.

I also created #60 to track the remaining tasks. I would appreciate if you could help with that!

[felixfontein]

Yes, it seems to work fine now (after some quick tests). Thanks a lot for implementing and fixing :)