Let worker wait for tasks running on other workers to complete.

[jimjh]

See #222 and #223.

[erikbern]

Nice! However I think it should be configurable and default should be the existing behavior. As I just pointed out in issue #222, there might be some side effects like creating hundreds of worker processes just sitting around waiting for more work. So I'd rather make this an "experimental" feature that you can turn on if needed. Does that make sense?

[jimjh]

I see where you are coming from. I guess I am using luigi slightly differently, and was operating under the assumption that all submitted tasks get run eventually unless there is a failure of some sort.

There is a real risk that the corner case you described could occur, especially if workflows are scheduled every minute. I will add a configuration option named worker-keep-alive and re-push.

[jimjh]

OH YEA travis passed. I was wondering what the errors were about.

[erikbern]

LGTM. Happy to merge unless anyone objects. I'll let it sit here for a day or two to give other people a chance

[jimjh]

Merge?

[duckontheweb]

Did this ever make it's way into the documentation? I see that it's referenced in a couple of places, but I didn't see an entry in the [core] section of the configuration docs for it.

[dlstadther]

@duckontheweb it's in the [worker] section

[duckontheweb]

It seems like the worker code here is getting that config from the [core] section, though. Am I misinterpreting that?

[dlstadther]

The legacy method was to provide the config_path=dict(section='core', name='worker-keep-alive'). So for legacy purposes, it will look under [core] for worker-keep-alive. But config classes (class worker(luigi.config)) will look under [worker] (the name of the class) for the parameter names listed. So in this case keep_alive.

So to be verbose in this example, you could provide

[core]
worker-keep-alive: True

or

[worker]
keep_alive: True

[duckontheweb]

Perfect, thanks for clarifying that for me!