There should be a way to specify jobs which could affect other jobs

[jcoveney]

Imagine you have a job where if it has run on date A, then it doesn't need to be run for any previous date. If you schedule a bunch of these, however, they will all be scheduled and run. You can manually short circuit the run with a check, but it would be nice to have a cleaner way to deal with these sort of situations.

[erikbern]

Is it a recursive dependency? In requires(), you can just step backwards and find the last invocation of the same task and depend on it

[freider]

@jcoveney Isn't the basic issue that you have different task instances with the same output target? Another way of solving it is to remove the parameters on the conflicting task...
I agree it would be useful and logical to make luigi track targets instead of task ids and it's something I have discussed offline with Erik a few times, but so far we've always found some non-trivial issue that prevents us from implementing it... Still think there should be a way to do it while maintaining the task/target model, but it will probably complicate the graph visualisation. For now, I think you can circumvent the problem by either doing the short circuit (self.complete()) check at the start of run() or just removing the parameter (making any run of the task allowable for the duration of your pipeline)

[Tarrasch]

I'm closing this issue.

• It has been inactive for +4 months.
• It's not about luigi core, so not as many users are affected about this.
• The change seems quite big, it's unlikely to be sporadically picked up.
• The owner haven't responded or disappeared.
• I don't understand what this is issue is about.
• There exists a reasonable workaround for this.
• We need to check if this hasn't been fixed by now (for old issues).

Every open issue adds some clutter, and we try to make the issues fewer and make it easier for new collaborators to find. Currently we try to close any issue that meets the first checkbox + one other.

Feel free to reopen this issue at any point if you have the intent to continue to work this. :)

[Tarrasch]

big change because: Luigi relies on configuration in code. Hard to analyze

workaround: There's the deps tools now.