Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Identical tasks with different target paths should be allowed #42

Open
richpsharp opened this issue Oct 11, 2020 · 0 comments
Open

Identical tasks with different target paths should be allowed #42

richpsharp opened this issue Oct 11, 2020 · 0 comments
Assignees
Labels
bug Something isn't working in progress
Milestone

Comments

@richpsharp
Copy link
Collaborator

It's common to add two tasks that have different file outputs without explicitly trying to do so. TaskGraph will raise an exception if two tasks could overwrite the same file but it also has the behavior of treating a Task uniquely based on its input hash arguments. This leads to cases when doing batch 2D convolution where coincidentally two kernel rasters have the same input arguments even though the context is different. Here's what TaskGraph does in that case:

  1. Accept first task
  2. Note second task is the same except for output so waits for the first one to be done
  3. First task completes and updates the TaskGraph database that the task is complete
  4. Second task runs, sees that the expected output is not its expected output so it runs the second identical task.
  5. Second task completes and updates the TaskGraph database that the task is complete.

On a second iteration the first Task in the database will not have a target path list entry for its output because it was overwritten by the second task, so it churns triggering a re-run of every task below this that uses those files.

This is a bug and I suspect an an early attempt at copying a result to avoid recomputation but now we have explicit flags for that. This also turned out to be an apparently bad idea since it causes more recomputation than it prevents. This issue is about removing that duplicate functionality avoidance and instead allowing tasks to be uniquely defined by their input AND output arguments.

@richpsharp richpsharp self-assigned this Oct 11, 2020
@richpsharp richpsharp added the bug Something isn't working label Oct 11, 2020
@phargogh phargogh added this to the 1.0 milestone Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working in progress
Projects
None yet
Development

No branches or pull requests

2 participants