Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Having multiple Datasets ending in parallel tasks doesn't work #985

Closed
gnn opened this issue Oct 19, 2022 · 0 comments · Fixed by #986
Closed

Having multiple Datasets ending in parallel tasks doesn't work #985

gnn opened this issue Oct 19, 2022 · 0 comments · Fixed by #986

Comments

@gnn
Copy link
Collaborator

gnn commented Oct 19, 2022

More specifically, having multiple Datasets ending in parallel tasks doesn't work if those Datasets are in a module below the egon.data.datasets package. In that case the code removing the module name prefix from task ids and the code generating the final dataset task which updates the dataset version once all parallel tasks have finished interact in a way that generates non-distinct task ids so that tasks generated later clobber the ones generated earlier. This leads to spurious cycles and other inconsistencies and bugs in the graph.

gnn added a commit that referenced this issue Oct 19, 2022
In order to reproduce the issue, the test creates two `Dataset`s ending
in parallel tasks and sets their `__module__` attribute to a module
below the `egon.data.datasets` package to simulate having the `Dataset`s
created there. Then the test checks that the ids of the final `Dataset`
tasks are distinct.
nailend added a commit that referenced this issue Oct 19, 2022
gnn added a commit that referenced this issue Nov 1, 2022
Merge the "fixes/#985-create-unique-ids-on-generated-tasks" branch into
the "continuous-integration/run-everything-2022-11-01" branch so that
the fix can pass its final test in order to move into the "dev" branch.
@nailend nailend mentioned this issue Jan 23, 2023
15 tasks
@gnn gnn closed this as completed in 5f0f521 Jan 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant