Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not revisiting task nodes and correctly incrementing parallelism #4318

Merged
merged 2 commits into from
Nov 1, 2023

Conversation

hamersaw
Copy link
Contributor

Tracking issue

NA

Describe your changes

Currently, if the plugin state has already been observed we immediately return without incrementing parallelism or marking the task as "dirty", indicating that it has already been processed this round. This means that, in scenarios where a task is the downstream dependency of multiple nodes, tasks may be unnecessarily evaluated multiple times within a single round. More seriously, if the task state has no changed (ex. in same phase as previous round) then additional tasks will be scheduled to satisfy the parallelism. This means that, effectively max_parallelism settings may be ignored.

In this PR we correctly increment parallelism if the state has already been seen and mark the task node as "dirty" to indicate that it should not be evaluated again this round.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Screenshots

NA

Note to reviewers

NA

…arallelism

Signed-off-by: Daniel Rammer <daniel@union.ai>
@codecov
Copy link

codecov bot commented Oct 27, 2023

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (1074826) 59.04% compared to head (e278412) 59.32%.
Report is 10 commits behind head on master.

❗ Current head e278412 differs from pull request most recent head fafd673. Consider uploading reports for the commit fafd673 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4318      +/-   ##
==========================================
+ Coverage   59.04%   59.32%   +0.28%     
==========================================
  Files         614      544      -70     
  Lines       52006    38936   -13070     
==========================================
- Hits        30705    23099    -7606     
+ Misses      18850    13556    -5294     
+ Partials     2451     2281     -170     
Flag Coverage Δ
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...lytepropeller/pkg/controller/nodes/task/handler.go 60.39% <33.33%> (+1.74%) ⬆️

... and 551 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pingsutw
pingsutw previously approved these changes Oct 27, 2023
Signed-off-by: Daniel Rammer <daniel@union.ai>
@hamersaw hamersaw merged commit b8cd12c into master Nov 1, 2023
40 of 41 checks passed
@hamersaw hamersaw deleted the bug/max-parallelism branch November 1, 2023 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants