Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

completing retries even if minSuccesses are achieved #4338

Merged
merged 3 commits into from
Nov 7, 2023

Conversation

hamersaw
Copy link
Contributor

Tracking issue

fixes #4331

Describe your changes

Check if any subtasks are in phase retryable failure before transitioning to succeeded.

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Screenshots

NA

Note to reviewers

NA

Signed-off-by: Daniel Rammer <daniel@union.ai>
Copy link

codecov bot commented Oct 31, 2023

Codecov Report

Attention: 16 lines in your changes are missing coverage. Please review.

Comparison is base (ac4ec7d) 59.07% compared to head (759c516) 59.35%.
Report is 11 commits behind head on master.

❗ Current head 759c516 differs from pull request most recent head 4817d72. Consider uploading reports for the commit 4817d72 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4338      +/-   ##
==========================================
+ Coverage   59.07%   59.35%   +0.27%     
==========================================
  Files         614      544      -70     
  Lines       52006    38975   -13031     
==========================================
- Hits        30723    23133    -7590     
+ Misses      18832    13562    -5270     
+ Partials     2451     2280     -171     
Flag Coverage Δ
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
flyteplugins/go/tasks/plugins/array/core/state.go 69.13% <100.00%> (-2.22%) ⬇️
flyteplugins/go/tasks/plugins/k8s/ray/config.go 36.36% <ø> (+7.79%) ⬆️
...ugins/go/tasks/plugins/webapi/databricks/plugin.go 65.34% <77.77%> (+3.84%) ⬆️
...ytepropeller/pkg/controller/nodes/array/handler.go 70.20% <50.00%> (+0.57%) ⬆️
...g/controller/nodes/array/node_execution_context.go 62.26% <60.00%> (+2.94%) ⬆️
...lytepropeller/pkg/controller/nodes/task/handler.go 60.39% <33.33%> (+1.74%) ⬆️
flyteplugins/go/tasks/plugins/array/catalog.go 47.71% <75.00%> (+2.41%) ⬆️
flyteplugins/go/tasks/plugins/k8s/ray/ray.go 83.68% <91.93%> (+0.30%) ⬆️

... and 544 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

eapolinario
eapolinario previously approved these changes Nov 2, 2023
Signed-off-by: Daniel Rammer <daniel@union.ai>
@@ -273,7 +273,8 @@ func SummaryToPhase(ctx context.Context, minSuccesses int64, summary arraystatus
logger.Infof(ctx, "Array is still running and waiting for resources totalWaitingForResources[%v]", totalWaitingForResources)
return PhaseWaitingForResources
}
if totalSuccesses >= minSuccesses && totalRunning == 0 {
//if totalSuccesses >= minSuccesses && totalRunning == 0 && totalRetryableFailures == 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this.

Signed-off-by: Daniel Rammer <daniel@union.ai>
@hamersaw hamersaw merged commit e794063 into master Nov 7, 2023
39 checks passed
@hamersaw hamersaw deleted the bug/maptask-retry-min-success-ratio branch November 7, 2023 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] maptasks with min_success_ratio may not always retry failed tasks
2 participants