Skip to content

Don't retry on grpc message too large errors#916

Merged
tconley1428 merged 11 commits intomasterfrom
no_retry_on_grpc_too_large
May 15, 2025
Merged

Don't retry on grpc message too large errors#916
tconley1428 merged 11 commits intomasterfrom
no_retry_on_grpc_too_large

Conversation

@tconley1428
Copy link
Copy Markdown
Contributor

What was changed

Don't retry on grpc message too large errors. Forward the error and fail the workflow task

Why?

Retries will not succeed, as they don't change the size of the message

Checklist

  1. Closes [Bug] Do not auto-retry gRPC-message-size-too-large errors #462

  2. How was this tested:
    New worker tests

  3. Any docs updates needed?

@tconley1428 tconley1428 requested a review from a team as a code owner May 14, 2025 23:44
Copy link
Copy Markdown
Member

@Sushisource Sushisource left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall idea looks good to me. Will want to add cloud test as discussed on Slack.

Comment thread client/src/retry.rs Outdated
Comment thread tests/integ_tests/worker_tests.rs Outdated
Copy link
Copy Markdown
Contributor

@cretz cretz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No comment on the general Rust logic, just a couple of notes. Leaving code review to @Sushisource.

Comment thread client/src/retry.rs
Comment thread core/src/worker/workflow/mod.rs
@tconley1428 tconley1428 force-pushed the no_retry_on_grpc_too_large branch from 8060969 to 6e8ef10 Compare May 15, 2025 15:50
Copy link
Copy Markdown
Member

@Sushisource Sushisource left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet! Looking good

Comment thread .cargo/config.toml Outdated
@tconley1428 tconley1428 force-pushed the no_retry_on_grpc_too_large branch from 1f80d8e to 60746f9 Compare May 15, 2025 19:44
@tconley1428 tconley1428 merged commit 8a4bd7f into master May 15, 2025
20 checks passed
@tconley1428 tconley1428 deleted the no_retry_on_grpc_too_large branch May 15, 2025 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Do not auto-retry gRPC-message-size-too-large errors

3 participants