Skip to content

fix(test): increase SDK maxRetries from 1 to 3 for smoketest resilience#786

Merged
sid-rl merged 1 commit into
mainfrom
sid/increase-smoketest-retries
May 5, 2026
Merged

fix(test): increase SDK maxRetries from 1 to 3 for smoketest resilience#786
sid-rl merged 1 commit into
mainfrom
sid/increase-smoketest-retries

Conversation

@sid-rl
Copy link
Copy Markdown
Contributor

@sid-rl sid-rl commented May 5, 2026

User description

Description

Increases maxRetries from 1 to 3 in both makeClient() and makeClientSDK() smoketest utility functions. With maxRetries=1, the SDK makes only 2 total attempts — a single transient 5xx or connection timeout fails the test immediately. With maxRetries=3, the SDK makes 4 total attempts using its built-in exponential backoff (1s base, 2x factor, 60s cap, jitter).

Companion to https://github.com/runloopai/runloop/pull/9069 which reduces Python smoketest concurrency and fixes blueprint name collisions.

Testing

Run smoketest workflows after merge and verify TS smoketest pass rate improves, particularly for blueprint timeout failures.

🤖 Generated with Claude Code


CodeAnt-AI Description

Make smoketests retry transient API failures more often

What Changed

  • Smoketests now retry failed requests up to 3 times instead of 1, so temporary timeouts and 5xx errors are less likely to fail the run.
  • This applies to both test clients used by the smoketest suite.

Impact

✅ Fewer flaky smoketest failures
✅ More reliable test runs after brief backend errors
✅ Less rerunning of successful tests

🔄 Retrigger CodeAnt AI Review

Details

💡 Usage Guide

Checking Your Pull Request

Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.

Talking to CodeAnt AI

Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:

@codeant-ai ask: Your question here

This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.

Example

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

Preserve Org Learnings with CodeAnt

You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:

@codeant-ai: Your feedback here

This helps CodeAnt AI learn and adapt to your team's coding style and standards.

Example

@codeant-ai: Do not flag unused imports.

Retrigger review

Ask CodeAnt AI to review the PR again, by typing:

@codeant-ai: review

Check Your Repository Health

To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.

With maxRetries=1, the SDK makes only 2 total attempts before failing.
Transient 5xx errors and connection timeouts during smoketests cause
immediate test failure. Bumping to maxRetries=3 gives 4 total attempts
with the SDK's built-in exponential backoff (1s base, 2x factor, 60s
cap, jitter), making smoketests resilient to transient backend issues.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codeant-ai
Copy link
Copy Markdown
Contributor

codeant-ai Bot commented May 5, 2026

CodeAnt AI is reviewing your PR.


Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

@codeant-ai codeant-ai Bot added the size:XS This PR changes 0-9 lines, ignoring generated files label May 5, 2026
@sid-rl sid-rl requested a review from dines-rl May 5, 2026 01:47
@codeant-ai
Copy link
Copy Markdown
Contributor

codeant-ai Bot commented May 5, 2026

CodeAnt AI finished reviewing your PR.

Copy link
Copy Markdown
Contributor

@dines-rl dines-rl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait why do we need 3? Shouldn't have anythign that fails... more than once?

@sid-rl
Copy link
Copy Markdown
Contributor Author

sid-rl commented May 5, 2026

Wait why do we need 3? Shouldn't have anythign that fails... more than once?

this is to harden flaky smoketests. changing maxRetries from 1 to 3 changes the total retries from 2 to 4

@sid-rl sid-rl merged commit c8b5448 into main May 5, 2026
8 checks passed
@sid-rl sid-rl deleted the sid/increase-smoketest-retries branch May 5, 2026 01:52
@stainless-app stainless-app Bot mentioned this pull request May 5, 2026
@dines-rl
Copy link
Copy Markdown
Contributor

dines-rl commented May 5, 2026

Yeah but I guess whats failing more than once? Logs ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS This PR changes 0-9 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants