Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

retry 5xx responses when attempting to start a job #1777

Merged
merged 1 commit into from Oct 10, 2022

Conversation

jonahbull
Copy link
Contributor

We occasionally see 502 responses from the Agent API when starting a job:

Failed to run job: PUT https://agent.buildkite.com/v3/jobs/0183a95c-c17e-4b0f-8058-e405362c6dc7/start: 502

Since those responses likely indicate a transient issue, I think it makes sense to attempt to retry those and other 5xx responses for this call rather than failing and having the job be re-scheduled. Open to any suggestions or input y'all might have.

Copy link
Contributor

@moskyb moskyb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i love this! i've bikeshedded the review a tad, which i'm really sorry about, but this would be an awesome change to bring into the agent.

agent/job_runner.go Outdated Show resolved Hide resolved
agent/job_runner.go Show resolved Hide resolved
api/retryable.go Outdated Show resolved Hide resolved
api/retryable.go Outdated Show resolved Hide resolved
We occasionally see 502 responses from the Agent API when starting
a job. Since those responses likely indicate a transient issue, I think
it makes sense to attempt to retry those and other 5xx responses for
this call rather than failing and having the job be re-scheduled.
@moskyb moskyb force-pushed the retry-server-errors-for-startjob branch from a627fea to 6e01873 Compare October 10, 2022 21:06
@moskyb moskyb enabled auto-merge October 10, 2022 21:09
@moskyb
Copy link
Contributor

moskyb commented Oct 10, 2022

awesome work @jonahbull! thanks so much ^_^

@jonahbull
Copy link
Contributor Author

Thanks @moskyb! Appreciate your quick and helpful feedback 🙏🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants