Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout for long running Step Intialization #4151

Open
kcmannem opened this issue Jul 23, 2019 · 2 comments
Open

Timeout for long running Step Intialization #4151

kcmannem opened this issue Jul 23, 2019 · 2 comments

Comments

@kcmannem
Copy link
Member

With the addition of contexts to the garden/baggageclaim clients. We're now able to clean up more reliably or remotely cancel a fork of the code when needed.

We have also introduced speed improvements to streaming; to the point where strabo takes around 2.5 mins to initialize. This is the worst case scenario of how long a task may take to initialize. This gives us the tolerance to add a timeout (~30 mins is generous first guess) for the whole initialization portion of step execution.

We currently place context.Todo() in the places we'd have to add this timeout in our add_context pr.

@kcmannem kcmannem added this to Icebox in Runtime via automation Jul 23, 2019
@kcmannem kcmannem moved this from Icebox to Backlog in Runtime Jul 23, 2019
@ddadlani
Copy link
Contributor

Keep in mind that some Concourse instances are run in environments with high network latency. It may be too much to shorten the timeout so aggressively. We can definitely thread aborting and idle timeouts all the way down to Baggageclaim/Garden.

@ddadlani ddadlani removed this from Backlog in Runtime Jul 30, 2019
@ddadlani
Copy link
Contributor

ddadlani commented Aug 6, 2019

It would be nice if we can retry the step on another worker if the initialization times out, but that might be another piece of work entirely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants