New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/playground: deploy may take over 10 minute and timeout #28046

Open
dmitshur opened this Issue Oct 5, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@dmitshur
Member

dmitshur commented Oct 5, 2018

I've ran into this when deploying the Go Playground with Go 1.11.1 recently (see CL 140097), where the cloud build failed because it took more than the default 10 minute timeout:

ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at https://console.cloud.google.com Failure status: UNKNOWN: Error Response: [4] DEADLINE_EXCEEDED

In the recent past, some builds have taken around 9~ minutes. They were faster earlier on.

People have run into the 10 minute timeout in other contexts too. E.g., see https://stackoverflow.com/questions/50046545/google-app-engine-build-timed-out-during-deployment.

For now, this issue is to gather data and see if this keeps happening. We can then decide on how to best deal with it. If you run into this, please post data here. /cc @bradfitz @andybons

Workaround

The easiest temporary workaround I'm currently aware of is to increase the timeout before the gcloud app deploy step. For example:

gcloud config set app/cloud_build_timeout 900 # 15 mins

Note that it's a global config value.

@gopherbot gopherbot added this to the Unreleased milestone Oct 5, 2018

@ysmolsky

This comment has been minimized.

Show comment
Hide comment
@ysmolsky

ysmolsky Oct 6, 2018

Member

I wonder if this build command includes building the docker container? I have noticed that building playground docker container became very slow. Specifically fetching packages is slow. Since I was not sure if it was my network, I did not create ticket for that.

Member

ysmolsky commented Oct 6, 2018

I wonder if this build command includes building the docker container? I have noticed that building playground docker container became very slow. Specifically fetching packages is slow. Since I was not sure if it was my network, I did not create ticket for that.

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Oct 7, 2018

Member

I wonder if this build command includes building the docker container?

Yes, it does. That's the part that takes up to 8-9 minutes.

Specifically fetching packages is slow.

I didn't notice that to be the slow part, but thanks for pointing out. I'll watch it more closely next time.

Member

dmitshur commented Oct 7, 2018

I wonder if this build command includes building the docker container?

Yes, it does. That's the part that takes up to 8-9 minutes.

Specifically fetching packages is slow.

I didn't notice that to be the slow part, but thanks for pointing out. I'll watch it more closely next time.

@dmitshur

This comment has been minimized.

Show comment
Hide comment
@dmitshur

dmitshur Oct 8, 2018

Member

A data point from today's deploy (for #25462): the cloud build step took 10 min 58 sec. (It was within my 15 minute timeout.)

Member

dmitshur commented Oct 8, 2018

A data point from today's deploy (for #25462): the cloud build step took 10 min 58 sec. (It was within my 15 minute timeout.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment