Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/release: in -upload mode, difficult to recover/resume if third step fails #45893

Open
1 task done
dmitshur opened this issue Apr 30, 2021 · 1 comment
Open
1 task done
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Apr 30, 2021

When release runs in upload mode (that is, with -upload flag), it goes through 3 high level stages:

  1. uploads files to GCS
  2. waits for edge cache
  3. uploads payloads to golang.org

Before starting, it helpfully checks for gcloud Application Default Credentials and refuses to continue if not (see code here). This is needed for step 1, uploading files to GCS.

A different set of permissions is needed for step 3, uploading payloads to golang.org. Specifically, a valid gomote token that's been added to validUser is needed.

What can currently happen is if release -upload is run and step 3 fails (e.g., due to a missing gomote token), then it is difficult to recover from that state. Rerunning release -upload after fixing the gomote token will fail at step 1, because the files will have been uploaded, and permissions for release managers by default allow them to add files, not overwrite existing ones.

This pitfall can be improved in a few ways:

  • add a check that a gomote token is present before starting (done in CL 315929)
    • this is trivial to implement, and will likely catch most instances
  • consider making it so that release -upload can be restarted safely without needing manual intervention
    • for example, skip over files that have been uploaded successfully instead of attempting to reupload, or modify permissions if it's safe to always allow file overwriting

I'll send a CL to resolve the first bullet point before the next release.

CC @golang/release, @dr2chase.

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Apr 30, 2021
@dmitshur dmitshur added this to the Unreleased milestone Apr 30, 2021
@gopherbot
Copy link

gopherbot commented Apr 30, 2021

Change https://golang.org/cl/315929 mentions this issue: cmd/release: check that a gomote token exists before uploading

gopherbot pushed a commit to golang/build that referenced this issue May 3, 2021
It's helpful to check that required credentials exist before starting
the upload operation. That way, if the gomote token is missing, the
user will be told right away, rather than the operation failing midway.

For golang/go#45893.

Change-Id: I70aec0a9021a714032f2d0a9a8e51465cefd262e
Reviewed-on: https://go-review.googlesource.com/c/build/+/315929
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Trust: Alexander Rakoczy <alex@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

2 participants