Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: tag imported versions of repos #48523

Open
toothrot opened this issue Sep 21, 2021 · 10 comments
Open

x/build: tag imported versions of repos #48523

toothrot opened this issue Sep 21, 2021 · 10 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@toothrot
Copy link
Contributor

toothrot commented Sep 21, 2021

Plan: once a month (or whatever time unit)
For each x repo in dependency order (lowest first)
go get update all the deps to their latest tags
Wait for that to be green
Tag it with next version

Can probably shoehorn into existing process, maybe not running automatically at first.
0.1.0, 0.2.0, 0.3.0...

@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Sep 21, 2021
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Sep 21, 2021
@gopherbot gopherbot added this to the Unreleased milestone Sep 21, 2021
@toothrot toothrot added the Soon This needs to be done soon. (regressions, serious bugs, outages) label Oct 5, 2021
@dmitshur dmitshur added this to Planned in Go Release Team Oct 5, 2021
@dmitshur dmitshur changed the title x/build/updatedstd: tag imported versions of repos x/build: tag imported versions of repos Nov 9, 2021
@dmitshur
Copy link
Contributor

dmitshur commented Feb 8, 2022

Wait for that to be green

Automating this process will be blocked on coming up with a machine-understandable criteria of what "green" means.

@gopherbot
Copy link

gopherbot commented Aug 11, 2022

Change https://go.dev/cl/423036 mentions this issue: internal/workflow: clarify ownership

@gopherbot
Copy link

gopherbot commented Aug 11, 2022

Change https://go.dev/cl/423037 mentions this issue: internal/workflow: add expansions

gopherbot pushed a commit to golang/build that referenced this issue Aug 11, 2022
For golang/go#48523.

Change-Id: Ie55cdf3c90687d065fb1b81d0a9f97a81d6ffd1f
Reviewed-on: https://go-review.googlesource.com/c/build/+/423036
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
@heschi heschi moved this from Planned to In Progress in Go Release Team Aug 12, 2022
@heschi heschi self-assigned this Aug 12, 2022
@gopherbot
Copy link

gopherbot commented Aug 22, 2022

Change https://go.dev/cl/425088 mentions this issue: internal/task: start x repo tagging tasks

@gopherbot
Copy link

gopherbot commented Aug 22, 2022

Change https://go.dev/cl/425089 mentions this issue: internal/task: continue x repo tagging

@gopherbot
Copy link

gopherbot commented Aug 22, 2022

Change https://go.dev/cl/425090 mentions this issue: internal/relui,internal/task: move useful structs to task

@gopherbot
Copy link

gopherbot commented Aug 22, 2022

Change https://go.dev/cl/425091 mentions this issue: internal/task: finish initial implementation of x/ repo tagging

gopherbot pushed a commit to golang/build that referenced this issue Aug 25, 2022
One of the upcoming use cases for relui, tagging x/ repos, has a
dependency graph we can't know without doing significant work. This is
in conflict with one of the workflow package's design principles, that
workflows be defined ahead of time. So here's one idea for bending that
rule.

Introduce the concept of an expansion task. An expansion task runs the
same as a normal task, but in addition to its inputs it receives (a copy
of) the workflow definition and can add tasks and outputs to it as
usual. Once the task finishes running, the workflow swaps in the new
definition and continues. Other tasks can run concurrently.

One significant difference is that the expansion task must run every
time the workflow is started or resumed, so they really should be pure
functions with no network access or side effects. To encourage this,
they don't receive a Context.

Introducing this required some refactoring of the existing Run loop; we
need to allow these apparently-unused tasks to run, and deal with new
tasks popping up mid-run. But overall I think it's not too huge a
distortion to the design.

For golang/go#48523.

Change-Id: I92c164883e06474fa951abba5fa18fd78eacb8ce
Reviewed-on: https://go-review.googlesource.com/c/build/+/423037
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>
@gopherbot
Copy link

gopherbot commented Aug 25, 2022

Change https://go.dev/cl/425495 mentions this issue: internal/task: integration test for x repo tagging

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Sep 14, 2022
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
First step: list all our projects/repositories, and select those that
have a go.mod labeling them as golang.org/x as candidates to release as
the next version, or v0.1.0 for untagged repositories.

For golang/go#48523.

Change-Id: Ice92319a0726daf3bf5f94581582d8802640dffc
Reviewed-on: https://go-review.googlesource.com/c/build/+/425088
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
Next step: expand the workflow with tagging tasks, currently all no-ops.

For golang/go#48523.

Change-Id: I48cb8804dc1db6c17b748b54a194e9ef64ef6bce
Reviewed-on: https://go-review.googlesource.com/c/build/+/425089
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Sep 15, 2022
For whatever reason I've decided to write the tagging workflow in the
task package, so I need the test helpers here. I also need the website
structs to find the latest Go release later.

For golang/go#48523.

Change-Id: I92a1430bd27f0aa90c9fccefa12fb057e7e7f864
Reviewed-on: https://go-review.googlesource.com/c/build/+/425090
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Heschi Kreinick <heschi@google.com>
@gopherbot
Copy link

gopherbot commented Sep 16, 2022

Change https://go.dev/cl/431356 mentions this issue: maintner/maintnerd/maintapi: validate page size in mixed mode

@gopherbot
Copy link

gopherbot commented Sep 16, 2022

Change https://go.dev/cl/431355 mentions this issue: cmd/coordinator/internal/legacydash: uncap maintner request

gopherbot pushed a commit to golang/build that referenced this issue Sep 16, 2022
We've been seeing the coordinator fail to schedule subrepo runs on
release branches. As far as I can tell, the problem is the request to
maintner: branch = "mixed" means to get at least one commit from each
branch in the repository, but we're asking for 30 commits when we have
50-some branches. So obviously some are going to be left out. That
absence breaks findWork when it tries to map commits to branches.

I don't fully understand which branches happen to be included and which
don't, but this behavior seems clearly wrong. Let maintner decide how
many responses to send back, currently hardcoded at 500.

For golang/go#48523.

Change-Id: I1792e89de96e83ceed6a430a7456b119f633869b
Reviewed-on: https://go-review.googlesource.com/c/build/+/431355
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 20, 2022
This one got away from me a little, sorry.

Add cycle detection and breaking: we need to break some cycles involving
x/tools. We could consider fixing them by creating nested modules but I
don't want to deal with that now.

Implement UpdateGoMod: fetch the latest Go release, download it to a
buildlet with network access, and run go get to update to the previously
tagged versions. But only do this if there are dependencies, both to
save time and because go get throws an error if you try to get nothing.

Implement Tag: if there hasn't been a change since the last tag, reuse
that one. Otherwise, tag it with the selected new verison.

For golang/go#48523.

Change-Id: I29c05eb639fd748af1e0e356596adc8c3f15ccf0
Reviewed-on: https://go-review.googlesource.com/c/build/+/425091
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit to golang/build that referenced this issue Sep 20, 2022
Add an integration test for the x repo tagging workflow. Most of the
work is test infrastructure. Move the fake buildlet types into the task
package, and create a new fake Gerrit implementation that tracks tags
and commits. Also use it in the older release integration tests.

The coverage of the x repo tagging process itself is a bit thin but this
is a good starting point.

For golang/go#48523.

Change-Id: Iabf9391b56f6becadd401b8ae37a9e4b8cc5d24f
Reviewed-on: https://go-review.googlesource.com/c/build/+/425495
Auto-Submit: Heschi Kreinick <heschi@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
Go Release Team
In Progress
Development

No branches or pull requests

4 participants