build,Makefile: break up the release process into two phases #33516
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
All modifications are done in the next commit. Release note: None
|
This change is |
|
There are a number of loose ends to tie up, but this is ready for an initial pass. The |
|
Oh and if anyone stumbles on this, be aware that I'm still working out the least disruptive way to transition over to this (and on migrating the docs). |
danhhz
force-pushed
the
release_process
branch
from
03f6bf7
to
05675ec
Compare
|
Prototyped the doc update https://github.com/cockroachdb/cockroach/wiki/WIP-New-Release-Process. Looks like rolling_restart_cluster.sh will have to be updated to support release binaries. It currently only uses the edge binaries. |
bdarnell
approved these changes
Jan 8, 2019
There was a problem hiding this comment.
Reviewed 5 of 5 files at r2.
Reviewable status:complete! 0 of 0 LGTMs obtained
pkg/cmd/publish-provisional-artifacts/main.go, line 102 at r2 (raw file):
// The old behavior of publish-artifacts was to publish the binaries and // immediately bless them. The new release process breaks this up, but for // now setting neither will (hopefully) exactly match the old behavior.
Why do we need to match the old behavior if this is a new command?
pkg/cmd/publish-provisional-artifacts/main.go, line 150 at r2 (raw file):
// `beta` for `1.0.1-beta+metadata` if ver.PreRelease() == "" { // TODO(dan): This is what it did before, but isn't this wrong? It
Yeah, I think you're right. The "latest" link updated here is rarely (if ever) used, though. We use "latest" builds from the master branch, but I think whenever we use a release build it's a specified version. (at one point the idea was to have "latest-2.0", "latest-2.1", etc links, but I don't see that here now)
pkg/cmd/publish-provisional-artifacts/main.go, line 206 at r2 (raw file):
// // {suffix: ".deadlock", tags: "deadlock"}, {suffix: ".race", goflags: "-race"},
While you're here, you can comment this out - we don't currently use these race binaries so they just waste build time. I'm not sure whether we're ever likely to bring them back.
I need the bound the time I can spend babysitting the next release if
anything goes wrong, so this is done in a purely additive manner which,
in the worst case, will allow falling back to the previous release
process. Once we're happy with the new one, we can clean up the old code
and scripts.
The first phase is to build provisional binaries. These are exactly the
bits we intend to ship as whatever version we're releasing. The
provisional SHA is marked in git by a tag with the form
`provisional_yyyymmddhhss_<semver>`. The various binaries (a combination
of platform, fully static vs not, race vs not) are uploaded to their
final places in s3, but nothing points at them.
At this point, the release coordinator will stage these binaries on the
long running clusters for verification. A followup PR will also allow
them to be used for a run of the nightly and weekly roachtests.
If any issues are found, a new SHA is selected, a new provisional tag is
pushed for it, binaries are built again, and verification is resumed.
Once we have provisional binaries that we're happy with, they're
blessed, which is the second phase. This makes and uploads the source
archive artifact. If the release is the latest stable release, it also
updates the various latest pointers.
The continuous publication of binaries for every green master build is
still supported but currently a bit of a hack. Ideally, the teamcity
build for it would work by releasing the provisional binaries and
immediately blessing them. However, the latest pointers work differently
between the two, so for now, this entire process is left inside the
provisional binaries step and blessing is disallowed for non-releases.
Perhaps we can unify the behaviors a bit and clean this up at some
point. The standalone workload binary is no longer continuously
published, since it's now built into the cockroach binary.
To pull all of this off, a small bit of functionality had to be added to
the Makefile. The version that a binary self reports is usually
determined automatically based on the "closest" git tag of whatever
checkout built it. We don't want the provisional tags to be considered
in this, so when we build the provisional binary, we need to override
the version. A BUILDINFO_TAG override is added for this to the various
ways to make a binary `make {mkrelease,build,...}` as well as `make
archive`.
As far as I can tell, everything is uploaded to the same place and with
the same folder and binary names as before, _except_ the folder inside
the artifact for the latest binary is now `cockroach_<semver>_<suffix`
instead of `cockroach_latest_<suffix>`. This should be fine, since both
`crl-stage-binaries` and `roachprod stage untar it with
`--strip-components 1`. (In fact, I think finally doing the TODO to
always make latest a redirect may be as simple as adding the `-L` flags
to curl in those two places.)
Release note: None
danhhz
force-pushed
the
release_process
branch
from
05675ec
to
c74f12f
Compare
danhhz
commented
Jan 8, 2019
There was a problem hiding this comment.
Thanks for the review!
bors r=bdarnell
Reviewable status:
pkg/cmd/publish-provisional-artifacts/main.go, line 102 at r2 (raw file):
Previously, bdarnell (Ben Darnell) wrote…
Why do we need to match the old behavior if this is a new command?
We don't. Removed
pkg/cmd/publish-provisional-artifacts/main.go, line 150 at r2 (raw file):
Previously, bdarnell (Ben Darnell) wrote…
Yeah, I think you're right. The "latest" link updated here is rarely (if ever) used, though. We use "latest" builds from the master branch, but I think whenever we use a release build it's a specified version. (at one point the idea was to have "latest-2.0", "latest-2.1", etc links, but I don't see that here now)
When I went to test everything end-to-end, I couldn't find anything that used this latest release build thingy, so I'm pretty sure you're right. "latest-2.0", "latest-2.1" sounds like a good idea and seems easy enough to add, but I'll leave it for a followup.
pkg/cmd/publish-provisional-artifacts/main.go, line 206 at r2 (raw file):
Previously, bdarnell (Ben Darnell) wrote…
While you're here, you can comment this out - we don't currently use these race binaries so they just waste build time. I'm not sure whether we're ever likely to bring them back.
Done.
craig bot
pushed a commit
that referenced
this pull request
Jan 8, 2019
33516: build,Makefile: break up the release process into two phases r=bdarnell a=danhhz
I need the bound the time I can spend babysitting the next release if
anything goes wrong, so this is done in a purely additive manner which,
in the worst case, will allow falling back to the previous release
process. Once we're happy with the new one, we can clean up the old code
and scripts.
The first phase is to build provisional binaries. These are exactly the
bits we intend to ship as whatever version we're releasing. The
provisional SHA is marked in git by a tag with the form
`provisional_yyyymmddhhss_<semver>`. The various binaries (a combination
of platform, fully static vs not, race vs not) are uploaded to their
final places in s3, but nothing points at them.
At this point, the release coordinator will stage these binaries on the
long running clusters for verification. A followup PR will also allow
them to be used for a run of the nightly and weekly roachtests.
If any issues are found, a new SHA is selected, a new provisional tag is
pushed for it, binaries are built again, and verification is resumed.
Once we have provisional binaries that we're happy with, they're
blessed, which is the second phase. This makes and uploads the source
archive artifact. If the release is the latest stable release, it also
updates the various latest pointers.
The continuous publication of binaries for every green master build is
still supported but currently a bit of a hack. Ideally, the teamcity
build for it would work by releasing the provisional binaries and
immediately blessing them. However, the latest pointers work differently
between the two, so for now, this entire process is left inside the
provisional binaries step and blessing is disallowed for non-releases.
Perhaps we can unify the behaviors a bit and clean this up at some
point. The standalone workload binary is no longer continuously
published, since it's now built into the cockroach binary.
To pull all of this off, a small bit of functionality had to be added to
the Makefile. The version that a binary self reports is usually
determined automatically based on the "closest" git tag of whatever
checkout built it. We don't want the provisional tags to be considered
in this, so when we build the provisional binary, we need to override
the version. A BUILDINFO_TAG override is added for this to the various
ways to make a binary `make {mkrelease,build,...}` as well as `make
archive`.
As far as I can tell, everything is uploaded to the same place and with
the same folder and binary names as before, _except_ the folder inside
the artifact for the latest binary is now `cockroach_<semver>_<suffix`
instead of `cockroach_latest_<suffix>`. This should be fine, since both
`crl-stage-binaries` and `roachprod stage untar it with
`--strip-components 1`. (In fact, I think finally doing the TODO to
always make latest a redirect may be as simple as adding the `-L` flags
to curl in those two places.)
Release note: None
Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
Build succeeded |
craig bot
pushed a commit
that referenced
this pull request
Jan 15, 2019
33718: release-2.1: build,Makefile: break up the release process into two phases r=bobvawter a=danhhz Backport 2/2 commits from #33516. /cc @cockroachdb/release --- I need the bound the time I can spend babysitting the next release if anything goes wrong, so this is done in a purely additive manner which, in the worst case, will allow falling back to the previous release process. Once we're happy with the new one, we can clean up the old code and scripts. The first phase is to build provisional binaries. These are exactly the bits we intend to ship as whatever version we're releasing. The provisional SHA is marked in git by a tag with the form `provisional_yyyymmddhhss_<semver>`. The various binaries (a combination of platform, fully static vs not, race vs not) are uploaded to their final places in s3, but nothing points at them. At this point, the release coordinator will stage these binaries on the long running clusters for verification. A followup PR will also allow them to be used for a run of the nightly and weekly roachtests. If any issues are found, a new SHA is selected, a new provisional tag is pushed for it, binaries are built again, and verification is resumed. Once we have provisional binaries that we're happy with, they're blessed, which is the second phase. This makes and uploads the source archive artifact. If the release is the latest stable release, it also updates the various latest pointers. The continuous publication of binaries for every green master build is still supported but currently a bit of a hack. Ideally, the teamcity build for it would work by releasing the provisional binaries and immediately blessing them. However, the latest pointers work differently between the two, so for now, this entire process is left inside the provisional binaries step and blessing is disallowed for non-releases. Perhaps we can unify the behaviors a bit and clean this up at some point. The standalone workload binary is no longer continuously published, since it's now built into the cockroach binary. To pull all of this off, a small bit of functionality had to be added to the Makefile. The version that a binary self reports is usually determined automatically based on the "closest" git tag of whatever checkout built it. We don't want the provisional tags to be considered in this, so when we build the provisional binary, we need to override the version. A BUILDINFO_TAG override is added for this to the various ways to make a binary `make {mkrelease,build,...}` as well as `make archive`. As far as I can tell, everything is uploaded to the same place and with the same folder and binary names as before, _except_ the folder inside the artifact for the latest binary is now `cockroach_<semver>_<suffix` instead of `cockroach_latest_<suffix>`. This should be fine, since both `crl-stage-binaries` and `roachprod stage untar it with `--strip-components 1`. (In fact, I think finally doing the TODO to always make latest a redirect may be as simple as adding the `-L` flags to curl in those two places.) Release note: None Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I need the bound the time I can spend babysitting the next release if
anything goes wrong, so this is done in a purely additive manner which,
in the worst case, will allow falling back to the previous release
process. Once we're happy with the new one, we can clean up the old code
and scripts.
The first phase is to build provisional binaries. These are exactly the
bits we intend to ship as whatever version we're releasing. The
provisional SHA is marked in git by a tag with the form
provisional_yyyymmddhhss_<semver>. The various binaries (a combinationof platform, fully static vs not, race vs not) are uploaded to their
final places in s3, but nothing points at them.
At this point, the release coordinator will stage these binaries on the
long running clusters for verification. A followup PR will also allow
them to be used for a run of the nightly and weekly roachtests.
If any issues are found, a new SHA is selected, a new provisional tag is
pushed for it, binaries are built again, and verification is resumed.
Once we have provisional binaries that we're happy with, they're
blessed, which is the second phase. This makes and uploads the source
archive artifact. If the release is the latest stable release, it also
updates the various latest pointers.
The continuous publication of binaries for every green master build is
still supported but currently a bit of a hack. Ideally, the teamcity
build for it would work by releasing the provisional binaries and
immediately blessing them. However, the latest pointers work differently
between the two, so for now, this entire process is left inside the
provisional binaries step and blessing is disallowed for non-releases.
Perhaps we can unify the behaviors a bit and clean this up at some
point. The standalone workload binary is no longer continuously
published, since it's now built into the cockroach binary.
To pull all of this off, a small bit of functionality had to be added to
the Makefile. The version that a binary self reports is usually
determined automatically based on the "closest" git tag of whatever
checkout built it. We don't want the provisional tags to be considered
in this, so when we build the provisional binary, we need to override
the version. A BUILDINFO_TAG override is added for this to the various
ways to make a binary
make {mkrelease,build,...}as well asmake archive.As far as I can tell, everything is uploaded to the same place and with
the same folder and binary names as before, except the folder inside
the artifact for the latest binary is now
cockroach_<semver>_<suffixinstead of
cockroach_latest_<suffix>. This should be fine, since bothcrl-stage-binariesandroachprod stage untar it with--strip-components 1. (In fact, I think finally doing the TODO to always make latest a redirect may be as simple as adding the-L` flagsto curl in those two places.)
Release note: None