Release 0.24 - March 2019

[laurentlb]

Target RC date: March 1st

[katre]

Looks like we have a good baseline for the release in 235e76b. Test run is here: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/831. @meteorcloudy has investigated the two failing projects and they are not release-related.

[meteorcloudy]

@oquenchil 235e76b won't contain your flip of --incompatible_disable_legacy_cc_provider

[katre]

The bazel-at-head-plus-downstream tests that include the flag flip for
incompatible_disable_legacy_cc_provider show many failures of downstream projects related to the missing "cc" provider (see https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/834). I think this feature needs more time to migrate downstream projects.

[meteorcloudy]

@katre Yes, @oquenchil and I just agreed to rollback the flag flip

[katre]

Creating rc1:

scripts/release/release.sh create 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86

Tests are running now.

[katre]

Release is blocked by #7605: I cannot actually get the release binaries to build.

[meteorcloudy]

@katre Thanks for adding migration-0.24 for the incompatible flags didn't make into 0.24.0.
But we should not remove migration-0.22 and migration-0.23 for them, because if we are using an old version of Bazel on CI we won't be able to fetch them for testing. (For example, we currently still uses 0.22.0 because 0.23.0 has a bug and 0.23.1 is not yet released)
https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/22#98b39372-548a-43be-827f-b096f70ab30e only got 7 incompatible flags, but there are more that should be tested.

I'm adding the migration labels back for 0.22 and 0.23

[philwo]

Thank you @meteorcloudy!

[katre]

@meteorcloudy My mistake. We should update the docs at https://github.com/bazelbuild/continuous-integration/blob/master/docs/release-playbook.md and clarify that (I think I see where I went wrong, so expect a PR from me today).

[katre]

Creating rc2:

scripts/release/release.sh create --force_rc=2 0.24.0 235e76b0e756d05599a6cbe1663ff8e13df84a86 badd82e4c5cda7b3232481e1c63a5550ac898cd8

[vbatts]

@hlopko fedora 30 is soon to release, and the current "rawhide" will be fedora 31 about 6 months later. I bet there will be a fix before then

[werkt]

I'd like to suggest that a 0.24.1 be released which incorporates changes from #7860 - it will not apply cleanly, as it comes after 9813c58, but it represents a serious bug in 24 tracked in #7856 that we'd much rather not wait for a cut of 25 for a fix of.

[katre]

@werkt Is that issue a release blocker? Asking @buchgr to help triage.

[werkt]

I'll defer to @buchgr on that, but we have been broken since 0.22 with this particular issue (0.23 introduced the break), are still broken on 0.24, won't be fixed until 0.25 without intervention, and have only had a chance to properly diagnose it now (we were late to adopt 0.23 as well).

[katre]

Makes sense for a patch release. I want to wait for the result on #7874 before I cut it, however.

[hlopko]

Oh wait gcc-9 is not yet released :) Ok I say #7874 doesn't warrant a patch release.

[vbatts]

On 29/03/19 03:50 -0700, Marcel Hlopko wrote: Oh wait gcc-9 is not yet released :) Ok I say [1]#7874 doesn't warrant a patch release.

yeah. Fedora Rawhide, is literally that. It's meant as an early canary. :-)

[katre]

SInce #7874 isn't critical, I will cut a patch release today that includes 80bff99 (to address #7856).

[katre]

I tried to create 0.24.1rc1:

./scripts/release/release.sh create --force_rc=1 0.24.1 f092ec388a4b58788285b928c86f0f81561d4be4 80bff99811356ea5ec2a66f0217df75a56923069

However, I cannot merge this cherry-pick: the code has changed too far in between and I have no idea what should actually be fixed.

@werkt and @buchgr Please give me directions (or, better yet, a commit I can apply to 0.24.0) on how to resolve this. Or wait for the 0.25.0 RC to be delivered early next week.

[werkt]

Working one up based on release-0.24.0 now

[werkt]

https://github.com/werkt/bazel/tree/retry-exception-fix-0.24 contains a single commit that addresses the issue and applies to the base commit of 0.24

[katre]

Okay, second attempt at 0.24.1rc1:

$ git branch -D release-0.24.1
$ git remote add werkt git@github.com:werkt/bazel.git
$ git fetch werkt
$ ./scripts/release/release.sh create --force_rc=1 0.24.1 f092ec388a4b58788285b928c86f0f81561d4be4 c56c489119e6587975964c44ceb9e429ad950736

Waiting on tests and binaries now.

[katre]

0.24.1rc1 is now out: https://releases.bazel.build/0.24.1/rc1/index.html
@werkt Please test this.

[werkt]

Tested, this correctly generates an error that would result in a fallback, excellent!

[werkt]

This release candidate looks acceptable from my end, is there any update on this?

[katre]

I am releasing it today (two business days have passed with no new release blockers).

[katre]

0.24.1 is now released (https://github.com/bazelbuild/bazel/releases/tag/0.24.1).

@petemounce, @vbatts, and @excitoon: can you update the packages you each maintain? Thanks!

[vbatts]

fedora and centos build: https://copr.fedorainfracloud.org/coprs/build/876626

[petemounce]

Pushed.

[menny]

Bazel crash immediately with error:

/var/lib/spotify/buildagent/teamcity/work/70b3e9bc32b57ee2/bin/bazel: relocation error: /var/lib/spotify/buildagent/teamcity/work/70b3e9bc32b57ee2/bin/bazel: symbol _ZTVNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEEE, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference

On Linux gew1-buildagent-ssd-0933-d.gew1.spotify.net 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

0.23.1 works fine.

[katre]

What version of glibcxx are you using?

@hlopko, is this something that is easy to fix?

[hlopko]

@menny, do you use your own C++ toolchain? Did you migrate for #6861, including enabling features that used to be enabled by default in the past:

• dependency_file
• random_seed
• module_maps
• module_map_home_cwd
• header_module_compile
• include_paths,
• pic
• preprocessor_define

[hlopko]

Oh ignore me, I misuderstood the issue.

[katre]

@menny @hlopko Please file this as a separate issue to ensure it isn't missed for 0.25.

[hlopko]

Hi @menny - is this still an issue? Can you repro with a clean install of Ubuntu 14.04 (docker will be fine)? Would you mind creating a new issue and pinging me there so we can investigate properly? Thanks!

[menny]

@hlopko, It's still an issue on some of our CI pools (the ones that have ubuntu 14).
I'll try to reproduce it on a Docker container.

[katre]

I am closing this issue since 0.24.1 is fully released. Once the followup issue about GLIBCXX_3.4.21 is triaged and we determine if a further patch to 0.24 is required, I will re-open.

[menny]

@hlopko I've decided to upgrade the old CI machine to Ubuntu 18.
The crash is not an issue any longer.