General Retrospective for Oct 2022 Releases

[zdtsw]

Summary

A retrospective for all efforts surrounding the titular releases.

All community members are welcome to contribute to the agenda via comments below.

This will be a virtual meeting after the release, with at least a week of notice in the #release Slack channel.

On the day of the meeting the agenda items will be iterated over, with a list of actions added in a comment at/near the end.

Invited: Everyone.

Scorecard: #184

Time, Date, and URL

Time: 08.30 EST - 14.30 CEST - 13.30 GMT
Date: 2022-11-16

URL: https://bluejeans.com/917304424
Dial-in:
PIN:

Details

Retrospective Owner Tasks (in order):

• Post this issue's URL on the #Release Slack channel around the start of the new release.
• Copy actions from the previous retrospective into this issue, while ignoring actions that are ticked or have issue links (i.e. clearly-completed actions).
• Wait until 90% of all builds have been released, with no clear signs of additional respins.
• Announce the retrospective Slack call's date + time on #community at least one full week in advance, and send out meeting invites.
• Host the slack call for the retrospective, including:
  • Iterating over the actions from the previous retrospective issue, ticking off completed items.
  • Iterate over the agenda, ensuring everything gets debated.
  • Create a clear list of actions at the end of the retrospective, including volunteer names.
• Create a new retrospective issue for the next release.
• Set yourself a calendar reminder so that you remember to commence step 2 (in the new issue) just before the next release.
• Close this issue.

[zdtsw]

Something can be considered as improvement in the coming release when it comes to jenkins pipeline and parameters.

• skip re-generate per every commit to ci-jenkins-pipeline during release period : detail discussion: release: set aqaReference tag for 2022 Oct CPU release ci-jenkins-pipelines#464 (comment)
• add support in the weekly job , so openjdkX-pipeline can use the correct aqaReference for the dry-run
• if it is better to create dedicated release pipeline to keep a clean history of release and no need to keep log of nightly/weekly openjdkX-pipeline

[andrew-m-leonard]

jdk19u mirror merge conflict issue took some time to resolve, conflict due to our patch to .guthub/workflows/submit.yml, that was originally changed upstream, and then deleted.

Q: Given jdk11+ we basically have no patches, just README.md, should be do away with a "mirror" and build upstream repo, with a "patch apply" process if needed ? Issue maybe we don't have a SCM_REF SHA of exactly what is built? instead the SHA would be the upstream code SHA + "applied patches"..

[andrew-m-leonard]

Following on with the mirror discussion, jdk8u failed due to the issue of -b00 tags that mark upstream "fork" points for next release (jdk8u362-b00), which then causes the incorrect latest tag to be found...
This further makes me suggest we should alter our approach to having a mirror, and instead build upstream directly with "patches" files applied...?
(Discuss!)

[zdtsw]

make CUSTOMIZED_SDK_URL_CREDENTIAL_ID with default eclipse_temurin_bot_email_and_token

when run grinder for triage failed AQA tests, CUSTOMIZED_SDK_URL_CREDENTIAL_ID is not set, need manually choose and the only available choice is eclipse_temurin_bot_email_and_token, feels like can be easier to just set it as the default.

[llxia]

re #181 (comment), I think this is a known issue adoptium/aqa-tests#4049

[smlambert]

also re: #181 (comment) defaults for credentials are not straight-forward

eclipse_temurin_bot_email_and_token is the right one for the ci.adoptopenjdk.net server, different one on temurin-compliance, different one on openj9, different on alibaba server, etc. we chose not to have a default one set, but it may be that the comment relates to bug 4049, or it may be that the autogeneration of the test jobs did not set the initial config of all of them to be the correct default credential for the particular Jenkins server in question.

[zdtsw]

make TARGET in grinder job be more flex:
currently not support example: testList TESTLIST=jdk_util_0, jdk_util_1 , jdk_net
support example: testList TESTLIST=jdk_util_0,jdk_util_1,jdk_net

should add support to trim whitespaces

updates
adoptium/aqa-tests#4063

[zdtsw]

pipeline for jdk8 https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk8-pipeline/2322/ indicated it is all green
but downstream job on x64mac actually failed https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-mac-x64-temurin/178/ (build was fine, aborted in sign, not sure why)

[smlambert]

re: #181 (comment) - please see already raised issue adoptium/aqa-tests#4063

[zdtsw]

new PR and approval process during release period need to be documented somewhere.

update:
some work has been done in #185

[zdtsw]

make installer test in GH action more flexible.
detail see adoptium/installer#546

[zdtsw]

Is it possible to upload/publish binaries to github binaries repository once openjdk pipeline is done (just like the nightly builds) but make it as "pre-release", until we finish AQA+TCK then the official "request shipping" to mark it "release"?

Currently, steps after publish are blocked due to missing binaries to be downloaded from github, so for example, PR for installer cannot be tested by GH action (see comments above). docker image cannot be tested locally , with same reason.

[zdtsw]

why build and run AQA for aarch64AlpineLinux on jdk8 if it is not released? should this be skipped
same as AIX if we do not release for jdk17/19, should that be part of the work to AQA triage and TCK test?

[smlambert]

re: #181 (comment) - at the moment that on the release champion to remove before launching build pipelines. Speaks to the notion of having a separate pipeline for things that are not currently in our "official releases" table (from program plan).

[zdtsw]

Do we need to create a public issue for checklist if it is intended for the one driving release only.
Got comments and DM from community users, seems this checklist confused them more: what they should use is #180 to follow up release progress.

[tellison]

Do we need to create a public issue for checklist if it is intended for the one driving release only. Got comments and DM from community users, seems this checklist confused them more: what they should use is #180 to follow up release progress.

Can discuss in the retrospective, but I think we do wish to keep the release checklist open to all, but maybe remove it from the website banner etc. and only point people to platform progress issue (which itself could be more user friendly)

[zdtsw]

Carry over old comments from previous retro: update testenv.properties. This should be documented somewhere.
#155 (comment)
#155 (comment)

Update after 16th Nov call:
adoptium/ci-jenkins-pipelines#498

[zdtsw]

refactor_openjdk_release_tool job need to update, output confusing.
e.g https://ci.adoptopenjdk.net/job/build-scripts/job/release/job/refactor_openjdk_release_tool/5762/console
for jdk8 ppc aix

Linux on x64: Not published:
Linux on aarch64: Not published:
Linux on ppc64le: Not published:
Linux on arm: Not published:
AIX on ppc64: Not published:
Alpine on x64: Not published:
Solaris on x64: Not published:
Solaris on sparcv9: Not published:
Windows on x64: Not published
Windows on x86-32: Not published
MacOS on x64: Not Published:
Source images: Not complete:

[smlambert]

Consider wrappers around the release job and creating 1 job for Linux platforms, 1 for Windows, 1 for Mac to prepopulate the artifact regex and ease the use for a release champion. This also allows for easier finding of dry-runs (ran early / automatically as part of a release pipeline), then simply rerun the dry-run with that boolean unchecked.

Will try this work via adoptium/github-release-scripts#85

[zdtsw]

Something can be considered as improvement in the coming release when it comes to jenkins pipeline and parameters.

• skip re-generate per every commit to ci-jenkins-pipeline during release period : detail discussion: release: set aqaReference tag for 2022 Oct CPU release ci-jenkins-pipelines#464 (comment)

So I give a bit more thoughts after 16th Nov call: keep re-generate running during release can cause problem even we fix build with tag on ci-jenkins-pipeline and temurin-build repos, an extreme example can be:

1. tag v1.1.1 is applied to both ci-jenkins-pipeline and temurin-build before release day.
2. a new commit lands to ci-jenkins-pipeline which renamed scmReference to openJDKReference as input parameter to build jobs.
3. the one doing release enters correct openjdk tag jdkXXXXX in openjdkX-pipeline Web UI and v1.1.1 to ciReference and buildReference. pipeline start and trigger downstream jobs, passing down openJDKReference=jdkXXXXX
4. since downstream job is fixed to use v1.1.1 and not get value of scmReference (empty string by default), it wont check out openjdk repo with jdkXXXXX tag but HEAD (as nightly is doing)

[zdtsw]

close issue since we have gone thro all comments in 23th Nov's community call and created issues respectively