Define a Full/Complete Production Release for AdoptOpenJDK project

[smlambert]

We have a Release doc in the openjdk-build repo that tells how to run a build/test pipeline for releasing JDK/JRE builds, but it does not cover all aspects of what is released by the project, nor does it list all the activities that need to take place in order to declare the Release was successful and complete.

This issue will attempt to summarize all of the activities that occur during release week. This summarization is meant to make it clearer what improvements in automation and flow can be done in order to further simplify and harden the release process at the project.

This activity is also needed so we can achieve fully reproducible builds.

Of interest: https://maven.apache.org/plugins/maven-artifact-plugin/

[smlambert]

WIP summary table for release (will draw diagram eventually, start with simple table):

Input / Trigger	Actions Required	Output produced
Release schedule	Repo Freeze	Temporary lockdown of critical repos to avoid churn issues
"	Update website banner	Temporary banner alerting consumers of timeframes
OpenJDK repo tags	Mirror repos updated/verified	Confirm any source code repos used in build contain correct content
"	Launch openjdk-build pipelines (includes JFR builds)	JDK/JREs for signing
Successful JDK/JRE builds	Sign binaries	Signed JDK/JREs for testing
"	Launch openjdk-tests pipelines	Results summary (Jenkins/TRSS)
Results summary	Triage test failures	Thumbs-down to Publish
"	More Triage / Grinders / Investigation / Issue Resolution	Thumbs-up to Publish
Published JDK/JREs	Launch installer jobs	Installers ready to test / Need tests
"	Launch Docker image jobs	Docker images ready to test / Need tests
"	Publish release notes	Release notes available via website
Installers created/tested	Notarize Mac installers	Notarized installers published

Open questions:

• Where are we at with installer specific testing tracked via Create a smoke test for installers adoptium/installer#205? There appears to be something started for Linux installers via Add installation tests for Linux packages adoptium/installer#206, what about others?
• Testing for Docker images is gated on Adding Scanner Tool to Identify Problematic Docker Images openjdk-docker#345 and Accommodate other container base images in testing adoptium/aqa-tests#1109
• Some tests run against JFR builds, what JFR specific testing is done?
• Where to JMC builds fit into the release story (or is it standalone has its how release cadence/schedule)?
• Are there other items that get produced during release cycle that are not identified in the output column?

[smlambert]

[smlambert]

Each step in the diagram above can either be decomposed further into substeps or if its as granular as it will become, the step can be described by a bunch of metadata.

This metadata can be categorized loosely by:

• inputs (materials such as repo/branch, commands/actions, environment/machine info)
• outputs (products/archives, by-products/logs where products get passed on as inputs to a next step and by-products are merely created but not used explicitly as input to a next step). If a step is the last step in a pipeline, it creates a final product that is consumed by consumers.

Note: I've used InToto terminology to describe the metadata, shown in this diagram:

Next each component team (build/test/docker/installers/website) should decompose each step and then detail each step in their swimlane of the diagram, filling out a complete picture of the metadata (inputs/outputs) of each step.

[aahlenst]

Thanks for putting this together.

• Missing: Code freeze (~ 1 week before closest Tuesday to the 17th), disabling/enabling nightly builds (still needed?), release announcement
• In your diagram, installer jobs start after the publication. Currently, this is only the case for Linux packages. MSI/PKG run as part of the JDK production. I think it makes sense to separate JDK production and packaging, especially if an intermediate step fails. But still: Was this change intentional?
• It would be a big help if you could indicate where we have to do JCK testing.

To address some of your questions:

• JMC is on a separate schedule as far as I know.
• MSI testing: I have an idea how to solve this. I want to solve this like the Linux packages (TestContainers + Gradle/Java + some glue) and even do it beforehand (less variables/dimensions than Linux packages), but my day does not have enough hours.

[hendrikebbers]

@smlambert what software have you used to create the diagram?

[smlambert]

I am using yEd, and have attached the file to this issue if others want to edit and share directly (remove the .txt suffix it was added so the file could be attached to this issue).
AdoptiumRelease.graphml.txt

[smlambert]

re: #158 (comment) question was the change to separate JDK production and packaging, it was how I expected the pipelines to logically to look, without looking too closely at how some of them are today (as in I thought it was the case for all packagers).

We can raise issues in each of the repos that owns a swimlane, so that we can at very least get a standardized way of describing the inputs/outputs, which to me seems like a good short-form documentation for the repo. I did a very rudimentary pass of the test pipelines to create in-toto link files using this beta tool, https://in-toto.engineering.nyu.edu/ but I think even just listing everything in a common format for each step in the pipeline is a good start.

Regarding JCKs, they can be triggered as soon as there is a JDK binary to test (as this very simplistic diagram shows: https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/doc/diagrams/ciPipeline.jpg). They would get triggered to run on a remote/private Jenkins server and separate machine pool (I had previously setup a remote trigger that was able to trigger test jobs on the private server we had... one can optionally set the pipeline to wait for those remote jobs to complete, but we would have to see how many machines we would have at the project's disposal in order to know by how much we can reduce the execution time... we can run all test targets in parallel, but its a large set and execution time even when split up across several machines). As we would be holding off to publish only after completion, we should/could re-add the hard stop on the publish step (where its gated on test steps passing and which we do not enforce for nightly builds). For release builds, publishing is left as a manual step of the release manager.

[tellison]

We can raise issues in each of the repos that owns a swimlane, so that we can at very least get a standardized way of describing the inputs/outputs, which to me seems like a good short-form documentation for the repo.

So +1 to the goal of defining the input/output artefacts for many reasons, verification, testing, traceability, etc.
Since it feels like quite a large task for the entire release process, how about we focus on just one of the steps. That will get us started with figuring out how we create the metadata, where we store it, how we view it, etc.

[karianna]

Now at Adoptium

[sxa]

I'm going to reopen this one as we do NOT have this definition documented anywhere at adoptium yet.

[sxa]

Adding comment as an excuse to play with the new mermaid support in github and to update the whole flow to the current status :-)

Automated build flows:

  graph TD;
      A[Skara mirror scripts] --> B
      B[Extract and build OpenJDK source per platform]-->C
      C[Win+Mac: sign_build] -->D
      D[Run smoke tests] -->E
      D-->F
      E[Run nightly subset of AQA test jobs] --> G
      F[Weekly/Release: Run full set of AQA test jobs] --> G
      G[win/mac: create_installer] --> H
      H[win/mac: sign_installer] -->I
      I[Publish to github temurinXX-binaries repository]

For release builds the process is as per the RELEASING.md. In addition to the workflow above the builds are triggered with extra parameters to specify the tag to use and a couple of flags to set release identification parameters. Prior to the day of the release this we would also generally have done the following:

• Select a release manager
• Perform AQA triage on a weekly run prior to running the release builds so we are an in "known state" to allow us to have minimal unexpected problems during the release cycle
• Create a visible release status issue which will be linked from a banner on the adoptium web site, checklist for the release manager, and TCK execution status issues (restricted) for each release.
• Declare a lockdown on changes in critical repositories (build, containers, infrastructure installers etc.)

  graph TD;

    X[Add banner saying we're working on releases] -->A
    A[Ensure correct release tag is included in our source mirror] --> B
    B[Run build pipeline as above] --> C
    B-->D
    C[Triage/fix non-TCK AQA runs] --> E
    D[Run TCK testing] --> E
    E[Obtain per-platform ship approval from PMC in slack] --> F
    F[Run release tool to ship to github] -- Mac -->G
    F--Linux+Alpine-->H
    F--Linux-->I
    G[Update and publish to homebrew] -->J
    H[Update and publish to dockerhub]-->J
    I[Create and ship installers]-->J
    J[Declare release complete]-->K
    J-->L
    J-->M
    J-->N
    K[Remove/update website banner]
    L[Update support table and release notes]
    M[Tweet about release]
    N[Close off all release issues]

[smlambert]

Updated diagram to reflect that release notes can be generated early, announcements can happen before all platforms completed (can be after primaries and/or at a certain percentage completion), to indicate different testing that is triggered and to include post release activities such as retrospective.

[karianna]

@smlambert Should we move this to the adoptium repo?

[smlambert]

Yes, I will be doing so soon, I just wanted to upload it here for posterity. (also referencing a version of it in adoptium/installer#559).

Added a wiki here to capture it in the adoptium github org:
https://github.com/adoptium/adoptium/wiki/Adoptium-Release-Process