EPP 2024-06 RC1

[jonahgraham]

The EPP Release Process

This guide contains the step-by-step process to complete an EPP release.

The individual releases are tracked with endgame issues on GitHub issues.
For each release (M1, M2, M3, RC1, RC2) an endgame ticket is created with the appropriate contents from the rest of this document:

EPP releases happen for each milestone and release candidate according to the Eclipse Simultaneous Release Plan.

Steps for all Milestones and RCs:

• Check for bad links to Bugzilla (other things?) especially in epp.website.xml
• Make sure any outstanding reviews are progressing - e.g. create progress review, get PMC approval, etc.
  • Annual progress review is normally done to line up with -06 release, e.g. 1 June
• Ensure that the CI build is green. Resolving non-green builds will require tracking down problems and incompatibilities across all Eclipse participating projects. cross-project-issues-dev mailing list is a good place to start when tracking such problems.
• Check that packages containing incubating projects have that information reflected in Help -> About dialog. See near the end of build output for report of check-incubating.sh script.
  • -incubation and (includes Incubating components) are not used in packageMetaData anymore (See Bug 564214)
• On RC1 check "new and noteworthy" version numbers - If any N&N are out of date, remove the N&N entries and notify the corresponding package maintainer.
  • Search for url= (notice the blank before url) in epp.website.xml to see which ones are contained in the different packages.
  • Remember that some of the features will release new versions together with the new Eclipse release. Therefore using the currently released version number may be wrong. Instead lookup the feature version to be released with the release train.
• Synchronize the following - Remember to check branch, these links are to master, but around RC2 master may be setup for next release already
  • Synchronize any changes to platform.product into all the epp.product files.
  • Synchronize any changes to platform.p2.inf into all the *.product/p2.inf files.
  • Synchronize any changes to platform's icons' into icons root directory.
• Update name of the release in strings with a "smart" global find&replace. Be careful on M3 that the replace did not match the Eclipse project name M2E! See this gerrit for an example. Use commit message like [releng] Prepare repo for 2020-12 M1. In particular, check:
  • TODO can this be automated On M1 add the M1 qualifier (e.g. 2021-03-R -> 2021-06-M1, on RC2 set it to R the qualifier e.g. 2021-03-RC1 -> 2021-03-R). Except for eclipse.simultaneous.release.name which should go from 2021-03 (4.19.0) -> 2021-06 M1 (4.20.0 M1) on M1 and 2021-03 RC1 (4.19.0 RC1) -> 2021-03 (4.19.0) on RC2
  • packages/*/epp.website.xml for product name= line
  • RELEASE_NAME, RELEASE_MILESTONE, RELEASE_DIR, SIMREL_REPO Variables in parent pom releng/org.eclipse.epp.config/parent/pom.xml
    • SIMREL_REPO should be updated to the URL published in the email to cross-project-issues announcing SimRel repo is ready for EPP build
  • TODO can this part below be automated
    • See comment in the pom.xml file around eclipse.simultaneous.release.name
    • On R build, for eclipse.simultaneous.release.name remove qualifier i.e. it should be 2020-12 (4.18.0)
    • On M1 build add the qualifier back in, for eclipse.simultaneous.release.name remove qualifier i.e. it should be 2020-12 M1 (4.18.0 M1)
  • TODO can this be automated on release builds release.xml template in releng/org.eclipse.epp.config/tools/promote-a-build.sh needs updating
• Update the Last Recorded +1 in the email template which any package and platform +1s that have been received since the last update.
• Wait for announcement that the staging repo is ready on cross-project-issues-dev. An example announcement.
  • Update SIMREL_REPO in releng/org.eclipse.epp.config/parent/pom.xml if not done above.
• Update the build qualifiers to ensure that packages are all updated. See this gerrit for an example. To do this run releng/org.eclipse.epp.config/tools/setGitDate (link) script. This script will make a local commit you need to push.
  • In some cases a respin/rebuild is needed and setGitDate needs to be run again. In that case you may need to manually add a minute or two to the applied timestamp in the script.
• Run a CI build that includes the above changes.
  • If the build fails there may be the opportunity to continue the build rather than restart it. This is relatively underused option but enabled by the multi-step Jenkins build in the Jenkinsfile. For example, running the build with the previously successful steps commented out can produce a build.
• Disable the CI build so that the build results are not overwritten while doing the promotion. You can disable the project once it has fully started running, you don't have to wait for the build to finish.
• Check that there are no warnings in the console output. Especially look for warnings about failure to sign.
  • If warnings about signings occur that leave the dmg unsigned and the build does not fail, please reopen Bug 567916
• Run the Notarize MacOSX Downloads CI job to notarize DMG packages on download.eclipse.org. This can be done after promotion if time is tight or the notarization fails repeatedly. See Bug 571669 for an example of failures.
• Check the build script output to make sure that the curl calls were successful (e.g. no curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2) messages) - If there is an error like the above the .dmg file that is copied to download.eclipse.org is corrupt. Run notarize-prepare-to-redo to rename the -signed file back to -tonotarize and then re-run the notarize job
• Other notes about notarization
  • NOTE It seems perfectly normal that the notarize job needs to be run multiple times as so many notarization attempts fail due to 500 and 000 response codes from the notarization server. See Bug 571669
  • NOTE Sometimes the notarization server has an error that causes a failure that requires Webmaster support. Error looks like "an existing transporter instance is currently uploading this package". To resolve request assistance in Bug 573875 (like what was done in Comment 11 of that bug). (TODO it may be possible to workaround this error by always using a different random ID when doing the notarization.)
• Sanity check the build for the following:
  • Download a package from the build's staging output
  • Made sure filenames contain expected build name and milestone, e.g. 2020-03-M2
  • Splash screen says expected release name (with no milestone), e.g. 2020-03
  • Help -> About says expected build name and milestone, e.g. 2020-03-M2
  • org.eclipse.epp.package.* features and bundles have the timestamp of the forced qualifier update or later
  • Upgrade from previous release works. To test the upgrade an equivalent to the simrel release composite site needs to done. Add the following software sites to available software, check for updates and then make sure stuff works. In particular check error log and that core features (Such as JDT, Platform) have been upgraded.
    • https://download.eclipse.org/staging/2024-06/ - NOTE Use SIMREL_REPO if the staging repo has been updated since the SIMREL_REPO location was created.
    • https://download.eclipse.org/technology/epp/staging/repository/
    • https://download.eclipse.org/justj/epp/milestone/latest - This is needed when there is a new version of JustJ that is not also published as a release. For example, Java 21 between 2024-06 M1 and release date of 2024-06.
  • Verify no non-EPP content is in the p2 repo (especially justj, update remove-justj-from-p2.xml if needed)
• Edit the Jenkins build
  • Mark build as Keep forever
  • Edit Jenkins Build Information and name it (e.g. 2020-03 M3)
• In the evening Ottawa time, run the Promote a Build CI job to prepare build artifacts and copy them to download.eclipse.org
  • Optional - useful when testing changes to the promotion scripts: Run the build once in DRY_RUN mode to ensure that the output is correct before it is copied to download.eclipse.org.
  • This is done late in the day to try and reduce impact of adding dozens of GB on the download server and having all the mirrors start to pick it up right away. See epp-dev emails that led to this decision.
  • The DRY_RUN can be done earlier in the day and is a good way to increase the chance that the final promotion step will be successful.
• Run the Notarize MacOSX Downloads CI job to notarize DMG packages on download.eclipse.org if the promoted build was unstable
• Update SIMREL_REPO to the staging repo so CI builds run against CI of SimRel (e.g. see this gerrit)
• Re-enable the CI build
• Send email to epp-dev to request package maintainers test it. The email is templated in email.txt in the release directory.
• Archive old milestones/RCs so that they don't accumulate on the mirrors
• 24 Hours before Final release Make sure files are in final location to allow downloads to mirror
  • Tag the release, e.g. with 2020-03_R. Example command line: git tag --annotate 2020-03_R -m"2020-03 Release" 1b7a1c1af156e3ac57768b87be258cd22b49456b
  • rename the provisional release milestone to final directory (E.g. 2020-09/202009101200 -> 2020-09/R (to match what is in release.xml) - this only applies to downloads, not to packages
    This can be done with a script like TODO: make a job for this :

ssh genie.packaging@projects-storage.eclipse.org /bin/bash << EOF
  set -u # run with unset flag error so that missing parameters cause build failure
  set -e # error out on any failed commands
  set -x # echo all commands used for debugging purposes
  mv -v /home/data/httpd/download.eclipse.org/technology/epp/downloads/release/2021-03/202103121200 /home/data/httpd/download.eclipse.org/technology/epp/downloads/release/2021-03/R
  touch /home/data/httpd/download.eclipse.org/technology/epp/downloads/release/2021-03/R/*
EOF

• On release days approximately 9:30am
  • Add Versions in Bugzilla
  • check that this worked: copy the composite*RC1.jar files over the composite*.jar files in https://download.eclipse.org/technology/epp/packages/2021-03/ - this is done automatically with the EPP Make Visible job which is automatically triggered by simrel's SimRel Make Visible Job
• On Final release day immediately after a release (just after 10am):
  • The current release needs to be promoted as "latest" under https://download.eclipse.org/technology/epp/packages/latest/ . This should be a composite pointing to specific https://download.eclipse.org/technology/epp/packages/yyyy-MM/ . This is achieved by triggering the epp-promoteReleaseToLatest.
  • The next release sub-directory needs to be created immediately after a release, i.e. when 2019-12 was released, a directory 2020-03 had been created with an empty p2 composite repository pointing to 2019-12 until M1. (Use Job https://ci.eclipse.org/packaging/job/epp-createNextRelease/) On M1 release day this changes to a composite p2 repository with M1 content. On other release days, add the new releases as children and on final release this changes to a composite with just the one child.

[jonahgraham]

We have a regression in terms of warnings in the build:

[2024-05-30T08:12:54.147Z] [WARNING] Problems resolving provisioning plan.:
[2024-05-30T08:12:54.147Z]    Unable to satisfy dependency from org.eclipse.ecf.filetransfer.httpclient5.feature.feature.group 1.1.702.v20231114-1017 to org.eclipse.equinox.p2.iu; org.eclipse.ecf.provider.filetransfer.httpclient5.win32 [1.1.0.v20230423-0417,1.1.0.v20230423-0417], filter=(osgi.os=win32).
[2024-05-30T08:12:54.147Z]    Unable to satisfy dependency from org.eclipse.ecf.filetransfer.httpclient5.feature.feature.group 1.1.702.v20231114-1017 to org.eclipse.equinox.p2.iu; org.apache.httpcomponents.client5.httpclient5-win [5.2.1.v20230802-0847,5.2.1.v20230802-0847], filter=(osgi.os=win32).

and

[2024-05-30T08:12:58.354Z] [INFO] --- tycho-p2-repository:4.0.8-SNAPSHOT:assemble-repository (default-assemble-repository) @ epp.package.scout ---
[2024-05-30T08:13:04.888Z] [WARNING] Mirror tool: Problems resolving provisioning plan.:
[2024-05-30T08:13:04.888Z]    Unable to satisfy dependency from org.eclipse.ecf.filetransfer.httpclient5.feature.feature.group 1.1.702.v20231114-1017 to org.eclipse.equinox.p2.iu; org.eclipse.ecf.provider.filetransfer.httpclient5.win32 [1.1.0.v20230423-0417,1.1.0.v20230423-0417], filter=(osgi.os=win32).
[2024-05-30T08:13:04.888Z]    Unable to satisfy dependency from org.eclipse.ecf.filetransfer.httpclient5.feature.feature.group 1.1.702.v20231114-1017 to org.eclipse.equinox.p2.iu; org.apache.httpcomponents.client5.httpclient5-win [5.2.1.v20230802-0847,5.2.1.v20230802-0847], filter=(osgi.os=win32).
[2024-05-30T08:13:43.543Z] [WARNING] More information on the preceding warning(s) can be found here:
[2024-05-30T08:13:43.543Z] [WARNING] - https://wiki.eclipse.org/Tycho_Messages_Explained#Mirror_tool

appear multiple times.

These appear in the RC1 build - they started in the nightly builds on 23 May - https://ci.eclipse.org/packaging/job/epp/job/master/61/.

These are the simrel changes on that day: Commits on May 23, 2024
DLTK 6.4.1 for 2024-06
Contribute Acceleo 3.7.16 to Eclipse 2024-06
Wild Web Developer 1.3.5

I'll need to investigate more fully to understand the effect - but @merks if this seems familiar to you please let me know.

[merks]

Yes, I saw those too. But given these doesn't end up in any EPP repository anyway, I wasn't concerned, just annoyed...

[merks]

I guess it didn't start with Tycho 4.0.8?

[jonahgraham]

I guess it didn't start with Tycho 4.0.8?

It may have - we have been using snapshots of 4.0.8 (fix for that in #170), so it may have changed between builds on 22nd and 23 may, here are those changes

Commits on May 23, 2024
Fix wrong (bnd resource) path is used under windows
Improve SignRepositoryArtifactsMojo handling of unsigned content
Set the MIRROR_PARSE_ERROR_LEVEL to info severity

[jonahgraham]

Lets continue warning discussions in #171 since that doesn't seem to be a blocker for this issue.

[jonahgraham]

RC1 done - onto RC2 in #173