Weekly release 2.352/2.353/2.354 is failing

[dduportal]

The weekly release 2.352, planned for the 6th of June 2022 failed.

Given that they were multiple causes, this issue is to track these causes and associated changes to share knowledge and for audit purposes.

[dduportal]

First issue was the "global weekly release job (defined by https://github.com/jenkins-infra/release/blob/master/Jenkinsfile.d/core/weekly) that failed to allocate a pod agent.

Root cause was jenkins-infra/kubernetes-management#2468 (Jenkins URL defined in the configuration was changed to map with what was done in infra.ci but DNS configuration are different between clusters).
Fix done in jenkins-infra/kubernetes-management#2473

[dduportal]

Then, the "release" job (defined by https://github.com/jenkins-infra/release/blob/master/Jenkinsfile.d/core/release) was also failing to start a pod agent.

Root cause is https://github.com/jenkins-infra/docker-packaging/pull/36/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R96 : the JAVA_HOME for JDK11 (used to start the agent process) was changed while removing the JDK8 from the image.
Fix done in jenkins-infra/release#238 by removing the explicit path to the java binary for agent startup: relying on the PATH instead.

[dduportal]

Then, we had an issue when preparing the signing system with the following kind of error message, during the release job execution:

+ openssl pkcs12 <redacted>
Error outputting keys and certificates
40270330917F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()

Root cause is https://github.com/jenkins-infra/docker-packaging/pull/33/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R7 which changes the Docker image Ubuntu version from 18.04 to 22.04.
Fix (short term) done in jenkins-infra/release#239 to use the legacy cipher.

[dduportal]

Some preliminary work was done before the release by @timja in jenkinsci/packaging#317 that caught the following elements before the failing release:

• Missing gnupg2 binary - Add gnupg2 required for rpm signing on Ubuntu 22.04 docker-packaging#37
• Missing java macros - Add Java packaging macros docker-packaging#34

[dduportal]

Next release pipeline failed during the "Push Commits" stage while the script release.bash was trying to get last commits to reconcile with its local git tree.

Notified and explained by @basil in jenkins-infra/docker-packaging#33 (comment)

• Root cause is Switch to Ubuntu 22.04 docker-packaging#33 has the ubuntu upgrade from 20.04 to 22.04 did upgrade the git version to 2.34.1 with a new (breaking) behavior around default git pull behavior when there is a reconciliation
• Fixed in Set git divergent branch strategy to 'merge' docker-packaging#39 by setting up the default behavior explictly, same as it was before

[dduportal]

Then, a set of Maven issues happenned.

First of all, thanks @basil for https://github.com/jenkins-infra/release/pull/240/files that helped (and will help) to diagnose \o/

Issue was related with the Javadoc, not generated as expected due to the behavioral change after the JDK8 -> JDK11 upgrade

• Make architecture independent docker-packaging#36
• Generate resources while staging release release#242
• generate-resources before generating javadoc jenkinsci/jenkins#6628
• Revert generate-resources / compile release#244

[dduportal]

Next issue occured during the packaging step:

• Notified by @basil in Upgrade to Ubuntu 22.04 build container jenkinsci/packaging#317 (comment)
• Root cause: Despite Redhat Packaging Process: switch to createrepo_c (and bump version) release#202, it appears that the "createrepo" commands are not executed withint the docker image used by the packaging, but remotely on the "pkg.jenkins.io" virtual machine running on Ubuntu 18.04. Since we changed the release scripts to use the new createrepo_c, which only exists on Ubuntu 22.04, then we had to fix this issue.
• Short term: Switch back to createrepo remotely jenkinsci/packaging#318 (switch back to createrepo on the remote commands)
• Long term:
  • Ability to generate the RPM repository inside the packaging container instead on the remote pkg VM release#245 to check if we could generate the RPM repositories within the "packaging" docker image and only copy files to the remote machine. Would reduce coupling from the remote VM (and would allow this machine to be migrated/changed.
  • Ubuntu 22.04 upgrade campaign #2982 => Ubuntu 22.04 upgrade campaign

[dduportal]

• The release was finalized as 2.354 (burning the 2.352 / 2.353 versions). Confirmed in the IRC channel #jenkins-release
• Works well for infra.ci and
  • https://github.com/jenkins-infra/docker-jenkins-weekly/releases/tag/0.61.0-2.354
  • https://github.com/jenkins-infra/kubernetes-management/pull/2483/files

[dduportal]

Opened jenkins-infra/release#248 to backport on the LTS line