Ansible request for ensuring jenkins agents are run with JDK11+ #2763

sxa · 2022-10-04T11:54:46Z

Delete as appropriate from this list:

Missing install

Details:
At the end of June Jenkins announced in a blog post that the Jenkins 2.357 and the forthcoming 2.361.1 LTS would require Java 11 or 17 on the server side (We are now running ours with Temurin 17).

With those new versions, using Java 8 for the jenkins agent systems will NOT BE SUPPORTED.

For this reason, we need to upgrade the jenkins agents to all run java 11 (or later) before we perform the next Jenkins LTS upgrade. We are currently on 2.346.3 (previous LTS) and will need to look at upgrading to 2.36.1. But before that, we need to ensure all the jenkins agents are running a suitable version.

Solaris in particular only has Java 8 on it (Temurin does not produce a later one) but there are versions of JDK11 for Solaris available from:

https://bell-sw.com/pages/downloads/
https://www.azul.com/downloads/?version=java-11-lts&package=jdk (Explicitly says Solaris 10 is supported)

Or we could try building our own JDK11, but I don't think our machine configurations supported that the last time I tried.

FYA @karianna and I'll tag @steelhead31 too since he likes playing with Solaris!

sxa · 2022-10-04T17:33:30Z

So far none of the above options are proving to be feasible for Solaris 10 due to the dependency on the Solaris 11 posix_fallocate symbol as referenced in this article.

sxa · 2022-10-13T13:32:01Z

We should aim for JDK17 on all systems (For AIX we can use a nightly build or GA candidate)

sxa · 2022-12-05T23:06:43Z

So far none of the above options are proving to be feasible for Solaris 10 due to the dependency on the Solaris 11 posix_fallocate symbol as referenced in this article.

Fixed. a JDK11 with a 'fake' posix_fallocate (Not a very common function) will allow a prebuilt java 11 to run if you set LD_PRELOAD in the environment to a trivial library created as follows (It should print a message if the function is called, but the jenkins agent does not appear to trigger it:

cat > fallocate.c << EOT
#include <stdio.h>
int posix_fallocate(int fd, off_t offset, off_t len)
{
  fprintf(stderr, "posix_fallocate() called but stubbed out\n");
}
EOT
cc -G -m64 -o fallocate.so -Kpic fallocate.c
$ LD_PRELOAD=$PWD/fallocate.so /usr/lib/jvm/zulu11.60.19-ca-jdk11.0.17-solaris/bin/java -version
openjdk version "11.0.17" 2022-10-18 LTS
OpenJDK Runtime Environment Zulu11.60+19-CA (build 11.0.17+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.60+19-CA (build 11.0.17+8-LTS, mixed mode)
bash-3.2$

A couple of extra adjustments were needed to make git work properly as it was, by default, getting the LD_PRELOAD setting from the java process. I created a small wrapper around git to squash LD_PRELOAD (since git was a 32-bit application and was trying to use the symbol) I created this:

mkdir -p /usr/local/sxabin
cat > /usr/local/sxabin/git
#!/bin/sh
/usr/local/bin/git "$@"

And set the jenkins configuration for the agent to point to Tool locations -> git to /usr/local/sxabin/git and als set /usr/local/sxabin/git first in the PATH. This appears to have worked: https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-solaris-x64-temurin/201/console

steelhead31 · 2022-12-06T09:31:14Z

This looks like a good solution to me, the only real impact is on the jenkins agent, its probably a cleaner (and less prone to errors) than running via ssh forwarding or similar.

sxa · 2022-12-06T10:49:45Z

Unfortunately I hadn't noticed that the linked job ran on the old Solaris machine (I thought it had been disabled) so this isn't fully working yet as the LD_PRELOAD is going to sub-processes and since most of them are 32-bit the library causes a failure:

ld.so.1: sh: fatal: /usr/lib/jvm/fallocate-preload.so: wrong ELF class: ELFCLASS64
Killed

sxa · 2022-12-06T10:56:55Z

Considering not persuing this and going down another route - build JDK11 on a Solaris 11 system but adjust the code so it doesn't require posix_fallocate

sxa · 2022-12-08T18:12:08Z

@ptribble's 11.0.2 works on Solaris 10/x64: https://pkgs.tribblix.org/openjdk/openjdk11.0.2-s10-x86_64.tar.bz2
sha256sum a5484bd35ed15ea7dc97870cea470aedf0c713ecca8075e57954a70e8b32cd89

ptribble · 2022-12-08T19:20:14Z

Using LD_PRELOAD_64 rather than a bare LD_PRELOAD ought to restrict it to 64-bit processes, if you want to pursue that route.

sxa · 2022-12-09T11:44:48Z

Using LD_PRELOAD_64 rather than a bare LD_PRELOAD ought to restrict it to 64-bit processes, if you want to pursue that route.

Thank you! I don't think I've used that before but it's EXACTLY what I needed and https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-solaris-x64-temurin/208/ has successfully built with the job running via a JDK11u jenkins agent on the machine.

So, for reference:

I built the shared library from the earlier comment on the machines and put it in /usr/lib/jvm
Extracted the X64 or SPARC version of Zulu under /usr/lib/jvm
Adjust the jenkins agent startup to have a JavaPath of /usr/lib/jvm/zulu11.60.19-ca-jdk11.0.17-solaris/bin/java and Prefix Start Agent Command of export LD_PRELOAD_64=/usr/lib/jvm/fallocate.so &&. Or if using a startup script use LD_PRELOAD_64=/usr/lib/jvm/fallocate-preload.so /usr/lib/jvm/zulu11.60.19-ca-jdk11.0.17-solaris/bin/java [...]

There were a few messages showing in the startup log on SPARC, but that appears to be building ok too.

FYI @speakjava - we might have a solution :-)

sxa · 2023-01-11T12:38:27Z

We still have over 50 systems running JDK8 as the agent. Thanks to @steelhead31 for collating this list. We can check these off as they are fixed. Also need to ensure that the playbooks deploy new ones with JDK11/17 available (and ideally as the default)

Windows (14)

Linux build + test (22)

Others (16)

Note that from the comments in #2847 the Solaris 10 machines will require Bellsoft Liberica 11 to be used (Azul's seems to result inhigh CPU load after a while on Solaris 10) and requires the fallocate preload mentioned in an earlier comment.

sxa · 2023-02-02T12:06:36Z

@steelhead31 @Haroon-Khel I've just adjusted the last comment to categorise the machines (Windows/Linux build+test/Other). I reckon at this point we should set an hour aside and do this. Would you be interested in doing it with an open call to discuss any issues? FYI @karianna in case you have someone who has access and would like to take on the windows subset :-)

@steelhead31 Can you easily tell how many of the ones already migrated are using a jenkins config that points to a specific java as opposed to the default version? (I'm not sure if the command lines you were getting would say just java in the latter case)

steelhead31 · 2023-02-02T12:26:12Z

@sxa / @Haroon-Khel sounds like a good idea to me, I'll produce an up to date ( and complete list ) of the current state, this afternoon in preperation for doing this.... Im not sure how easy it is to tell default java from specified, as it just shows a path.. lets see if the updated list offers any insight

sxa · 2023-02-02T12:52:21Z

I'm somewhat in two minds about whether to change the default or override in jenkins (and if we override we need to bear in mind me raising #2912 recently!) But I think an override for the remaining ones, then possibly look at adjusting the defaults and resetting the override after 2912 goes in is probably my preferred approach ...

Haroon-Khel · 2023-02-02T13:25:02Z

+1 from me on overriding. From what ive seen, the default is usually 8, so unless you guys have an objection I see no reason as to why we shouldnt override it with 17 (on platforms that have 17)

sxa · 2023-02-02T13:27:58Z

@steelhead31 I guess the table from #2879 (comment) would suggest that the ones with an "empty" Java version column are using the default on the system.

steelhead31 · 2023-02-02T13:28:07Z

New audit sheet is available here..
https://docs.google.com/spreadsheets/d/1MS4IcUSyJxOiKeQ14HN-uajstIv8hGM7Xz9hXFTmfnk/edit?usp=sharing

steelhead31 · 2023-02-02T13:31:21Z

@steelhead31 I guess the table from #2879 (comment) would suggest that the ones with an "empty" Java version column are using the default on the system.

I believe so for the online ones at least... they will be using the system default, or they may have been connected from the node itself with a different JDK from the default... I'll audit a few of them, and see if its obvious to discern those two cases.

sxa · 2023-02-02T13:31:23Z

Looks like there's plenty of precedent and no objections to tweaking the jenkins agent config so I suggest we go for that, using /usr/lib/jvm/jdk-17 where available and a suitable platform-specific alternative elsewhere.

Haroon-Khel · 2023-02-03T17:23:39Z

@sxa I'm guessing build-azure-win2012r2-x64-4-sxa is on your home network? I cant get into it

Since I deleted the jenkins service on build-alibaba-win2012r2-x64-1 and I cant get it back up, for now I have the jenkins agent running in a background process with java17.

Steps to change the java path for the jenkins service:

In the jenkins user folder (C:\Users\jenkins usually) there should be a jenkins-slave.xml file
Open it with an editor and change the executable path to your desired java path
Save and close the file
Restart the jenkins service in Services
Check the System Information page for the jenkins node to see if the update has taken place

sxa · 2023-02-03T18:09:01Z

@sxa I'm guessing build-azure-win2012r2-x64-4-sxa is on your home network? I cant get into it

No - the only ones hosted be me are the test-sxa ones. I'll check that definition as I think it's one of a set that I used when working with Andrew on reproducible builds. It probably just needs to be deleted.

sxa · 2023-02-03T18:18:15Z

Actions arising from today's activities:

Decommission (probably) build-azure-win2012r2-x64-4-sxa as per the comment above
Diagnose connection problem on build-osuosl-aix71-ppc64-2 System unavailable: build-osuosl-aix71-ppc64-2 #2922 (It's twin build-osuosl-aix71-ppc64-1 is unaffected)
Fix Bastillion for DigitalOcean CentOS system
Check vagrant.yml and dockerhost.yml to ensure they will install a suitable JDK17
Fix build-alibaba-win2012r2-x64-1 which was unable to pull the remoting.jar down (It's twin build-alibaba-win2012r2-x64-2 appears unaffected)

release
You are trying to get resource http://47.111.84.87:8080/jnlpJars/remoting.jar but it is not in cache and could not be downloaded. Attempting to continue, but you may expect failure
JAR http://47.111.84.87:8080/jnlpJars/remoting.jar not found. Continuing.
JAR http://47.111.84.87:8080/jnlpJars/remoting.jar not found. Continuing.
netx: Initialization Error: Could not initialize application. (Fatal: Initialization Error: Unknown Main-Class. Could not determine the main class for this application.)

Follow-on actions: Change all the definitions once we implement #2912 ;-)

sxa · 2023-02-23T13:16:14Z

Based on the new column from the plugin in #2950 there are five machines still running JDK8 for the agent:

steelhead31 · 2023-03-01T12:01:49Z

The final machines have all been upgraded to JDK17 for jenkins agents.

sxa added the ansible label Oct 4, 2022

sxa added this to the 2022-10 (October) milestone Oct 4, 2022

sxa added this to Adoptium October 2022 Plan Oct 4, 2022

This was referenced Oct 4, 2022

Secure the public Jenkins server #2108

Closed

Upgrade jenkins agent's JDK to 11 or newer #2771

Closed

Haroon-Khel mentioned this issue Oct 5, 2022

Dockerstatic: Upgrade default JDK on dockerstatic containers to 17 #2770

Merged

6 tasks

sxa mentioned this issue Oct 13, 2022

unixPB: Adjust boot JDKs to current required ones. Remove JDK11 #2782

Merged

6 tasks

sxa removed this from Adoptium October 2022 Plan Nov 2, 2022

sxa added this to Adoptium November 2022 Plan Nov 2, 2022

sxa modified the milestones: 2022-10 (October), 2022-11 (November) Nov 7, 2022

sxa modified the milestones: 2022-11 (November), 2022-12 (December) Dec 1, 2022

sxa mentioned this issue Dec 9, 2022

New Solaris/x64 machine is not building reliably #2847

Closed

steelhead31 self-assigned this Jan 11, 2023

smlambert moved this to Todo in Adoptium November 2022 Plan Jan 11, 2023

steelhead31 removed this from Adoptium November 2022 Plan Jan 12, 2023

steelhead31 added this to Adoptium 1Q 2023 Plan Jan 12, 2023

sxa modified the milestones: 2022-12 (December), 2023-02 (February) Jan 27, 2023

sxa added the critical label Jan 30, 2023

sxa mentioned this issue Jan 30, 2023

Sparcv9_solaris builds broken #2859

Closed

sxa mentioned this issue Feb 3, 2023

System unavailable: build-osuosl-aix71-ppc64-2 #2922

Closed

This was referenced Feb 14, 2023

jdk8 aarch64_alpine-linux test jobs fail due to DETECTED_JDK_VERSION mismatch from available artifact #2934

Closed

Install Liberica 11 on Solaris systems (for jenkins agent) #2947

Merged

This was referenced Feb 22, 2023

Install versions node plugin on the jenkins server #2950

Closed

Update jenkins to latest LTS #2961

Closed

steelhead31 moved this to In Progress in Adoptium 1Q 2023 Plan Feb 28, 2023

steelhead31 closed this as completed Mar 1, 2023

github-project-automation bot moved this from In Progress to Done in Adoptium 1Q 2023 Plan Mar 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ansible request for ensuring jenkins agents are run with JDK11+ #2763

Ansible request for ensuring jenkins agents are run with JDK11+ #2763

sxa commented Oct 4, 2022

sxa commented Oct 4, 2022 •

edited

Loading

sxa commented Oct 13, 2022

sxa commented Dec 5, 2022

steelhead31 commented Dec 6, 2022

sxa commented Dec 6, 2022 •

edited

Loading

sxa commented Dec 6, 2022

sxa commented Dec 8, 2022

ptribble commented Dec 8, 2022

sxa commented Dec 9, 2022 •

edited

Loading

sxa commented Jan 11, 2023 •

edited by Haroon-Khel

Loading

sxa commented Feb 2, 2023 •

edited

Loading

steelhead31 commented Feb 2, 2023

sxa commented Feb 2, 2023

Haroon-Khel commented Feb 2, 2023

sxa commented Feb 2, 2023

steelhead31 commented Feb 2, 2023

steelhead31 commented Feb 2, 2023

sxa commented Feb 2, 2023

Haroon-Khel commented Feb 3, 2023 •

edited

Loading

sxa commented Feb 3, 2023

sxa commented Feb 3, 2023

sxa commented Feb 23, 2023 •

edited by steelhead31

Loading

steelhead31 commented Mar 1, 2023

Ansible request for ensuring jenkins agents are run with JDK11+ #2763

Ansible request for ensuring jenkins agents are run with JDK11+ #2763

Comments

sxa commented Oct 4, 2022

sxa commented Oct 4, 2022 • edited Loading

sxa commented Oct 13, 2022

sxa commented Dec 5, 2022

steelhead31 commented Dec 6, 2022

sxa commented Dec 6, 2022 • edited Loading

sxa commented Dec 6, 2022

sxa commented Dec 8, 2022

ptribble commented Dec 8, 2022

sxa commented Dec 9, 2022 • edited Loading

sxa commented Jan 11, 2023 • edited by Haroon-Khel Loading

Windows (14)

Linux build + test (22)

Others (16)

sxa commented Feb 2, 2023 • edited Loading

steelhead31 commented Feb 2, 2023

sxa commented Feb 2, 2023

Haroon-Khel commented Feb 2, 2023

sxa commented Feb 2, 2023

steelhead31 commented Feb 2, 2023

steelhead31 commented Feb 2, 2023

sxa commented Feb 2, 2023

Haroon-Khel commented Feb 3, 2023 • edited Loading

sxa commented Feb 3, 2023

sxa commented Feb 3, 2023

sxa commented Feb 23, 2023 • edited by steelhead31 Loading

steelhead31 commented Mar 1, 2023

sxa commented Oct 4, 2022 •

edited

Loading

sxa commented Dec 6, 2022 •

edited

Loading

sxa commented Dec 9, 2022 •

edited

Loading

sxa commented Jan 11, 2023 •

edited by Haroon-Khel

Loading

sxa commented Feb 2, 2023 •

edited

Loading

Haroon-Khel commented Feb 3, 2023 •

edited

Loading

sxa commented Feb 23, 2023 •

edited by steelhead31

Loading