jdk8 aarch64_alpine-linux test jobs fail due to DETECTED_JDK_VERSION mismatch from available artifact #2934
Comments
|
try to reproduce this error: |
|
run grinder:
file does exist no problem to run "java --version" => previous error of "file not exists" is node related |
|
hi @zdtsw - re: #2934, in your container, can you try unpacking a nightly/beta build (not a release build), and check the version? https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u-2022-12-31-05-54-beta/OpenJDK8U-jdk_aarch64_alpine-linux_hotspot_2022-12-31-05-54.tar.gz Wondering if it's a packaging problem, seems at least temporarily, that the artifacts labeled with '8' are actually '17'. (see also @sophia-guo 's notes on the x86_alpine-linux similar issue: adoptium/ci-jenkins-pipelines#508). |
tried with https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u-2022-12-31-05-54-beta/OpenJDK8U-jdk_aarch64_alpine-linux_hotspot_2022-12-31-05-54.tar.gz nightly build from 2 days ago, |
|
i still cannot reproduce this locally: |
|
I am looking at https://github.com/adoptium/aqa-tests/blob/master/get.sh#L221-L238 and wondering if there is some additional handling needed for the alpine-linux platforms (which do not quite follow the same naming pattern as others). |
|
This should not be related with https://github.com/adoptium/aqa-tests/blob/master/get.sh#L221-L238. Nightly builds get SDK by upstream, which doesn't need the download by API. Also by console output we can see that file is correct |
|
Thanks @sophia-guo for looking at the dates of when this issue first started (Nov 21st builds). Listing the changes in repos from 21st to 23rd that would be likely candidates of where the issue was introduced: |
|
Looking back in TRSS history, I do not think it began on Nov 21, x86-64_alpine-linux:
Listing the changes in repos from 23rd to 25th that would be likely candidates of where the issue was introduced: |
|
It might also be a machine issue or cleanws issue.
jdk8 05:34:35 Run /home/jenkins/workspace/Test_openjdk8_hs_sanity.openjdk_x86-64_alpine-linux/openjdkbinary/j2sdk-image/bin/java -version aarch64_alpine-linux Might worth to login the machine or container to see what's those special java are, are they default java ? Are those machines' configuration changed after Nov 21st? In adoptium/ci-jenkins-pipelines#508 it did mention tests jobs triggered by same jdk build have different behavior, smoke tests and sanity tests passed and then all following test jobs failed, which suggests might be machine changes at that time? |
|
As alpine is split out to its own repository we might leave and wait to see if this still happen when alpine is stable. |
|
As per adoptium/aqa-tests#4258 (comment), we are able to run these pipelines without issue in https://ci.eclipse.org/temurin-compliance/job/AQA_Test_Pipeline/988/, which seems to cast suspicion on machine configuration. Need to look at and compare configuration of public nodes versus the compliance nodes. |
|
Looking at the log from https://ci.adoptopenjdk.net/job/Grinder/6620/console running on https://ci.adoptopenjdk.net/computer/test%2Ddocker%2Dalpine317%2Dx64%2D1/ it's running |
|
Test run executed correctly when the JDK used to run the jenkins agent was switched from jdk17 to jdk8 - that should not be affecting the use of JVM under test. (I've switched it back now since we need to be running 11+) |
|
Also the TC agent that is running the tests successfully (See Grinder 3163) is using what is probably an Alpine supplied JDK11 to run the agent, which may or may not be relevant... |
|
I have double checked all the x64_alpline and aarch64_alpine tests jobs with jdk8, jdk11,jdk17, jdk19 and jdk20. This only happens to test jdk8. Feels like if jdk version is lower than 11 that even running with specified jdk it will fall back to default jenkins java agent ( if it's 17). However if the default jenkins java agent is 8 or 11 ( lower than 17) then the specified jdk will run even if it's jdk8. This is interesting. |
|
It also happens to test-docker-alpine313-aarch64-1. So that may not be related with the newer version of apline. As it happened around Nov 23rd it might help to figure out what configuration has been changed specific to all aplines jenkins agents. |
I'm not quite sure why you're focussing on Nov. 23rd other than that being the first time it was noticed. Was there a visible change on the alpine313-aarch64-1 system you mentioned at that time? We have been upgrading jenkins agents to JDK17 over the last few months as part of #2763 and this has been publicised in the infrastructure channel (most recently a couple of weeks ago) and in quite a few of our team calls. It looks like alpine313-aarch64-1 was upgraded to JDK17 four days ago so presumably it was ok before then? If so, we know that the configuration change was the switch to using JDK17 for the agent |
|
@sophia-guo At this point I'd suggest trying to start up a JDK17 on an Alpine system which runs a Process exec call of some sort to fire up a shell and see if that gets the problematic |
|
I've only noticed this sort of thing as a problem in the past when I was debugging something on AIX. |
|
sxa/aqa-tests@f15a794 fixes it so that's definitely the source of the problem. |
|
Thanks @sxa ! The grinder https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6649/console reproduce the issue - tested jdk8 print jdk17. Default java is 8 Other java |
|
Just realised I didn't include my link earlier - https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/6647/console was the grinder with the patch I mentioned above - it still didn't run through properly (I haven't diagnosed why, but bear in mind my change would have been local to |
|
x64_alpine is similar login in the machine and run directly it works fine Alpine jobs have the environment variables print out( printenv), didn't see other linux have those two environment print out: We don't have the issue in Eclipse temurin. Alpine jobs in Eclispe temurin also have those two environment variables print out. Only difference is in Eclispe temurin those variables are 11 ( Though probably jenkins version is different too). @sxa mentioned if switch back to jdk8 as jenkins agent there will be no this issue. So suppose if change to 11 we will not have this issue too? This is only happen to jenkins agent use jdk17 on alpine? |
|
We don't have the issue in Eclipse temurin
What do you mean by "In Eclipse Temurin"?
…On Tue, 14 Feb 2023, 21:13 sophia-guo, ***@***.***> wrote:
x64_alpine is similar login in the machine and run directly it works fine
8a931c282a28:~$ _testJ=/home/sophia/GrinderJDK8/Grinder/openjdkbinary/j2sdk-image/bin/java
8a931c282a28:~$ ${_testJ} -version
openjdk version "1.8.0_372-beta"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_372-beta-202302131817-b02)
OpenJDK 64-Bit Server VM (Temurin)(build 25.372-b02, mixed mode)
8a931c282a28:/$ ls -l /usr/lib/jvm/
total 4
drwxr-xr-x 9 root root 4096 Nov 23 16:59 jdk-17
Alpine jobs have the environment variables print out( printenv), didn't
see other linux have those two environment print out:
_=/usr/lib/jvm/jdk-17/bin/java
LD_LIBRARY_PATH=/usr/lib/jvm/jdk-17/lib/server:/usr/lib/jvm/jdk-17/lib:/usr/lib/jvm/jdk-17/../lib
We don't have the issue in Eclipse temurin. Alpine jobs in Eclispe temurin
also have those two environment variables print out. Only difference is in
Eclispe temurin those variables are 11 ( Though probably jenkins version is
different too).
07:43:24 LD_LIBRARY_PATH=/usr/lib/jvm/java-11-openjdk/lib/server:/usr/lib/jvm/java-11-openjdk/lib:/usr/lib/jvm/java-11-openjdk/../lib
07:43:24 _=/usr/bin/java
—
Reply to this email directly, view it on GitHub
<#2934 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/APDJLOBI4J7E6PAHA65WV5LWXPYPPANCNFSM6AAAAAAU3XY7OU>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
I believe she is referring to the Temurin-compliance Jenkins server. |
|
Yes, Eclipse Temurin --> Temurin-compliance |
It needs to be tested, but I do not want to switch back from using JDK17 for the jenkins agent. While it works, it is not recommended to have the agents running at an earlier JDK version running on the server. I suggest trying the experiment I mentioned previously - using a Runtime/Process object to start up a shell from the different Java runtimes and see if the JDKs are setting those values themselves in subprocesses and what effect they have on java versions running from there so we have all the relevant data (and possibly determine if it is a bug in how java is operating on certain versions!), and then determine how best to work around it in the AQA test suite, possibly with something based on my patch above. |
Yes, I didn't ask for switching back to jdk11 or else. Just give more information to help to narrow down the issue. |
|
Summary of behaviour:
✔️ means it properly uses TEST_JDK_HOME as the java under test |
|
when Java 8 runs with this path to 17 it opens jdk-17.0.6+10/lib/modules in Java 11, in the path, it does not, and it works. What is the correct behaviour? Either use the LD_LIBRARY_PATH or not, but not inconsistent behaviour (between jdk11 and jdk17 on LD_LIBRARY_PATH). |
|
The java executable is dynamically linked against As an additional check, here's what you get when you point LD_LIBRARY_PATH at the JDK11 |
|
Another note is libjli.so exists in JDK17 lib directory doesn't affect jdk11, jdk17. Only jdk8 is affected. Suspect the reason is the https://bugs.openjdk.org/browse/JDK-8247589 only backport to jdk11 https://bugs.openjdk.org/browse/JDK-8284005. It is not backport to JDK8 This might explain why LD_LIBRARY_PATH is set and how it is set . Might be helpful to confirm with Aleksei Voitylov to see if this is the behaviour is expected for jdk8 and if the backport could fix it? |
|
Thanks to @sophia-guo for putting Stewart's workaround in place to mitigate this. |
What are you trying to do?
Test jdk8 aarch64_alpine-linux
https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-alpine-linux-aarch64-temurin/
This fails for both weekly and nightly testing.
Expected behaviour:
To be able to pick up the jdk8 binaries and run test jobs against them.
Observed behaviour:
When pulling jdk8 binaries, and unpack them, the jdk8 binaries unpack as jdk17 binaries.
Any other comments:
Same problem seen on related: adoptium/ci-jenkins-pipelines#508
All test jobs fail consistently, and these failures tank the Test job rating (published to Slack). We do not release jdk8 aarch64-alpine-linux formally, so also believe these should be moved to the evaluation pipeline (related: adoptium/ci-jenkins-pipelines#520).
Pulling the jdk binary directly from https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u-2022-12-31-05-54-beta/OpenJDK8U-jdk_aarch64_alpine-linux_hotspot_2022-12-31-05-54.tar.gz and running a test in https://ci.adoptopenjdk.net/job/Grinder/6292 indicates that the binary is not unpacked to openjdkbinary/j2sdk-image/ folder in an expected location.
Need to determine if this issue is a problem in test or build pipeline code, but also should raise a separate issue to remove jdk8 aarch64_alpine-linux from mainstream pipelines since its not a platform/version combo that we release.
The text was updated successfully, but these errors were encountered: