node-test-commit-aix jobs failing to git clone

[richardlau]

https://ci.nodejs.org/job/node-test-commit-aix/ jobs have been failing today, all on test-ibm-aix72-ppc64_be-1.

e.g. https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-power9/61999/console

05:20:48  > git rev-parse --resolve-git-dir /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-power9/.git # timeout=30
05:20:48 ERROR: Workspace has a .git repository, but it appears to be corrupt.
05:20:48 hudson.plugins.git.GitException: Error performing git command: git rev-parse --resolve-git-dir /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-power9/.git
05:20:48 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2864)
...
05:20:48 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-power9"): error=0, Failed to exec spawn helper: pid: 31654226, exit value: 127
05:20:48 	at java.base/java.lang.ProcessBuilder.start(Unknown Source)
05:20:48 	at java.base/java.lang.ProcessBuilder.start(Unknown Source)
05:20:48 	at Jenkins v2.541.3//hudson.Proc$LocalProc.<init>(Proc.java:252)
05:20:48 	at Jenkins v2.541.3//hudson.Proc$LocalProc.<init>(Proc.java:221)
05:20:48 	at Jenkins v2.541.3//hudson.Launcher$LocalLauncher.launch(Launcher.java:995)
05:20:48 	at Jenkins v2.541.3//hudson.Launcher$ProcStarter.start(Launcher.java:507)
05:20:48 	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2841)
05:20:48 	... 21 more
05:20:48 Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 31654226, exit value: 127
05:20:48 	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
05:20:48 	at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
05:20:48 	at java.base/java.lang.ProcessImpl.start(Unknown Source)
05:20:48 	... 28 more

[richardlau]

I tried to restart the agent but it's failing to restart:

bash-5.3# cat /home/iojs/jenkins_console.log
Error: dl failure on line 591
Error: failed /opt/jdk-17.0.18+8-jre/lib/server/libjvm.so, because Could not load module /opt/jdk-17.0.18+8-jre/lib/server/libjvm.so.
System error: Exec format error
bash-5.3#

cc @abmusse

[richardlau]

FWIW the .so in question is 0 bytes.

bash-5.3# ls -al /opt/jdk-17.0.18+8-jre/lib/server/
total 4
drwxr-xr-x 2 root staff  256 Feb 26 00:10 .
drwxr-xr-x 9 root staff 4096 Feb 26 00:10 ..
-rw-r--r-- 1 root staff    0 Feb 26 00:10 libjsig.so
-rw-r--r-- 1 root staff    0 Feb 26 00:10 libjvm.so
bash-5.3#

I've run the playbook from main against the machine. Kept getting the above Java error until I removed /opt/jdk-17.0.18+8-jre even though the playbook was succeeding, including the unarchive task that unpack the tarball. Maybe there is some sort of optimization in Ansible's unarchive that results in the files not being overwritten?

[richardlau]

https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix72-power9/62013/console passed the git clone/checkout but failed (maybe an issue with the PR being tested?)

13:44:22 Building addon in /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-power9/test/ffi/fixture_library
13:44:22 Unexpected error when building addon in /home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-power9/test/ffi/fixture_library. Error: [Errno 26] Text file busy: '/home/iojs/build/workspace/node-test-commit-aix/nodes/aix72-power9/out/Release/node'
13:44:22 gmake[1]: *** [Makefile:558: test/ffi/.buildstamp] Error 1
13:44:22 gmake[1]: *** Waiting for unfinished jobs....

Running https://ci.nodejs.org/job/node-test-commit-aix/62014/nodes=aix72-power9/console on the machine building main.

[richardlau]

Running https://ci.nodejs.org/job/node-test-commit-aix/62014/nodes=aix72-power9/console on the machine building main.

This one is unstable -- known flakes so within the realms of working as expected.