build-azure-win2012r2-x64-2: Intermittent make Error 127 in JDK14/J9&HS/windowsXL&windows

[M-Davies]

Platform:
jdk14/hotspot and openj9/windows and windowsXL
Architecture:
x64

The build intermittently fails at almost random places and sometimes continues to build after this error occurs. Error code 127 is not in the Jenkins Exit statuses

• First seen on Monday 26th April
  Link: https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk14u/job/jdk14u-windows-x64-hotspot/50/console on build-azure-win2012r2-x64-2.

Compiling 5 files for BUILD_VM_COMPILER_NODEINFO_PROCESSOR
make[2]: *** [make/Main.gmk:364: create-source-revision-tracker] Error 127

• Seen again on Tuesday 12th May
  Link: https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk14u/job/jdk14u-windows-x64-openj9/69/consoleFull on build-azure-win2012r2-x64-2.

LightweightNonReentrantReaderWriterLock.cpp
make[3]: *** [Gensrc-jdk.localedata.gmk:45: /cygdrive/j/jenkins/tmp/workspace/build/src/build/windows-x86_64-server-release/support/gensrc/jdk.localedata/_cldr-gensrc.marker] Error 127
ConcurrentGCIncrementalUpdate.cpp
make[2]: *** [make/Main.gmk:109: jdk.localedata-gensrc-src] Error 2

• Seen again Today on build-azure-win2012r2-x64-2
  Link: https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk14u/job/jdk14u-windows-x64-openj9-windowsXL/68/consoleFull on build-azure-win2012r2-x64-2.

Updating support/demos/image/jfc/TransparentRuler/src.zip
make[2]: *** [make/Main.gmk:326: demos-jdk] Error 127
make[2]: *** Waiting for unfinished jobs....

@sxa @andrew-m-leonard FYI

[sxa]

I'm going to reboot that machine. Not really expecting it to resolve this but it'll eliminate the possibility that it's something in the background on the machine that's causing issues.

[M-Davies]

https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk/job/jdk-windows-x64-hotspot/103/console on build-azure-win2012r2-x64-2

Compiling 224 properties into resource bundles for jdk.localedata
make[3]: *** [modules/jdk.localedata/Gensrc.gmk:42: /cygdrive/j/jenkins/tmp/workspace/build/src/build/windows-x86_64-clientANDserver-release/support/gensrc/jdk.localedata/_cldr-gensrc.marker] Error 127

Not sure if this build was kicked off before or after you restarted it

[sxa]

That looks to have been initiated 8 hours ago so yes it would have been after yesterday's reboot

[karianna]

127 is file not found, so I suspect some path/cygwin shenanigans

[M-Davies]

https://ci.adoptopenjdk.net/view/Failing%20Builds/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-windows-x86-32-hotspot/544/consoleFull (build-azure-win2012r2-x64-2)

01:01:49  make[2]: *** [make/Main.gmk:219: jdk.pack-libs] Error 127

[M-Davies]

adoptium/temurin-build#1832 was a reoccurance of this in jdk11u-windows-x64-hotspot (build-azure-win2012r2-x64-2)

00:55:36 make[2]: *** [make/Main.gmk:203: jdk.internal.vm.compiler-java] Error 127
00:55:36 make[2]: *** Waiting for unfinished jobs....

[M-Davies]

If the reason is the same as this stackoverflow instance of it occurring, it seems likely we have a corrupt directory/command somewhere on that machine. Either way, this seems to be an infra problem over build (this doesn't occur on any other machines from what I could tell), can this issue be transferred to infrastructure @karianna ?

[Willsparker]

I've had a look, unfortunately all of the build runs linked in this issue and the dupe issue are old and have been removed so I'm unable to see the logs. When this recurs, I'll look into it a bit more , and I can see what file (and/ or command) is missing :-)

[Willsparker]

Given that it's been >2 weeks now and the issue (AFAIK) hasn't reappeared, @M-Davies do you have any quarrels with me closing the issue for now?
I'm also going to rename the issue to reflect that it's seemingly specific to build-azure-win2012r2-x64-2

[M-Davies]

@Willsparker Yes this should be ok to close for now