System unavailable: test-azure-win2012r2-x64-1: ERROR: Cannot delete workspace

[lumpfish]

Jobs are failing with:

Running on test-azure-win2012r2-x64-1 in C:\Users\jenkins\workspace\Grinder
[Pipeline] {
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: Cannot delete workspace :Unable to delete 'C:\Users\jenkins\workspace\Grinder'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 6 additional exceptions)

e.g. https://ci.adoptopenjdk.net/job/Grinder/804/console

[sophia-guo]

adoptium/aqa-tests#2637 (comment)

[sophia-guo]

This may be related with test cases. Job was cancelled by timeout and tests thread hangs.

[lumpfish]

This will stop some hanging tests from running: adoptium/aqa-tests#2639

[sophia-guo]

Same issue with test-ibmcloud-win2012r2-x64-1 , which failed job https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/508/

adoptium/aqa-tests#2639 will stop new hanging happen. We will need some process to clean all window workspace so the error won't happen again and no job will fail for this.

[smlambert]

Was hoping that the processCheck (see adoptium/aqa-tests#2059) would cover us for being able to delete workspace, wonder if it means that not all processes are being killed, so therefore can not delete workspace?

[sophia-guo]

The kill process is in try catch finally block https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/JenkinsfileBase#L736. My guess is if job terminates due to timeout the kill process will not run at all.

[Haroon-Khel]

The 3 machines, test-azure-win2012r2-x64-1 and -3, and test-ibmcloud-win2012r2-x64-1 have hanging processes which I can't delete as the jenkins user. The processes are

/Users/jenkins/workspace/Test_openjdk8_dragonwell_extended.openjdk_x86-64_windows_testList_0/openjdkbinary/j2sdk-image/bin/java

/Users/jenkins/workspace/Test_openjdk11_j9_extended.functional_x86-64_windows/openjdkbinary/j2sdk-image/bin/java

/e/workspace/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/openjdkbinary/j2sdk-image/bin/jdb

respectively.

I get a permissions error when trying to delete them as the Jenkins user

[Haroon-Khel]

I was able to kill the process running on test-ibmcloud-win2012r2-x64-1

Running the failed jenkins job on the machine
https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/512/console

[Haroon-Khel]

The hanging processes on the azure machines were ended too

[Haroon-Khel]

Running

https://ci.adoptopenjdk.net/job/Test_openjdk8_dragonwell_extended.openjdk_x86-64_windows_testList_0/9/console on test-azure-win2012r2-x64-1

and

https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.functional_x86-64_windows/140/console on test-azure-win2012r2-x64-3

[Haroon-Khel]

https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.functional_x86-64_windows/140/console failed on test-azure-win2012r2-x64-3 because it couldn't delete the workspace, strange

ERROR: Cannot delete workspace :Unable to delete 'C:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 60 additional exceptions)
[Pipeline] }
[Pipeline] // timeout
[Pipeline] echo
Exception: hudson.AbortException: Cannot delete workspace: Unable to delete 'C:\Users\jenkins\workspace\Test_openjdk11_j9_extended.functional_x86-64_windows'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. (Discarded 60 additional exceptions)

[Haroon-Khel]

On test-azure-win2012r2-x64-3, I've deleted the problem workspace, ended the hanging process and kicked off another job
https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.functional_x86-64_windows/141/console

[Haroon-Khel]

The job on test-ibmcloud-win2012r2-x64-1 ran without a workspace error or hanging processes. Running another job on the same machine to observe any 'failure to delete workspace' errors

https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/513/console

[Haroon-Khel]

test-ibmcloud-win2012r2-x64-1 looks to be fine. Two sanity jobs ran back to back without a workspace complaint or a hanging process

[Haroon-Khel]

Test_openjdk8_dragonwell_extended.openjdk_x86-64_windows_testList_0 hung twice on test-azure-win2012r2-x64-1. this being the most recent. I ran extended openjdk tests on a dragonwell jdk on the machine directly, and this ran to completion without hanging.

I ran https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk8_hs_extended.openjdk_x86-64_windows_testList_0/12/console on the same machine to test. This ran to completion without hanging

[Haroon-Khel]

The other aborted Test_openjdk8_dragonwell_extended.openjdk_x86-64_windows_testList_0 job on test-azure-win2012r2-x64-1 was https://ci.adoptopenjdk.net/job/Test_openjdk8_dragonwell_extended.openjdk_x86-64_windows_testList_0/9

Both jobs hung for 4 hours during hotspot_jre_0 after which I had to abort

[Haroon-Khel]

hotspot_jre_0 grinder from yesterday on test-azure-win2012r2-x64-1, https://ci.adoptopenjdk.net/job/Grinder/1159/console. Jenkins aborted it a after it hung for 10 hours

[Haroon-Khel]

Rerunning on the same machine without a dragonwell build
https://ci.adoptopenjdk.net/job/Grinder/1161/console

[Haroon-Khel]

Rerunning on the same machine without a dragonwell build
https://ci.adoptopenjdk.net/job/Grinder/1161/console

This ran to completion, didn't hang nor did it leave any hanging processes on the machine

[Haroon-Khel]

hotspot_jre_0 grinder from yesterday on test-azure-win2012r2-x64-1, https://ci.adoptopenjdk.net/job/Grinder/1159/console. Jenkins aborted it a after it hung for 10 hours

Same grinder ran on test-ibmcloud-win2012r2-x64-1, https://ci.adoptopenjdk.net/job/Grinder/1160/console. The test didn't pass, but the grinder did not hang as it does on test-azure-win2012r2-x64-1

[Haroon-Khel]

Re ran the job on test-azure-win2012r2-x64-3 https://ci.adoptopenjdk.net/job/Grinder/1162/console. Test doesn't hang. Seems to be a problem with -1 only

[Haroon-Khel]

I think the test class that causes the hang is TestNoMinidumpAtFullGC. It appears as the last test before the job hangs, on all 3 occasions. Despite the directory being hotspot\test\serviceability\sa\jmap-minidump\TestNoMinidumpAtFullGC.java I can't find any mention of this test file in https://github.com/adoptium/jdk8u

[sxa]

Machine currently offline due to low disk space

[sophia-guo]

test-azure-win2012r2-x64-1

https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/647/console

[sxa]

test-azure-win2012r2-x64-1
https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/647/console

There was a bash shell sitting in that directory running as the administrative user, which was preventing jenkins from deleting it. Now resolved and it's running through ok at https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_windows/651/console

[sxa]

As mentioned above the machine is quite low on disk space with 1Gb in C: (Although there is around 26Gb on the workspace drive D:

[Haroon-Khel]

Machine is back online with enough space. Closing