Windows machines can't delete workspace files

[M-Davies]

Windows machines are seeing errors where they struggle to remove a files at various locations. All versions (from what I can see) are affected by this issue and the error message is different for each one but all revolve around failing to delete a file (albeit, there is a lot of spam from other clean up tasks so I can't be sure of that).

• https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk8u/job/jdk8u-windows-x64-hotspot/222/console build-azure-win2012r2-x64-2

12:16:24  rm: cannot remove '/cygdrive/j/jenkins/tmp/workspace/./build//src/build/windows-x86_64-server-release/support/native/java.desktop/libawt': Directory not empty
12:16:26  rm: cannot remove '/cygdrive/j/jenkins/tmp/workspace/./build//src/make': Device or resource busy

• https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk11u/job/jdk11u-windows-x64-hotspot/257/console build-softlayer-win2012r2-x64-2

12:19:08  warning: failed to remove build/windows-x86_64-normal-clientANDserver-release/make-support/failure-logs: Directory not empty

• https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk15u/job/jdk15u-windows-x64-hotspot/176/console build-softlayer-win2012r2-x64-1

12:17:39  rm: cannot remove '/cygdrive/c/workspace/openjdk-build/workspace/./build//src/build/windows-x86_64-server-release/make-support/failure-logs': Directory not empty

• https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk/job/jdk-windows-x64-hotspot/209/console build-azure-win2012r2-x64-2

12:16:47  stderr: warning: failed to remove workspace/build/src/build/windows-x86_64-server-release/make-support/failure-logs: Directory not empty
12:16:47  warning: failed to remove workspace/build/src/build/windows-x86_64-server-release/support/native/java.desktop/libfreetype: Directory not empty
12:16:47  warning: failed to remove workspace/build/src/make: Device or resource busy

[Willsparker]

So, Directory not empty is output from rm when you try and delete a directory that isn't empty (obviously). However, we do appear to be doing this in PrepareWorkspace.sh:

rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]:?}/${BUILD_CONFIG[WORKING_DIR]}/${BUILD_CONFIG[OPENJDK_SOURCE_DIR]}"

In addition, on the build-softlayer-win2012r2-x64-2 (within both the cygwin terminal and cmd), I was about to create a directory, populate it and then delete it with rm -rf with no issues. This makes me think that the error message is false reporting what's going on here.

(While I'm not sure this is exactly the same sort of issue, ref #1396 )

[M-Davies]

Recurrences in pr tester run on build-azure-win2012r2-x64-1: https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk8u/job/jdk8u-windows-x64-hotspot/402/console and https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk11u/job/jdk11u-windows-x64-hotspot/441/console and https://ci.adoptopenjdk.net/job/build-scripts-pr-tester/job/build-test/job/jobs/job/jdk15u/job/jdk15u-windows-x64-hotspot/356/console

[Willsparker]

It looks like the 2 later ones failed due to a Jenkins git issue first, followed by the cleaning workspace error.
I don't need to manually clear up the workspace however, as it appears that it was able to eventually as the workspace is not a clone of the openjdk-jdk workspace. No stray java processes are on it either ...

[sxa]

I have implemented @Willsparker's suggestion (including the commands to kill the processes) into maketest.sh at https://github.com/sxa/openjdk-tests/commits/win_process_killer ... Testing at https://ci.adoptopenjdk.net/job/Grinder/4746/console to see if the code causes any problems. I've also added the logic to check to check , but not kill, rogue processes on UNIX (Copied from SXA-processCheck).

NOTE: As of yesterday the SXA-processCheck job now has an extra (non-default) option to issue kill -KILL if the regular kill doesn't work, as we were seeing a significant build-up of java processes on some machines that were causing tests to get stuck and time out

[sxa]

SXA-processcheck now modified to show in red (failed) if it is running on a windows box that has more than one java process running (i.e. something other than the agent) but since it's not easy to detect which one is the agent I'm leaving them to be cleared up manually for now, but the job provides a good visual alert

[sxa]

Note that as a follow-on to the changes in adoptium/aqa-tests#2059 we have update it to run within the groovy code in order to be able to clean up when jobs are manually aborted or hit their timeout - these changes are in adoptium/aqa-tests#2121