Flaky build failures on Windows: PermissionError: [WinError 5] Access is denied: 'dart-sdk/bin/dartaotruntime.exe'

[alexmarkov]

On pkg-win-release, pkg-win-release-arm64 bots Dart build sometimes fails with the following errors:

[3634/3829] COPY dart_precompiled_runtime_product.exe dart-sdk/bin/dartaotruntime.exe
FAILED: dart-sdk/bin/dartaotruntime.exe 
C:/b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.exe ../../build/toolchain/win/tool_wrapper.py recursive-mirror dart_precompiled_runtime_product.exe dart-sdk/bin/dartaotruntime.exe
Traceback (most recent call last):
  File "../../build/toolchain/win/tool_wrapper.py", line 259, in <module>
    sys.exit(main(sys.argv[1:]))
  File "../../build/toolchain/win/tool_wrapper.py", line 27, in main
    exit_code = executor.Dispatch(args)
  File "../../build/toolchain/win/tool_wrapper.py", line 69, in Dispatch
    return getattr(self, method)(*args[1:])
  File "../../build/toolchain/win/tool_wrapper.py", line 110, in ExecRecursiveMirror
    os.unlink(dest)
PermissionError: [WinError 5] Access is denied: 'dart-sdk/bin/dartaotruntime.exe'

It looks like dartaotruntime.exe process is hanging from the previous(?) testing.

Note that "kill processes" step doesn't mention dartaotruntime:

***************** Killing dart *****************
  No dart processes found.
***************** Killing gen_snapshot *****************
  No gen_snapshot processes found.
***************** Killing dart_precompiled_runtime *****************
  No dart_precompiled_runtime processes found.
***************** Killing git *****************
  No git processes found.
***************** Killing vctip *****************
  No vctip processes found.
***************** Killing mspdbsrv *****************
  No mspdbsrv processes found.
***************** Killing firefox *****************
  No firefox processes found.
***************** Killing chrome *****************
  No chrome processes found.
***************** Killing iexplore *****************
  No iexplore processes found.

Maybe it should be also killed before running the build.
@athomas @whesse Could you help with that?

[alexmarkov]

I found tools/task_kill.py which is likely used to kill processes on bots. If that's the case, then https://dart-review.googlesource.com/c/sdk/+/362140 should help.