Flaky standalone_2/io/process_run_test test failure on iso-stress bot

[alexmarkov]

Log:

[/b/s/w/ir/cache/builder/sdk/runtime/tests/concurrency/../../../tests/standalone_2/io/process_run_test.dart] finished
[/b/s/w/ir/cache/builder/sdk/runtime/tests/concurrency/../../../tests/standalone_2/io/process_run_test.dart] finished
[/b/s/w/ir/cache/builder/sdk/runtime/tests/concurrency/../../../tests/standalone_2/io/process_run_test.dart] finished
[/b/s/w/ir/cache/builder/sdk/runtime/tests/concurrency/../../../tests/standalone_2/io/process_run_test.dart] finished
Spawning tests in isolates resulted in the following errors:
------------------------------------------------------------
[/b/s/w/ir/cache/builder/sdk/runtime/tests/concurrency/../../../tests/standalone_2/io/process_run_test.dart] error: [Expect.fail('System should find process_run_test executable (ProcessException: Text file busy
  Command: /b/s/w/ityr17vl2e/process_run_testXNVBVK/path with space/process_run_test )'), #0      Expect._fail (package:expect/expect.dart:742:5)
#1      Expect.fail (package:expect/expect.dart:310:5)
#2      testProcessPathWithSpace (file:///b/s/w/ir/cache/builder/sdk/tests/standalone_2/io/process_run_test.dart:54:12)
#3      main (file:///b/s/w/ir/cache/builder/sdk/tests/standalone_2/io/process_run_test.dart:63:3)
#4      wrapper3238.<anonymous closure> (file:///b/s/w/ir/cache/builder/sdk/runtime/tests/concurrency/generated_stress_test.dart:58452:14)
#5      _rootRun (dart:async/zone.dart:1426:13)
#6      _CustomZone.run (dart:async/zone.dart:1328:19)
#7      _runZoned (dart:async/zone.dart:1861:10)
#8      runZoned (dart:async/zone.dart:1783:10)
#9      wrapper3238 (file:///b/s/w/ir/cache/builder/sdk/runtime/tests/concurrency/generated_stress_test.dart:58451:3)
#10     _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:300:17)
#11     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:192:12)
]
------------------------------------------------------------
-> Setting exitCode to 255

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8824432375318156721/+/u/Run_Isolate_Stress_Tests_shard_1/task_stdout_stderr:_Run_Isolate_Stress_Tests_shard_1

/cc @aam @mkustermann

[mkustermann]

Today I've looked at this a bit deeper. What happens it that multiple isolates run this process_run_test.dart code which makes a temporary directory, copies the process_test binary into it and runs it.

My first thought is that the random number generator will cause collisions in the temporary directory names, thereby making two isolates run in the same temp dir which would race on running & deleting that executable. Though it doesn't "seem" like this is happening.

The root cause is that the subprocess (after fork()ing) calls execvp and gets ETXTBSY. There's a special mention about this in the manpage:

The behavior of execlp() and execvp() when errors occur while attempting to execute the file is historic practice, but has not traditionally been documented and is not specified by the POSIX  standard.
BSD (and possibly other systems) do an automatic sleep and retry if ETXTBSY is encountered. 
Linux treats it as a hard error and returns immediately.

So we're observing the Linux treats it as a hard error situation.

Though I have been so far unable to identify what other thread / process may have the very same binary opened in write mode. It is written once (with a File.copySync()) and not touched until the exec().

In any case - this is dart:io related so I'm going to exclude this test from the iso-stress builder.