Process execution sometimes fails on ppc64le and s390x

[AdamWill]

I'm working on updating Fedora's openQA packages to the latest git versions. os-autoinst was recently improved to include exit codes in output when running commands via run_diag (which uses ReadWriteProcess's process to do the heavy lifting), and a test for this was added.

In test builds, this test is quite often failing on ppc64le or s390x arch, because the command - a simple echo foo - exits with 1 instead of the expected 0:

2:     #   Failed test 'Exit code appear in log'
2:     #   at ./13-osutils.t line 167.
2:     # STDERR:
2:     # [2021-11-26T19:00:09.618335Z] [debug] running `echo foo`
2:     # [2021-11-26T19:00:09.619749Z] [debug] Command `echo foo` terminated with 1
2:     #   foo
2:     # 
2:     # doesn't match:
2:     # (?^u:terminated with 0)
2:     # as expected
2:     # Looks like you failed 1 test of 10.
2: 
2: #   Failed test 'run_diag'
2: #   at ./13-osutils.t line 190.
2: # Looks like you failed 1 test of 9.

I don't know why this is happening, but it seems likely it's an issue in RWP, I don't see how the code on the os-autoinst side could be causing it.

[AdamWill]

@cfconrad

[AdamWill]

Annoyingly, if I isolate the relevant code into a test script and run it over and over on a ppc64le box (a different one - one of our worker boxes, since I don't have shell access to the package build hosts), it works fine every time. Sigh.

[cfconrad]

Yeah I think you are right. This is also a race in Mojo-IOLoop-ReadWriteProcess with short executed commands.
Currently I have no idea to solve them.

There is a race in fork/open and session->register($pid). If the later is called after the process is already finished (collected by collected_orphan), then we get that error.

It's actually really easy to reproduce, I get it just by calling:

perl <<'EOT'
use Mojo::Base -strict;
use Mojo::IOLoop::ReadWriteProcess 'process';
say process(execute => '/usr/bin/true')->quirkiness(1)->start()->wait_stop()->exit_status();
EOT

If you don't get it, you could produce this race by adding a sleep in here: https://github.com/mudler/Mojo-IOLoop-ReadWriteProcess/blob/master/lib/Mojo/IOLoop/ReadWriteProcess.pm#L482

[cfconrad]

I only found this workaround... let's see what others think.

[foursixnine]

This is fixed by #22

[AdamWill]

awesome, thanks.