Running shell tests with bazel run intermittently omits outputs

[crydell-ericsson]

Description of the bug:

We have a use case wherein we want to pipe the output of a Bazel test target implemented with a shell script to an external tool. There are certain parts of the output that are essential to pass to the external tool. In addition, we want to be able to run this target with bazel run in addition to bazel test.

Assuming that the target is named :foo and that the external tool is named my_tool, we can do this with the command:

$ bazel run :foo | my_tool

This usually works, but my_tool will intermittently not receive the full intended output from the :foo run. The failures seem to be caused by a race condition that sometimes results in subprocesses being killed before they have had the opportunity to write to stdout. This problem seems to have been introduced by the commit 9051faa.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Create a new workspace with the files:

# File: BUILD

sh_test(
  name = "foo",
  srcs = ["foo.sh"],
)

# File: foo.sh

echo "useless echo"
echo "useful echo"
exit 0

Then, try to grep "useful echo" from the script output repeatedly until failure. This can for example be done with:

$ while bazel run :foo 2> /dev/null | grep "useful echo" > /dev/null ; do echo -n "." ; done && echo "Failed to grep 'useful echo'"

Alternatively, it's faster to use the --script_path option to create a separate script for running the test.

$ bazel run :foo --script_path=foo_script.sh
$ while ./foo_script.sh 2> /dev/null | grep "useful echo" > /dev/null ; do echo -n "." ; done && echo "Failed to grep 'useful echo'"

We have found that this usually fails within 1000 iterations, but under heavy load from other processes on the machine it been observed to go below 50.

Which operating system are you running Bazel on?

Red Hat Enterprise Linux Server 7.9

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[meisterT]

Can you please try this with bazel from head? We recently fixed issues with bazel run that sound similar.

[crydell-ericsson]

Can you please try this with bazel from head? We recently fixed issues with bazel run that sound similar.

I do not currently have an environment set up where I can build the latest head, but I have tentatively tested the following:

1. Create a script with bazel run :foo --script_path=foo_script.sh -- this was done with release 7.0.0-pre.20230302.1
2. Replace the path to test-setup.sh on the last line of the generated script with a path to the test-setup.sh in the source repository of head (dbb09c9)
3. Run the same command as in the original issue, i.e. while ./foo_script.sh 2> /dev/null | grep "useful echo" > /dev/null ; do echo -n "." ; done && echo "Failed to grep 'useful echo'"

This still fails as described in the issue. While foo_script.sh was not built with a Bazel built from head, it does use the test-setup.sh found on head, which I reckon is where the problem is. I can try to investigate this further once I have an environment where I can build Bazel from head set up.

[crydell-ericsson]

I have now tried this with a Bazel built from head (8a23169) and can verify that the problem still exists.