'buck2 build //tests:' can not find 'sh' when using case 'buck2/examples/remote_execution/buildbarn/tests'

[yaoddao]

• Prerequisite: installed Buildbarn with https://github.com/buildbarn/bb-deployments and use default configuration.
  I have verified that Buildbarn runs correctly and can build c++ using Bazel+Buildbarn.
• Run command below using default configuration in 'buck2/examples/remote_execution/buildbarn/'

  cd ./buck2/examples/remote_execution/buildbarn; 
  buck2 clean; buck2 build -v 4 //tests:

But buck2 reports error:

user@./buck2/examples/remote_execution/buildbarn (main)>buck2 clean; buck2 build -v 4 //tests:
killing buckd server
Buck2 daemon pid 231513 has exited
./tools/buck2/examples/remote_execution/buildbarn/buck-out/v2/forkserver
/tools/buck2/examples/remote_execution/buildbarn/buck-out/v2/log
/tools/buck2/examples/remote_execution/buildbarn/buck-out/v2/tmp
/tools/buck2/examples/remote_execution/buildbarn/buck-out/v2/gen
/tools/buck2/examples/remote_execution/buildbarn/v2
Running action: root//tests:tests (<unspecified>) (stage0) (build), local executor: env -- "TMPDIR=/../tools/buck2/examples/remote_execution/buildbarn/buck-out/v2/tmp/root/6dd044292ff31ae1/tests/__tests__/stage0" "BUCK_SCRATCH_PATH=buck-out/v2/tmp/root/6dd044292ff31ae1/tests/__tests__/stage0" "BUCK2_DAEMON_UUID=6abfee25-4ecc-4593-a77f-932d92a0047a" "BUCK_BUILD_ID=5444dcc8-5a74-4844-9c2d-4dd01e962f36" /usr/bin/sh -c "head -c 10 /dev/urandom > \"\$1\"" -- buck-out/v2/gen/root/6dd044292ff31ae1/tests/__tests__/stage0
Running action: root//tests:tests (<unspecified>) (stage1) (build), re(OSFamily=linux,container-image=docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448) executor: f78fdff32e694fd29b647d9bfdb87739476f39f57ddc361b92582d18fd577140:142
Action failed: root//tests:tests (stage1)
Internal error (stage: remote_call_error): Remote Execution Error (GRPC-SESSION-ID): RE: execution with digest f78fdff32e694fd29b647d9bfdb87739476f39f57ddc361b92582d18fd577140:142: Error was returned on the stream by RE: Failed to run command: **Cannot find executable "sh" in search paths ""**
stdout:
stderr:
Build ID: 5444dcc8-5a74-4844-9c2d-4dd01e962f36
Network: (GRPC-SESSION-ID)
Jobs completed: 10. Time elapsed: 0.1s.
Cache hits: 0%. Commands: 1 (cached: 0, remote: 0, local: 1)
BUILD FAILED
Failed to build 'root//tests:tests (<unspecified>)'

• But after I changed
  ctx.actions.run(["sh", "-c", 'cat "$1" "$1" > "$2"', "--", stage0, stage1.as_output()], category = "stage1")
  to
  ctx.actions.run(["/usr/bin/sh", "-c", 'cat "$1" "$1" > "$2"', "--", stage0, stage1.as_output()], category = "stage1")

  I run 'buck2 build -v 4 //tests:', and BUILD SUCCEEDED.
  So I think the reason of issue above may be Buildbarn worker's container does not search 'sh' in PATH.

• But I entered Buildbarn worker and runner container, I export:

user@../tool/bb-deployments/docker-compose (master)>podman --root=/share exec -it 3e58d0d02f76 sh
# export
export DEBIAN_FRONTEND='noninteractive'
export HOME='/root'
export PATH='/usr/bin/:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin'
export PWD='/tmp'
export TERM='xterm'
export container='podman'

PATH is correct and include /usr/bin;/bin and so on.

And, when I start up Buildbarn using 'podman-compose up', I have added PATH environment in docker-compose.yml

   environment:
   - PATH=/usr/bin/:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/sbin:$PATH

Question:

• Why cannot find executable "sh" in search paths "" when I run 'buck2 build //tests:'?
• Excuse me, may I ask you some questions @krallin : do you set special environment for Buck2 or Buildbarn when you run 'buck2/examples/remote_execution/buildbarn/tests'? Thank you very much!

PS. Related an issue: #549