[Bug]: Data race when running multiple py_binary instances in parallel

[nmercaldo-bdai]

What happened?

When launching multiple instances of a py_binary in parallel, we intermittently encounter the following error:

Error:   × Unable to run command:
  ├─▶ failed to remove directory `/home/nmercaldo/.cache/bazel/_bazel_nmercaldo/4930e7091c9bc99e4bf079ca7ea04ef3/execroot/_main/bazel-out/k8-opt/bin/projects/
  │   manipulation/launch.runfiles/.bin.venv`
  ╰─▶ Directory not empty (os error 39)

The error is more likely to occur if the executed binary is dependency heavy.

Version

Development (host) and target OS/architectures:

Ubuntu 22.04.5 LTS, x86_64

Output of bazel --version:

bazel 8.3.1

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

aspect_rules_py: 1.6.3
rules_uv: 0.86.0
rules_python: 1.5.2 

Language(s) and/or frameworks involved:

Python

How to reproduce

Create a Bazel module with the following files:

launcher.py

import subprocess
import sys

p1 = subprocess.Popen([sys.argv[1]])
p2 = subprocess.Popen([sys.argv[1]])

p1.wait()
p2.wait()

bin.py

print("hello")

requirements.in

numpy

BUILD.bazel

load("@rules_uv//uv:pip.bzl", "pip_compile")
load("@aspect_rules_py//py:defs.bzl", "py_binary")
load("@pip//:requirements.bzl", "requirement")

pip_compile(
    name = "lock",
    requirements_in  = "//:requirements.in",
    requirements_txt = "//:requirements.txt",
)

py_binary(
    name = "bin",
    srcs = ["bin.py"],
    deps = [
        "@pip//numpy",
    ],
)

py_binary(
    name = "launch",
    srcs = ["launcher.py"],
    args = [
        "$(location :bin)",
    ],
    data = [
        ":bin",
    ],
)

MODULE.bazel

bazel_dep(name = "rules_uv", version = "0.86.0")
bazel_dep(name = "rules_python", version = "1.5.2")
bazel_dep(name = "aspect_rules_py", version = "1.6.3")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    configure_coverage_tool = True,
    is_default = True,
    python_version = "3.10",
)
use_repo(python, "python_3_10")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pip",
    python_version = "3.10",
    requirements_lock = "//:requirements.txt",
)
use_repo(pip, "pip")

Then run the following:

`bazel run //:lock`
`bazel run //:launch` 

Any other information?

No response

[arrdem]

It may be possible to add file locking to guard venv creation, but doing so would have to be coupled with a new mechanism to allow for the validation and reuse of dynamically created venv directories, otherwise you've just moved the data race because the second tool will try to recreate a virtualenv that's actively in use. It seems unwise to develop a new and immature mechanism for doing so when for instance PEX already has a much more mature solution to this specific problem. And it still suffers from startup time overheads discussed offline.

The current dynamically created venv strategy is unsound on a number of levels and will likely be removed or at least relegated to a more limited role in rules_py@v2.0.0.

The new static venv machinery is, overall, a much better approach and proper upstream interpreter support for relocatable venvs will improve that story in forthcoming Pythons.

Won't fix. See #587.