Running Python tests with Pytest fails with: Error expanding output globs: Failed to scan directory /private/...

[ruslan-roboto]

Describe the bug
I run all integration tests for my application using pants --integ test --force ::. This used to succeed, but now it fails with an IntrinsicError related to "expanding output globs".

Here's a (sanitized) example of the failure. The test file name is different every time:

11:52:53.85 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

IntrinsicError: Failed to execute: Process {
    argv: [
        "./pytest_runner.pex_pex_shim.sh",
        "--color=yes",
        "--junit-xml=packages.myapp.test.myapp_integ_tests.integ.test_interesting_condition.py.integ_tests.xml",
        "-o",
        "junit_family=xunit2",
        "--cov-report=",
        "--cov-config=build-support/.coveragerc",
        "--cov=packages/myapp/src",
        "--cov=packages/myapp_integ/src",
        "--cov=packages/myapp_integ/test",
        "packages/myapp_integ/test/myapp_integ_tests/integ/test_interesting_condition.py",
    ],
    env: {
        "AWS_PROFILE": "<redacted>",
        "PEX_EXTRA_SYS_PATH": "packages/myapp/src:packages/myapp_integ/src:packages/myapp_integ/test",
    },
    working_directory: None,
    input_digests: InputDigests {
        complete: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<deeec633893aa0d8bf3190c13044ad29bf7f83b8b1ee6d1ce574a29340ff5045>,
                size_bytes: 831,
            },
            tree: "Some(..)",
        },
        nailgun: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>,
                size_bytes: 0,
            },
            tree: "Some(..)",
        },
        inputs: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<deeec633893aa0d8bf3190c13044ad29bf7f83b8b1ee6d1ce574a29340ff5045>,
                size_bytes: 831,
            },
            tree: "Some(..)",
        },
        immutable_inputs: {},
        use_nailgun: {},
    },
    output_files: {
        RelativePath(
            ".coverage",
        ),
        RelativePath(
            "packages.myapp_integ.test.myapp_integ_tests.integ.test_interesting_condition.py.integ_tests.xml",
        ),
    },
    output_directories: {
        RelativePath(
            "extra-output",
        ),
    },
    timeout: None,
    execution_slot_variable: None,
    concurrency_available: 0,
    description: "Run Pytest for packages/myapp_integ/test/myapp_integ_tests/integ/test_interesting_condition.py:integ_tests",
    level: Debug,
    append_only_caches: {
        CacheName(
            "pex_root",
        ): RelativePath(
            ".cache/pex_root",
        ),
        CacheName(
            "python_build_standalone",
        ): RelativePath(
            ".python-build-standalone",
        ),
    },
    jdk_home: None,
    cache_scope: PerSession,
    execution_environment: ProcessExecutionEnvironment {
        name: None,
        platform: Macos_arm64,
        strategy: Local,
    },
    remote_cache_speculation_delay: 0ns,
    attempt: 0,
}

Error expanding output globs: Failed to scan directory "/private/var/folders/7c/3ny04jh95k9fmjxtksxqyh2m0000gn/T/pants-sandbox-tK9sl1/": No such file or directory (os error 2)

Pants version
2.23.0

OS
MacOS Sonoma 14.7.4

Additional info

• I tried deleting .pants.d/ and re-running - no effect
• Checked whether I was running out of disk space - got plenty left.
• Restarted my Mac since I hadn't done so in months - no effect

[huonw]

Sorry for the issue and thanks for filing a bug.

This used to succeed

Do you remember anything that changed between when it worked and now? For instance, upgrading Pants versions?

[ruslan-roboto]

@huonw we updated from Pants 2.21 to Pants 2.23 about a month ago. I'll see if the issue persists on Pants 2.24. I can't think of anything else about our integration test setup that has materially changed. Thanks for taking a look!

[huonw]

Please let us know the results of your investigation! If it still reproduces, the next steps will be:

• best case: make a reduced reproducer than you can share
• if not, at least share as much of your pants.toml and the relevant BUILD files as you can

[ruslan-roboto]

Hi, sorry for only now updating this issue. It has been a busy time.

Several observations:

• I've now sporadically hit this issue when running just a subset of all our integ tests, or even individual tests. It's not required to run the entire integ test suite.
• Re-running tests with --keep-sandboxes=on_failure almost always succeeds. I was trying to catch the bug in the act, but instead it disappears 🙂
• The one time where I still got the above failure after having specified --keep-sandboxes=on_failure, the sandbox directory was precisely the directory that "couldn't be scanned", and I confirmed that it existed and had files and sub-directories.
• I've only tried running the entire integ test suite with Pants 2.24 once, and the run succeeded. I'll try several more times to see if it succeeds consistently.

[ruslan-roboto]

More observations: switching to Pants 2.26 (latest as of this writing) doesn't resolve the issue.

Here's an anonymized version of our pants.toml:

[GLOBAL]
pants_version = "2.26.0"

backend_packages = [
    "pants.backend.awslambda.python",
    "pants.backend.docker",
    "pants.backend.experimental.adhoc",
    "pants.backend.experimental.python",
    "pants.backend.tools.preamble",
    "pants.backend.python",
    "pants.backend.python.lint.autoflake",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.flake8",
    "pants.backend.python.lint.isort",
    "pants.backend.python.typecheck.mypy",
    "pants.backend.shell",
]

pants_ignore = [
    "foo",
    "packages/bar",  # Built by CDK
    "web-ui"
]

[source]
root_patterns = [
    "build-support/src",
    "packages/*/build-support",
    "packages/*/src",
    "packages/*/test",
    "packages/myapp",
    "scripts",
]

[python]
enable_resolves = true
default_resolve = "python-default"
interpreter_constraints = ["==3.10.*"]

[python.resolves]
python-default = "build-support/lockfiles/python-default.lock"

[python.resolves_to_constraints_file]
python-default = "build-support/lockfiles/constraints.txt"

[autoflake]
args = [
    "--exclude",
    "'**/build/*,**/antlrgen/*,**/vendor/*'",
    "--in-place",
    "--remove-all-unused-imports",
    "--remove-unused-variables"
]
install_from_resolve = "python-default"

[black]
install_from_resolve = "python-default"

[coverage-py]
report = ["xml", "html"]
config = "build-support/.coveragerc"
install_from_resolve = "python-default"

[flake8]
config = "build-support/.flake8"

[isort]
args = ["--verbose"]
config = "build-support/requirements/tools/pyproject.toml"
install_from_resolve = "python-default"

[mypy]
args = [
    "--check-untyped-defs",
    "--follow-imports silent",
]
config = "build-support/requirements/tools/pyproject.toml"
install_from_resolve = "python-default"

[preamble]
template_by_globs = "@build-support/preamble-config.yaml"

[pytest]
install_from_resolve = "python-default"

[cli.alias]
--integ = "--tag=integ"
--unit = "--tag='-integ' --tag='-cli'"
unittest = "--tag='-integ' --tag='-cli' test"

[tailor]
# 2024-07-12: Should use 'ignore_adding_targets' instead, but that isn't working.
ignore_paths = ["packages/myapp/BUILD"]

[test]
use_coverage = true
show_rerun_command = true

[ruslan-roboto]

Any ideas? Seems like a race condition where the test goal tries to do a file operation in the Pants sandbox, but what it's looking for in the sandbox doesn't exist yet.

[ruslan-roboto]

@huonw could anyone take a look?