Skip to content

BUG test_video_scorer fixture uses CWD-relative path, races between xdist workers under --dist=load #1794

@immu4989

Description

@immu4989

What happens

tests/unit/score/test_video_scorer.py fails non-deterministically when the unit suite is run with pytest tests/unit -n auto (or any -n N without --dist=loadfile):

FAILED tests/unit/score/test_video_scorer.py::test_score_video_no_scores
FAILED tests/unit/score/test_video_scorer.py::test_score_video_true_false
  FileNotFoundError: Video file not found: test_video.mp4

The failure raises in pyrit/score/video_scorer.py:116 inside VideoHelper._score_frames_async when os.path.exists(video_path) returns False.

Root cause

The video_converter_sample_video fixture in tests/unit/score/test_video_scorer.py (line 35) creates the sample video at a CWD-relative path:

video_path = "test_video.mp4"
...
output_video = cv2.VideoWriter(video_path, ...)
...
if os.path.exists(video_path):
    os.remove(video_path)

Under pytest-xdist's default --dist=load distribution, the 17 tests in this file get sharded across workers. All workers share the same CWD, so they race on a single test_video.mp4: one worker's teardown deletes the file while another worker's test is mid-read.

Why CI doesn't catch it

The Makefile runs pytest -n 4 --dist=loadfile (lines 51, 54, 57, 60, 63). loadfile pins all tests from one file to one worker, which serialises the fixture and masks the race. Devs running plain pytest -n auto locally hit it.

Repro

pytest tests/unit/score/test_video_scorer.py -n auto
# -> 2 failed, 3 passed (failures vary by run)

Proposed fix

Switch the fixture to pytest's built-in tmp_path for per-test isolation. The pattern is already used by the sister file tests/unit/prompt_converter/test_add_image_video_converter.py (line 20), so this just brings test_video_scorer.py in line. PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions