What happens
tests/unit/score/test_video_scorer.py fails non-deterministically when the unit suite is run with pytest tests/unit -n auto (or any -n N without --dist=loadfile):
FAILED tests/unit/score/test_video_scorer.py::test_score_video_no_scores
FAILED tests/unit/score/test_video_scorer.py::test_score_video_true_false
FileNotFoundError: Video file not found: test_video.mp4
The failure raises in pyrit/score/video_scorer.py:116 inside VideoHelper._score_frames_async when os.path.exists(video_path) returns False.
Root cause
The video_converter_sample_video fixture in tests/unit/score/test_video_scorer.py (line 35) creates the sample video at a CWD-relative path:
video_path = "test_video.mp4"
...
output_video = cv2.VideoWriter(video_path, ...)
...
if os.path.exists(video_path):
os.remove(video_path)
Under pytest-xdist's default --dist=load distribution, the 17 tests in this file get sharded across workers. All workers share the same CWD, so they race on a single test_video.mp4: one worker's teardown deletes the file while another worker's test is mid-read.
Why CI doesn't catch it
The Makefile runs pytest -n 4 --dist=loadfile (lines 51, 54, 57, 60, 63). loadfile pins all tests from one file to one worker, which serialises the fixture and masks the race. Devs running plain pytest -n auto locally hit it.
Repro
pytest tests/unit/score/test_video_scorer.py -n auto
# -> 2 failed, 3 passed (failures vary by run)
Proposed fix
Switch the fixture to pytest's built-in tmp_path for per-test isolation. The pattern is already used by the sister file tests/unit/prompt_converter/test_add_image_video_converter.py (line 20), so this just brings test_video_scorer.py in line. PR to follow.
What happens
tests/unit/score/test_video_scorer.pyfails non-deterministically when the unit suite is run withpytest tests/unit -n auto(or any-n Nwithout--dist=loadfile):The failure raises in
pyrit/score/video_scorer.py:116insideVideoHelper._score_frames_asyncwhenos.path.exists(video_path)returns False.Root cause
The
video_converter_sample_videofixture intests/unit/score/test_video_scorer.py(line 35) creates the sample video at a CWD-relative path:Under pytest-xdist's default
--dist=loaddistribution, the 17 tests in this file get sharded across workers. All workers share the same CWD, so they race on a singletest_video.mp4: one worker's teardown deletes the file while another worker's test is mid-read.Why CI doesn't catch it
The
Makefilerunspytest -n 4 --dist=loadfile(lines 51, 54, 57, 60, 63).loadfilepins all tests from one file to one worker, which serialises the fixture and masks the race. Devs running plainpytest -n autolocally hit it.Repro
Proposed fix
Switch the fixture to pytest's built-in
tmp_pathfor per-test isolation. The pattern is already used by the sister filetests/unit/prompt_converter/test_add_image_video_converter.py(line 20), so this just bringstest_video_scorer.pyin line. PR to follow.