Skip to content

FIX test_video_scorer: use tmp_path for sample video so tests are xdist-safe#1795

Merged
romanlutz merged 1 commit into
microsoft:mainfrom
immu4989:fix/video-scorer-tmp-path
May 25, 2026
Merged

FIX test_video_scorer: use tmp_path for sample video so tests are xdist-safe#1795
romanlutz merged 1 commit into
microsoft:mainfrom
immu4989:fix/video-scorer-tmp-path

Conversation

@immu4989
Copy link
Copy Markdown
Contributor

@immu4989 immu4989 commented May 24, 2026

Fixes #1794.

The video_converter_sample_video fixture in tests/unit/score/test_video_scorer.py wrote its sample MP4 to a CWD-relative path ("test_video.mp4"). Under pytest-xdist -n auto (default --dist=load), the 17 tests in this file shard across workers that share the same CWD, racing on a single file and producing flaky FileNotFoundErrors.

CI runs pytest -n 4 --dist=loadfile (Makefile) which pins each file to one worker and masks the race, so this only bites devs running the xdist default locally.

Fix

Inject tmp_path into the fixture and place the sample video there. Per-test temp dirs are unique across workers, so the race vanishes. The same pattern is already used by tests/unit/prompt_converter/test_add_image_video_converter.py; this brings test_video_scorer.py in line.

The explicit os.path.exists / os.remove teardown is removed -- tmp_path cleans itself up.

Verification

# Pre-fix:
pytest tests/unit/score/test_video_scorer.py -n auto   -> 2 failed, 3 passed
pytest tests/unit -n auto                              -> 2 failed in 99s

# Post-fix:
pytest tests/unit/score/test_video_scorer.py -n auto   -> 17 passed in 4.12s
pytest tests/unit -n auto                              -> 8215 passed in 37s

…st-safe

The video_converter_sample_video fixture wrote its sample MP4 to a CWD-relative path. Under pytest-xdist's default --dist=load mode, the file's tests shard across workers that share the same CWD and race on a single test_video.mp4, producing flaky FileNotFoundError. CI uses --dist=loadfile (Makefile) which pins file-to-worker and masks the race, so this only bites devs running plain pytest -n auto locally.

Switch the fixture to pytest's tmp_path. Same pattern already used by tests/unit/prompt_converter/test_add_image_video_converter.py.
Copy link
Copy Markdown
Contributor

@romanlutz romanlutz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea!

@immu4989
Copy link
Copy Markdown
Contributor Author

Appreciate the feedback. Thank you !

@romanlutz romanlutz added this pull request to the merge queue May 25, 2026
Merged via the queue into microsoft:main with commit 8003212 May 25, 2026
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG test_video_scorer fixture uses CWD-relative path, races between xdist workers under --dist=load

2 participants