build: add make target to run distributed tests locally - #2017
Merged
gabrielfruet merged 3 commits intoAug 7, 2026
Conversation
Follow-up to lightly-ai#2011: add `make test-distributed` so the DDP-marked tests can be run on the shared gloo pool locally without reconstructing the CI invocation by hand.
saud5150
force-pushed
the
build/test-distributed-make-target
branch
from
August 6, 2026 10:00
4083475 to
5465050
Compare
Contributor
|
/review |
gabrielfruet
enabled auto-merge
August 6, 2026 20:01
gabrielfruet
disabled auto-merge
August 6, 2026 21:02
gabrielfruet
enabled auto-merge
August 6, 2026 21:03
Contributor
|
Very confused that I somehow can't run the workflows here 🤔 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2011, requested by @liopeer in #2011 (comment).
Description
Adds
make test-distributedto run the@pytest.mark.DDPtests on the shared gloo pool locally, mirroring the CIdistributed-testsjob (USE_PYTEST_POOL=1 ... --runslow -m DDP) so the invocation doesn't have to be reconstructed by hand.One non-obvious detail, documented as a comment on the target: it must use
python -m pytestrather than plainpytestlike the neighbouring targets.python -mputs the repo root onsys.path, which the spawned pool workers inherit; with the barepytestentry point the workers crash on their first task withModuleNotFoundError: No module named 'tests'and the session hangs silently at the pool'sstarmap. (CI never hits this because it already invokespython -m pytest.)--runslowis kept for parity with the CI job even though neither current DDP test is slow-marked, so a future slow+DDP test behaves the same locally and in CI. On platforms without a usable backend (e.g. Windows),USE_PYTEST_POOLfolds in the platform checks, so the target just reports the DDP tests as skipped.Tests
macOS arm64, Python 3.12, fresh venv (
.[dev,matplotlib,minimal,timm,video]):Documentation
.rstfiles).Makefile-only change; the
python -m pytestrequirement is documented on the target itself.Implications / comments / further issues
.PHONYis omitted to match the neighbouring test targets (test,test-fast,lint, ...), none of which declare it. Happy to add it here (or as a repo-wide.PHONYcleanup in a separate PR) if you prefer.