Skip to content

CI Use pytest-xdist in debian 32 build #32031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 29, 2025
Merged

CI Use pytest-xdist in debian 32 build #32031

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
221a33f
CI Use pytest-xdist in debian 32 build
lesteve Aug 27, 2025
0595d58
[azure parallel]
lesteve Aug 27, 2025
bdba619
[azure parallel] add pytest-xdist in lock-file
lesteve Aug 28, 2025
af3b282
[azure parallel] use logical cores for pytest-xdist argument
lesteve Aug 28, 2025
be7f1d0
[azure parallel] tweak to avoid using pytest-xdist if there is only o…
lesteve Aug 28, 2025
2121aee
commit without azure parallel
lesteve Aug 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,8 +213,6 @@ jobs:
DISTRIB: 'debian-32'
COVERAGE: "true"
LOCK_FILE: './build_tools/azure/debian_32bit_lock.txt'
# disable pytest xdist due to unknown bug with 32-bit container
PYTEST_XDIST_VERSION: 'none'
SKLEARN_TESTS_GLOBAL_RANDOM_SEED: '4' # non-default seed

- template: build_tools/azure/posix.yml
Expand Down
7 changes: 6 additions & 1 deletion build_tools/azure/debian_32bit_lock.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,11 @@ coverage[toml]==7.10.5
# via pytest-cov
cython==3.1.3
# via -r build_tools/azure/debian_32bit_requirements.txt
execnet==2.1.1
# via pytest-xdist
iniconfig==2.1.0
# via pytest
joblib==1.5.1
joblib==1.5.2
# via -r build_tools/azure/debian_32bit_requirements.txt
meson==1.9.0
# via meson-python
Expand All @@ -35,7 +37,10 @@ pytest==8.4.1
# via
# -r build_tools/azure/debian_32bit_requirements.txt
# pytest-cov
# pytest-xdist
pytest-cov==6.2.1
# via -r build_tools/azure/debian_32bit_requirements.txt
pytest-xdist==3.8.0
# via -r build_tools/azure/debian_32bit_requirements.txt
threadpoolctl==3.6.0
# via -r build_tools/azure/debian_32bit_requirements.txt
1 change: 1 addition & 0 deletions build_tools/azure/debian_32bit_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ cython
joblib
threadpoolctl
pytest
pytest-xdist
pytest-cov
ninja
meson-python
6 changes: 4 additions & 2 deletions build_tools/azure/test_script.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,10 @@ if [[ "$COVERAGE" == "true" ]]; then
fi

if [[ "$PYTEST_XDIST_VERSION" != "none" ]]; then
XDIST_WORKERS=$(python -c "import joblib; print(joblib.cpu_count(only_physical_cores=True))")
TEST_CMD="$TEST_CMD -n$XDIST_WORKERS"
XDIST_WORKERS=$(python -c "import joblib; print(joblib.cpu_count())")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this change needed? Is this caused by a bug or limitation in joblib?

Does this negatively impact other build times?

Copy link
Member Author

@lesteve lesteve Aug 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically right now, most (all?) of the Linux Azure VMs appears to have 1 physical cores and 2 logical cores 1, so we run pytest -n1 2. pytest -n2 speeds things up on debian-32 build but for some reason (maybe BLAS?) does not seem to have much effect on the other Linux builds, see #32031 (comment).

Footnotes

  1. this seems weird since those are vCPUs so I would expect the number of physical and logical to be the same 🤷? ↩

  2. which is a bit weird we may as well not use pytest-xdist in this case. This is another one of my unrelated change 😅. ↩

if [[ "$XDIST_WORKERS" != 1 ]]; then
TEST_CMD="$TEST_CMD -n$XDIST_WORKERS"
fi
fi

if [[ -n "$SELECTED_TESTS" ]]; then
Expand Down
1 change: 1 addition & 0 deletions build_tools/update_environments_and_lock_files.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,7 @@ def remove_from(alist, to_remove):
"joblib",
"threadpoolctl",
"pytest",
"pytest-xdist",
"pytest-cov",
"ninja",
"meson-python",
Expand Down
Loading