New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Separate slow tests from quick ones #758
Changes from all commits
28fa666
4e3bb28
c6f9233
db8921e
f2de188
df03308
67087e3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -25,6 +25,9 @@ jobs: | |
runs-on: ubuntu-latest | ||
steps: | ||
- uses: actions/checkout@v3 | ||
with: | ||
ref: ${{ github.event.pull_request.head.ref }} | ||
repository: ${{github.event.pull_request.head.repo.full_name}} | ||
|
||
# Found this solution at | ||
# https://monadical.com/posts/filters-github-actions.html#Case-2-Pull-request | ||
|
@@ -45,7 +48,7 @@ jobs: | |
ci-tests: | ||
needs: check_commit | ||
if: ${{ needs.check_commit.outputs.match != 'true' }} | ||
name: ${{ matrix.os }}, ${{ matrix.tox_env }} | ||
# name: ${{ matrix.os }}, ${{ matrix.tox_env }} | ||
matteobachetti marked this conversation as resolved.
Show resolved
Hide resolved
|
||
runs-on: ${{ matrix.os }} | ||
continue-on-error: ${{ matrix.experimental }} | ||
strategy: | ||
|
@@ -56,47 +59,72 @@ jobs: | |
# we select only a few, in order to conserve resources. If you add | ||
# an additional environment, please include a comment to indicate | ||
# why it is useful. | ||
- os: ubuntu-latest | ||
- name: Black test | ||
os: ubuntu-latest | ||
python: '3.10' | ||
tox_env: 'black' | ||
experimental: false | ||
|
||
# Basic tests on the oldest & newest supported versions of Python, | ||
# recording coverage data for the latter. | ||
- os: ubuntu-latest | ||
- name: Py3.8 with old Astropy version | ||
os: ubuntu-latest | ||
python: '3.8' | ||
tox_env: 'py38-test-astropy4-cov' | ||
experimental: false | ||
|
||
- os: ubuntu-latest | ||
- name: Linux, Py3.10 with coverage | ||
os: ubuntu-latest | ||
python: '3.10' | ||
tox_env: 'py310-test-cov' | ||
experimental: false | ||
|
||
# Basic tests on alternative operating systems. | ||
- os: windows-latest | ||
- name: Windows, Py3.11 with coverage | ||
os: windows-latest | ||
python: '3.11' | ||
tox_env: 'py311-test-cov' | ||
experimental: false | ||
|
||
- os: macos-latest | ||
- name: Mac OS, Py3.11 | ||
os: macos-latest | ||
python: '3.11' | ||
tox_env: 'py311-test' | ||
experimental: false | ||
|
||
# Test with all optional dependencies installed. | ||
- os: ubuntu-latest | ||
- name: Linux, Py3.11 all dependencies and coverage | ||
os: ubuntu-latest | ||
python: '3.11' | ||
tox_env: 'py311-test-alldeps-cov' | ||
use_remote_data: true | ||
experimental: false | ||
|
||
# Development version of dependencies | ||
- os: ubuntu-latest | ||
- name: Linux, Py3.11 with dev versions of dependencies | ||
os: ubuntu-latest | ||
python: '3.11' | ||
tox_env: 'py311-test-devdeps' | ||
experimental: false | ||
|
||
# Test with all optional dependencies installed. | ||
- name: Slow tests on Linux, Py3.11, all deps and coverage | ||
os: ubuntu-latest | ||
python: '3.11' | ||
tox_env: 'py311-test-alldeps-cov' | ||
use_remote_data: true | ||
experimental: false | ||
slow: true | ||
|
||
# Test with all optional dependencies installed. | ||
- name: Slow tests on Linux, Py3.11, basic deps and coverage | ||
os: ubuntu-latest | ||
python: '3.11' | ||
tox_env: 'py311-test-cov' | ||
use_remote_data: true | ||
experimental: false | ||
slow: true | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So does this take the previous matrix of operating systems and environments, and basically just writes them out individually? Does this mean we're no longer testing python 3.9, for example, and we're only testing Windows and MacOS on Python 3.11? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Our matrix was never an actual matrix, but always a list of single tests, as far as I remember. At this moment, indeed, we are testing with py38, py310 and py311. I couldn't manage to make py312-dev work, so left it out. Would you prefer to also have py39? |
||
|
||
steps: | ||
- name: Check out repository | ||
uses: actions/checkout@v3 | ||
|
@@ -121,14 +149,19 @@ jobs: | |
python -c "import pip; print(f'pip {pip.__version__}')" | ||
python -c "import setuptools; print(f'setuptools {setuptools.__version__}')" | ||
python -c "import tox; print(f'tox {tox.__version__}')" | ||
- name: Run tests | ||
if: "! matrix.use_remote_data" | ||
run: tox -e ${{ matrix.tox_env }} | ||
- name: Run tests with remote data | ||
if: "matrix.use_remote_data" | ||
- name: Run quick tests | ||
if: "(! matrix.use_remote_data) && (! matrix.slow)" | ||
run: tox -e ${{ matrix.tox_env }} -- | ||
- name: Run quick tests with remote data | ||
if: "matrix.use_remote_data && (! matrix.slow)" | ||
run: | | ||
pip install pytest-remotedata | ||
tox -e ${{ matrix.tox_env }} -- --remote-data=any | ||
- name: Run slow tests with remote data | ||
if: "matrix.use_remote_data && matrix.slow" | ||
run: | | ||
pip install pytest-remotedata | ||
tox -e ${{ matrix.tox_env }} -- --remote-data=any -m slow --run-slow | ||
- name: Upload coverage to codecov | ||
if: "endsWith(matrix.tox_env, '-cov')" | ||
uses: codecov/codecov-action@v3 | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
import numpy as np | ||
|
||
from astropy.tests.helper import pytest | ||
import pytest | ||
import warnings | ||
import os | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does this do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This one fixes a problem we had with PRs from forks, while maintaining consistency in catching the correct commit message. There was some issue with the commit message, that sometimes was the correct one and sometimes the merge message (when github makes the merge with the current main before testing). Now it looks robust enough (I'm doing this PR from a fork to prove it)