CI updates #244

honno · 2024-03-15T13:02:22Z

Checks the first two points of Update libraries we test on CI #235
- Replace numpy.array_api with array-api-strict
- Cover meta tests
Removes --ci flag & splits the meta tests into a seperate folder ./meta_tests/
- Any test that wasn't in ./array_api_tests/meta/ was a CI test anywho
- Default runs of the test suite should probably ignore the meta tests, as they're not very well structured and noisy for users
- A seperate top-level folder avoids weird conftest.py hacks
Also marks any test that doesn't use vectorization with an introduced unvectorized marker
- Currently a no-op. My idea was to limit examples specifically on these on runtime, but I couldn't work that out easily so something for a follow-up PR.
- Generally good for us to denote which tests could possibly be improved on performance-wise.

array_api_tests/conftest.py

.github/workflows/numpy.yml

asmeurer · 2024-03-15T20:30:59Z

I get this error when I try to use this locally

$ARRAY_API_TESTS_MODULE=numpy.array_api pytest
=========================================================================== test session starts ===========================================================================
platform darwin -- Python 3.11.7, pytest-8.0.0, pluggy-1.4.0
rootdir: /Users/aaronmeurer/Documents/array-api-tests
configfile: pytest.ini
plugins: pudb-0.7.0, hypothesis-6.99.2, json-report-1.5.0, cov-4.1.0, doctestplus-1.1.0, typeguard-4.1.5, flakes-4.0.5
collected 69 items / 1 error
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1645, in getoption
INTERNALERROR>     val = getattr(self.option, name)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> AttributeError: 'Namespace' object has no attribute 'skips_file'
INTERNALERROR>
INTERNALERROR> The above exception was the direct cause of the following exception:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/main.py", line 272, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/logging.py", line 783, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/warnings.py", line 118, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1365, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/main.py", line 336, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/main.py", line 802, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/cacheprovider.py", line 413, in pytest_collection_modifyitems
INTERNALERROR>     res = yield
INTERNALERROR>           ^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/cacheprovider.py", line 339, in pytest_collection_modifyitems
INTERNALERROR>     res = yield
INTERNALERROR>           ^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/Documents/array-api-tests/array_api_tests/conftest.py", line 114, in pytest_collection_modifyitems
INTERNALERROR>     skips_file = skips_path = config.getoption('--skips-file')
INTERNALERROR>                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Users/aaronmeurer/miniconda3/envs/array-apis/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1656, in getoption
INTERNALERROR>     raise ValueError(f"no option named {name!r}") from e
INTERNALERROR> ValueError: no option named 'skips_file'

.github/workflows/numpy.yml

honno · 2024-03-18T15:47:50Z

I get this error when I try to use this locally...

Ah I think this is why conftest.py exists in the top-level then 😅 Reverted that change.

Left by mistake whilst debugging

And move meta tests to top-level meta_tests folder

Only assert mixed sign scenarios when signbit() is available

And update skips file

asmeurer · 2024-03-18T18:04:38Z

Tests seem to work locally now.

asmeurer · 2024-03-18T18:05:59Z

Ah I think this is why conftest.py exists in the top-level then 😅 Reverted that change.

pytest has some seriously annoying limitations like this. It's a good thing we aren't trying to distribute the test suite as a package because this sort of thing would make doing that extremely difficult.

honno · 2024-03-19T16:19:41Z

One more thing, I've figured out how to limit examples for the tests marked with unvectorized in the pytest_collection_modifyitems hook, so have added that functionality to conftest.py. There's room for improvement and I'll explore a way to do it without using private (if stable) Hypothesis APIs, but should be a nice quick win, and I should of guarded nicely if Hypothesis in the future complains at what we're doing.

asmeurer · 2024-03-19T19:04:26Z

Feel free to merge this whenever you feel it is ready.

asmeurer · 2024-03-20T19:03:33Z

This broke the array-api-compat CI. Fix at #245

asmeurer · 2024-03-20T19:04:30Z

How do you feel about adding these as required checks in the branch protection rules? Admins would still be able to bypass the checks if necessary, but it would making merging a PR that breaks the tests much less likely.

honno · 2024-03-21T14:39:00Z

How do you feel about adding these as required checks in the branch protection rules? Admins would still be able to bypass the checks if necessary, but it would making merging a PR that breaks the tests much less likely.

Mind my ignorance on branch protection rules—what would we be adding as required checks exactly? The tests pasing? Or new tests for array-api-compat?

asmeurer · 2024-03-21T18:52:05Z

The tests for this repo. It isn't possible to require checks across repos.

I went ahead and enabled it. You can go to https://github.com/data-apis/array-api-tests/settings/branch_protection_rules/22738169 to change the settings.

honno requested a review from asmeurer March 15, 2024 13:02

honno force-pushed the ci-updates branch from a602450 to 266e980 Compare March 15, 2024 16:21

asmeurer reviewed Mar 15, 2024

View reviewed changes

array_api_tests/conftest.py Outdated Show resolved Hide resolved

asmeurer reviewed Mar 15, 2024

View reviewed changes

.github/workflows/numpy.yml Outdated Show resolved Hide resolved

asmeurer reviewed Mar 15, 2024

View reviewed changes

.github/workflows/numpy.yml Outdated Show resolved Hide resolved

honno force-pushed the ci-updates branch from a64c72b to fe0acfa Compare March 18, 2024 15:46

honno requested a review from asmeurer March 18, 2024 15:56

honno added 9 commits March 18, 2024 16:01

Uncomment tracebackhide

7201c82

Left by mistake whilst debugging

Remove unnecessary f-string

8bac520

Remove --ci option

f08da7f

And move meta tests to top-level meta_tests folder

Test array_api_strict on CI

4f4d36a

Update test_assert_array_elements

87fdec9

Only assert mixed sign scenarios when signbit() is available

Introduce unvectorized pytest marker

613c3c4

Distinguish test folders in CI workflow

d180f02

Better naming for test workflow

f48e4f2

Change relevant references from NumPy to array-api-strict

582955c

And update skips file

honno force-pushed the ci-updates branch from fe0acfa to 582955c Compare March 18, 2024 16:01

honno added 3 commits March 19, 2024 15:20

Limit Hypothesis examples for unvectorized tests

1c1c4ab

Deprecate --ci instead

f46f45e

Guard against unforseen problems for decorating unvectorized tests

526c2d4

honno merged commit 5d883a8 into data-apis:master Mar 20, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI updates #244

CI updates #244

honno commented Mar 15, 2024 •

edited

Loading

asmeurer commented Mar 15, 2024

honno commented Mar 18, 2024 •

edited

Loading

asmeurer commented Mar 18, 2024

asmeurer commented Mar 18, 2024

honno commented Mar 19, 2024

asmeurer commented Mar 19, 2024

asmeurer commented Mar 20, 2024

asmeurer commented Mar 20, 2024

honno commented Mar 21, 2024

asmeurer commented Mar 21, 2024

CI updates #244

CI updates #244

Conversation

honno commented Mar 15, 2024 • edited Loading

asmeurer commented Mar 15, 2024

honno commented Mar 18, 2024 • edited Loading

asmeurer commented Mar 18, 2024

asmeurer commented Mar 18, 2024

honno commented Mar 19, 2024

asmeurer commented Mar 19, 2024

asmeurer commented Mar 20, 2024

asmeurer commented Mar 20, 2024

honno commented Mar 21, 2024

asmeurer commented Mar 21, 2024

honno commented Mar 15, 2024 •

edited

Loading

honno commented Mar 18, 2024 •

edited

Loading