CI: Run tests on GHA #139

amotl · 2023-01-07T13:13:39Z

Dear Brian,

this is the second iteration on GH-100, using a more native approach than GH-138. On Linux and macOS, it will use a pure-Python environment, while on Windows, it will use a Conda environment for setting up the test sandbox.

I think this patch brings in the best balance of platform/environment coverage vs. resource usage, with ~15 minutes run time in total for all test matrix slots on Linux + macOS combined. Unfortunately, Windows/Conda weighs in with another ~15 minutes, but C'est la vie.

With kind regards,
Andreas.

amotl · 2023-01-07T13:15:48Z

environment-test.yml

-
-  - pip:
-      - herbie-data


I've used the environment-test.yml file, because this file was already pretty close to what was needed for bootstrapping a Conda environment on CI, modulo this change, and adding pytest.

I see that you might have been using it differently, because it previously installed the herbie-data package from PyPI. However, on CI, when running code from a pull request, the code on this very Git ref from the repository should be used, instead of installing the package from PyPI.

Let me know if you can adjust your workflow, or if you have different suggestions.

.github/workflows/tests-conda.yml

amotl · 2023-01-07T13:17:08Z

.github/workflows/tests-python.yml

+    strategy:
+      fail-fast: false
+      matrix:
+        os: ["ubuntu-latest"]
+        python-version: ["3.9", "3.10", "3.11"]
+        # In order to save resources, only run particular
+        # matrix slots on other OS than Linux.
+        include:
+          - os: "macos-latest"
+            python-version: "3.11"
+          #- os: "windows-latest"
+          #  python-version: "3.11"


Let me know if you want to see any adjustments to the test matrix configuration.

I've not added Python 3.8 here, in order to save resources, and, in a similar spirit, the recipe will only run a single version of Python on macOS.

On regular operations, you would toggle to fail-fast: true, in order to save further resources: It will cancel other jobs when a test job fails on one test matrix slot, because it does not make sense to continue running them.

Using fail-fast: false makes sense when working on the workflow recipe itself, in order to get the big picture if specific tests would only fail on Windows, for example, and you don't want them to cancel the jobs running on Linux.

With 1ff1f36, I've configured fail-fast: true, to save resources in the default case, where it does not make sense to run the full test matrix, because the output from a single instance will usually be enough to find out about the root cause of the error.

amotl · 2023-01-07T13:17:20Z

README.md

- Python 3.8, 3.9, or **3.10** (recommended)
+- Python 3.8 to 3.11


I see that you designated Python 3.10 to be the "default" version supported by Herbie, but I think it is safe to bump that to Python 3.11.

- Meta: Explicitly list supported Python versions and operating systems - README: Add Python 3.11 to list of supported Python versions - README: Stop designating a specific Python version

amotl · 2023-01-10T23:46:49Z

After rebasing this patch, the first official test run on GHA succeeded. 🚀

Linux/macOS (pip): https://github.com/blaylockbk/Herbie/actions/runs/3888169482/jobs/6635218502
Windows (Conda): https://github.com/blaylockbk/Herbie/actions/runs/3888169479/jobs/6635218214

Usually, when a single job fails, it does not make sense to continue running the other jobs in the same context. When debugging dedicated issues which only manifest on specific versions of Python, or on specific operating system environments, it is easy to toggle that option back to `fail-fast: false`.

amotl · 2023-01-11T01:22:08Z

tests/test_rap.py

+@pytest.mark.skipif(
+    is_time_between("00:00", "02:30"),
+    reason="Test hibernated between 00:00 and 02:30 UTC, "
+    "because upstream data not available or incomplete at night.",
+)
 def test_rap_aws():


092ed18 mitigates a fluke on the test suite, happening at night time, and resolves GH-144, which has more details about it.

This works. The underlying issue with the test is that Herbie attempts to look for data at today = pd.to_datetime("today").floor("1D") at a time when the model hasn't had a chance to finish running. Setting today = pd.to_datetime("today").floor("1D")-pd.timedelta('12H') wouldn't run into lags with the real-time dissemination.

💔 Did not find ┊ model=rap ┊ product=awp130pgrb ┊ 2023-Jan-11 00:00 UTC F00

blaylockbk · 2023-01-11T17:31:12Z

I belive this is ready to merge. Do you have any objection?

amotl · 2023-01-11T20:13:41Z

I think it is ready to go in. Enjoy, and thanks!

blaylockbk · 2023-01-11T01:25:03Z

tests/test_rap.py

+@pytest.mark.skipif(
+    is_time_between("00:00", "02:30"),
+    reason="Test hibernated between 00:00 and 02:30 UTC, "
+    "because upstream data not available or incomplete at night.",
+)
 def test_rap_aws():


This works. The underlying issue with the test is that Herbie attempts to look for data at today = pd.to_datetime("today").floor("1D") at a time when the model hasn't had a chance to finish running. Setting today = pd.to_datetime("today").floor("1D")-pd.timedelta('12H') wouldn't run into lags with the real-time dissemination.

blaylockbk · 2023-01-11T22:32:11Z

Thanks @amotl for your effort putting this workflow together! I really appreciate you taking the time. I have learned so much from seeing this get put together. It definitely helps streamline development and makes Herbie feel a lot more "grown up."

amotl · 2023-01-11T23:51:39Z

I am happy you see such value in the CI configuration. Thank you for merging.

amotl mentioned this pull request Jan 7, 2023

CI: Run tests on GHA, using Anaconda #138

Closed

amotl commented Jan 7, 2023

View reviewed changes

amotl force-pushed the ci-tests-native branch from f9bb155 to c06751b Compare January 7, 2023 13:22

amotl marked this pull request as ready for review January 7, 2023 13:23

This was referenced Jan 7, 2023

Build some CI with GitHub actions #100

Closed

Documentation: Add configuration for "Read the Docs" #142

Merged

Fix RTD configuration #145

Merged

amotl added 7 commits January 11, 2023 00:29

Tests: Add pytest configuration section to pyproject.toml

42a1c55

Tests: Improve test_hrrr to skip test if wgrib2 is not installed

c888e8a

CI: Run tests on GHA, using Python

7d8aeef

CI: Run tests on GHA, using Python (macOS)

e1a4aee

CI: Run tests on GHA, using Conda (Windows)

fb09ac7

Chore: Update project metadata and README

ee2b2c4

- Meta: Explicitly list supported Python versions and operating systems - README: Add Python 3.11 to list of supported Python versions - README: Stop designating a specific Python version

CI: Run all tests

4d186ea

amotl force-pushed the ci-tests-native branch from 899fb5f to 4d186ea Compare January 10, 2023 23:30

amotl force-pushed the ci-tests-native branch from 1ff1f36 to 05dd3cb Compare January 11, 2023 00:24

amotl mentioned this pull request Jan 11, 2023

FAILED tests/test_rap.py::test_rap_aws - assert None is not None #144

Closed

amotl force-pushed the ci-tests-native branch from dc04e5b to 092ed18 Compare January 11, 2023 01:21

amotl commented Jan 11, 2023

View reviewed changes

Tests: Hibernate RAP AWS latest test between 00:00 and 02:30 UTC

b7aaa70

💔 Did not find ┊ model=rap ┊ product=awp130pgrb ┊ 2023-Jan-11 00:00 UTC F00

amotl force-pushed the ci-tests-native branch from 092ed18 to b7aaa70 Compare January 11, 2023 08:17

blaylockbk approved these changes Jan 11, 2023

View reviewed changes

blaylockbk merged commit cd27c11 into blaylockbk:main Jan 11, 2023

amotl deleted the ci-tests-native branch January 11, 2023 23:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Run tests on GHA #139

CI: Run tests on GHA #139

amotl commented Jan 7, 2023 •

edited

Loading

amotl Jan 7, 2023 •

edited

Loading

amotl Jan 7, 2023

amotl Jan 10, 2023

amotl Jan 7, 2023

amotl commented Jan 10, 2023 •

edited

Loading

amotl Jan 11, 2023

blaylockbk Jan 11, 2023

blaylockbk commented Jan 11, 2023

amotl commented Jan 11, 2023

blaylockbk Jan 11, 2023

blaylockbk commented Jan 11, 2023

amotl commented Jan 11, 2023

		- Python 3.8, 3.9, or 3.10 (recommended)
		- Python 3.8 to 3.11

CI: Run tests on GHA #139

CI: Run tests on GHA #139

Conversation

amotl commented Jan 7, 2023 • edited Loading

amotl Jan 7, 2023 • edited Loading

Choose a reason for hiding this comment

amotl Jan 7, 2023

Choose a reason for hiding this comment

amotl Jan 10, 2023

Choose a reason for hiding this comment

amotl Jan 7, 2023

Choose a reason for hiding this comment

amotl commented Jan 10, 2023 • edited Loading

amotl Jan 11, 2023

Choose a reason for hiding this comment

blaylockbk Jan 11, 2023

Choose a reason for hiding this comment

blaylockbk commented Jan 11, 2023

amotl commented Jan 11, 2023

blaylockbk Jan 11, 2023

Choose a reason for hiding this comment

blaylockbk commented Jan 11, 2023

amotl commented Jan 11, 2023

amotl commented Jan 7, 2023 •

edited

Loading

amotl Jan 7, 2023 •

edited

Loading

amotl commented Jan 10, 2023 •

edited

Loading