Run Continuous Integration tests on Github Actions #475

weiji14 · 2020-06-10T04:09:04Z

Description of proposed changes

Consolidate all our Linux/macOS/Windows unit tests into one .github/workflows/ci_tests.yaml file using a matrix build specification ~~(instead of having .travis.yml and .azure-pipelines.yml)~~. Will keep Travis CI because it's needed for the PyPI releases, but will disable Azure Pipelines for now.

Original attempt was made in weiji14#88, see some of the previous discussion/reviews there.

Fixes #

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If adding new functionality, add an example to docstrings or tutorials.

Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

See https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategyfail-fast Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Try just caching ~/.gmt/cache and ~/.gmt/server.

Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

weiji14 · 2020-06-10T04:26:12Z

Good work on the wget workaround @seisman! Tests seem to be failing on the new earth_relief grids only, should fix that first.

seisman · 2020-06-10T04:31:19Z

Good work on the wget workaround @seisman! Tests seem to be failing on the new earth_relief grids only, should fix that first.

FYI, we can't use curl to download files on Linux and windows, as it always timeouts. It's not clear to me why wget works and curl doesn't. GMT relies on the curl library to download remote files with a 10-s timeout, and of course GMT fails just like the curl command.

weiji14 · 2020-07-09T23:12:30Z

I think we shouldn't cache conda packages, to make the actions more readable.

Sounds good. We can always add it back in later if we decide a cache for conda packages is needed.

seisman · 2020-07-09T23:14:14Z

I think we shouldn't cache conda packages, to make the actions more readable.

Sounds good. We can always add it back in later if we decide a cache for conda packages is needed.

OK. I'll revert it after the CI jobs pass.

.github/workflows/ci_tests.yaml

seisman · 2020-07-09T23:33:36Z

.github/workflows/ci_tests.yaml

+        run: |
+          if [ "$RUNNER_OS" == "Windows" ]; then choco install wget; fi  # install wget on Windows
+          mkdir ~/.gmt ~/.gmt/cache ~/.gmt/server
+          wget --no-check-certificate https://oceania.generic-mapping-tools.org/gmt_hash_server.txt -P ~/.gmt/server/
+          for data in earth_relief_01d.grd earth_relief_30m.grd earth_relief_10m.grd; do
+            wget --no-check-certificate https://oceania.generic-mapping-tools.org/${data} -P ~/.gmt/server/
+          done


We need to change these lines when bumping to 6.1. In 6.1, the earth relief data are stored in ~/.gmt/server/earth/earth_relief/ directory.

seisman · 2020-07-09T23:34:12Z

.github/workflows/ci_tests.yaml

+      # Download remote files, if not already cached
+      - name: Download remote data (macOS)
+        shell: bash -l {0}
+        run: gmt which -Gu @earth_relief_01d @earth_relief_30m @earth_relief_10m @ridge.txt @Table_5_11.txt @tut_bathy.nc @tut_quakes.ngdc @tut_ship.xyz @usgs_quakes_22.txt


Need to change gmt which -Ga in GMT 6.1.

.github/workflows/ci_tests.yaml

seisman

I'm OK with the current changes in this PR. Please see if you want to make more changes.

weiji14 · 2020-07-10T01:36:39Z

Makefile

@@ -1,7 +1,7 @@
 # Build, package, test, and clean
 PROJECT=pygmt
 TESTDIR=tmp-test-dir-with-unique-name
-PYTEST_ARGS=--cov-config=../.coveragerc --cov-report=term-missing --cov=$(PROJECT) --doctest-modules -v --mpl --mpl-results-path=results --pyargs
+PYTEST_ARGS=--cov-config=../.coveragerc --cov-report=term-missing --cov=$(PROJECT) --doctest-modules -v --mpl --mpl-results-path=results --pyargs ${PYTEST_EXTRA}


This won't work, because we didn't declare the $PYTEST_EXTRA environment variable. Also need to remove the $PYTEST_EXTRA string from the ci_tests.yaml file.

I think it works. It doesn't have to be an environmental variable. make test and make test PYTEST_EXTRA="-r P" are using different pytext arguments.

Now although the tests all pass, we actually can see more "errors" by using -r P:

______________________ test_call_module_invalid_arguments ______________________ ----------------------------- Captured stderr call ----------------------------- gmtinfo [ERROR]: Error for input file: No such file (bogus-data.bla) ________________________ test_call_module_error_message ________________________ ----------------------------- Captured stderr call ----------------------------- gmtinfo [ERROR]: Error for input file: No such file (bogus-data.bla) __________________________ test_extract_region_fails ___________________________ ----------------------------- Captured stderr call ----------------------------- pygmt-session [ERROR]: No hidden PS file found ____________________________ test_get_default_fails ____________________________ ----------------------------- Captured stderr call ----------------------------- pygmt-session [ERROR]: Syntax error: Unrecognized keyword NOT_A_VALID_NAME ________________________ test_text_nonexistent_filename ________________________ ----------------------------- Captured stderr call ----------------------------- text [ERROR]: Error for input file: No such file (notexist.txt) _______________________________ test_which_fails _______________________________ ----------------------------- Captured stderr call ----------------------------- gmtwhich [ERROR]: File 5a9f5f5cb31c4bb99525d0d82eaf0fc6 not found!

Isn't it because the CI (Travis and GH Actions) still uses make test PYTEST_EXTRA="-r P"? If we run make test locally after checking out this branch, it probably won't work.

Yes. Look at the captured stderr above:

gmtinfo [ERROR]: Error for input file: No such file (bogus-data.bla)

This is a GMT error message, but I believe it's already corrected captured in the pygmt test.

I believe the initial idea of PYTEST_EXTRA is that, when we or other users run make test locally, we won't see these "error" messages, as they're a little confusing (tests pass but there are "errors"). But in CI, make test PYTEXT_EXTRA="-r P" gives more verbose messages, useful for debugging.

Then we don't actually need to change the Makefile. You'll see that PROJECT is declared before using $(PROJECT), but we don't declare PYTEST_EXTRA before using $(PYTEST_EXTRA), it is effectively blank.

If you run make test, then PYTEST_EXTRA is blank; if you run make test PYTEST_EXTRA="-r P", then variable PYTEST_EXTRA is defined in shell, and passed into Makefile.

make test runs:

cd tmp-test-dir-with-unique-name; pytest --cov-config=../.coveragerc --cov-report=term-missing --cov=pygmt --doctest-modules -v --mpl --mpl-results-path=results --pyargs pygmt

and make test PYTEST_EXTRA="-r P" runs:

cd tmp-test-dir-with-unique-name; pytest --cov-config=../.coveragerc --cov-report=term-missing --cov=pygmt --doctest-modules -v --mpl --mpl-results-path=results --pyargs -r P pygmt

Right, I think I get it now. It wasn't working properly before, but now it does.

.github/workflows/ci_tests.yaml

weiji14 · 2020-07-10T02:19:16Z

Thanks for all the help @seisman. The macOS tests seem to be a bit slower on Github Actions ~10min compared to before on Azure Pipelines ~7min. WIndows seems slightly faster sometimes. I'm sure we'll improve this once we get GMT 6.1 working properly, will merge this now 🎉

weiji14 and others added 10 commits June 5, 2020 15:04

Run Continuous Integration tests on Github Actions

4c0fd39

Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Install make earlier

31340ee

Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Set Linux/MacOS environment variables properly

9a4e7bc

Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Install pytest and other dev dependencies

51472dc

Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Don't fail-fast and cancel all jobs

9c2e393

See https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategyfail-fast Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Update setup-miniconda Github Action from v1 to v1.3.1

2220702

Signed-off-by: Wei Ji <weiji.leong@vuw.ac.nz>

Don't set any environment variables for all OSes

6936b3c

Cache GMT tutorial files in ${{ github.workspace }}/.gmt on 20200605

6906426

Try just caching ~/.gmt/cache and ~/.gmt/server.

Fix typo python_version to python-version

d2ce09b

Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

Fix python_version to python-version

e407081

weiji14 added the maintenance Boring but important stuff for the core devs label Jun 10, 2020

weiji14 mentioned this pull request Jun 10, 2020

Run Continuous Integration tests on Github Actions weiji14/pygmt#88

Closed

5 tasks

Disable tests to enable caches

841f937

vercel bot temporarily deployed to Preview June 10, 2020 05:53 Inactive

Re-enable tests to check caches

9fd73f0

vercel bot temporarily deployed to Preview June 10, 2020 06:50 Inactive

vercel bot temporarily deployed to Preview June 10, 2020 06:56 Inactive

seisman added 3 commits June 10, 2020 11:12

Change github action name

1a23b55

Simplify job name

0ba3d7e

The versioneer pacakge need tags to determine the PyGMT version

c744db9

vercel bot temporarily deployed to Preview June 10, 2020 15:15 Inactive

Change runner.os to matrix.os

3a06afc

vercel bot temporarily deployed to Preview June 10, 2020 15:44 Inactive

seisman added 2 commits June 10, 2020 11:58

Improve the docs

05c225f

Add a cron job for daily tests

8bd8a66

vercel bot temporarily deployed to Preview June 10, 2020 16:03 Inactive

weiji14 mentioned this pull request Jun 22, 2020

Github Actions CI for testing GMT master branch #485

Merged

5 tasks

seisman mentioned this pull request Jun 29, 2020

Fix several failures due to updates of earth relief data #498

Merged

5 tasks

Disable Azure Pipeline CI on Windows and macOS, but keep Style Checks on

b4c7492

vercel bot temporarily deployed to Preview July 9, 2020 23:07 Inactive

Do not cache the conda packages

b6445d0

vercel bot temporarily deployed to Preview July 9, 2020 23:26 Inactive

seisman reviewed Jul 9, 2020

View reviewed changes

.github/workflows/ci_tests.yaml Show resolved Hide resolved

seisman added 2 commits July 9, 2020 20:15

Add the status badge to README

b731f0c

Test Python 3.6-3.8

5b65455

vercel bot temporarily deployed to Preview July 10, 2020 00:15 Inactive

vercel bot temporarily deployed to Preview July 10, 2020 00:36 Inactive

Let the github actions run in all PRs

36fa209

seisman force-pushed the gh-actions-ci branch from 798fd44 to 36fa209 Compare July 10, 2020 00:48

vercel bot temporarily deployed to Preview July 10, 2020 00:48 Inactive

Use PYTEST_EXTRA in Makefile

a5af2dc

vercel bot temporarily deployed to Preview July 10, 2020 01:22 Inactive

seisman marked this pull request as ready for review July 10, 2020 01:22

seisman approved these changes Jul 10, 2020

View reviewed changes

weiji14 commented Jul 10, 2020

View reviewed changes

.github/workflows/ci_tests.yaml Show resolved Hide resolved

weiji14 merged commit 158ef09 into master Jul 10, 2020

weiji14 deleted the gh-actions-ci branch July 10, 2020 02:26

seisman added this to the 0.2.x milestone Jul 10, 2020

seisman added this to In progress in Release v0.2.x via automation Jul 10, 2020

This was referenced Jul 10, 2020

Patch issues with Github Actions CI cache #511

Merged

The GitHub Actions caches don't work for PRs #523

Closed

seisman moved this from In progress to Done in Release v0.2.x Jul 23, 2020

weiji14 mentioned this pull request Sep 5, 2020

Make test suite more efficient to run in 10 minutes #584

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run Continuous Integration tests on Github Actions #475

Run Continuous Integration tests on Github Actions #475

weiji14 commented Jun 10, 2020 •

edited

Loading

weiji14 commented Jun 10, 2020

seisman commented Jun 10, 2020

weiji14 commented Jul 9, 2020

seisman commented Jul 9, 2020

seisman Jul 9, 2020

seisman Jul 9, 2020

seisman left a comment

weiji14 Jul 10, 2020

seisman Jul 10, 2020

seisman Jul 10, 2020

weiji14 Jul 10, 2020

seisman Jul 10, 2020 •

edited

Loading

weiji14 Jul 10, 2020

seisman Jul 10, 2020

seisman Jul 10, 2020

weiji14 Jul 10, 2020

weiji14 commented Jul 10, 2020

Run Continuous Integration tests on Github Actions #475

Run Continuous Integration tests on Github Actions #475

Conversation

weiji14 commented Jun 10, 2020 • edited Loading

weiji14 commented Jun 10, 2020

seisman commented Jun 10, 2020

weiji14 commented Jul 9, 2020

seisman commented Jul 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seisman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seisman Jul 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weiji14 commented Jul 10, 2020

weiji14 commented Jun 10, 2020 •

edited

Loading

seisman Jul 10, 2020 •

edited

Loading