Bump to GMT 6.2.0rc1

[seisman]

The first release candidate of GMT 6.2.0 has been released (https://github.com/GenericMappingTools/gmt/releases/tag/6.2.0rc1) today.

Now GMT 6.2.0rc1 is available in the devel channel on conda-forge (https://anaconda.org/conda-forge/gmt/) and can be installed using:

conda install -c conda-forge/label/dev gmt=6.2.0rc1

We need to:

• Bump the GMT version to 6.2.0rc1 used in the CI tests (Bump the GMT version to 6.2.0rc1 in CI #1218)
• Some tests fail and we need to fix them
• Add some new tests for upstream bugs that were fixed in 6.2.0rc1

[weiji14]

Bumping to GMT 6.2.0rc1 will also affect several existing PRs. I think we should go ahead with bumping the version as soon as possible though (so that we can test and fix any new bugs). Is there any PR we would like to merge/finish before bumping PyGMT to use GMT 6.2.0rc1? Otherwise it will get lost under a torrent of PRs relating to fixing broken unit tests 😆

I was actually thinking of updating https://github.com/GenericMappingTools/pygmt/blob/e4cb775346814dce46555a27c0c94f8e0aed6f04/.github/workflows/dvc-diff.yml to have a side-by-side before/after comparison of the images (instead of just having a single new image shown). Unsure if I have time this week to update that workflow (have a presentation and my PhD defence to prep for) but will see.

[seisman]

I was actually thinking of updating e4cb775/.github/workflows/dvc-diff.yml to have a side-by-side before/after comparison of the images (instead of just having a single new image shown).

I can work on this.

[seisman]

@GenericMappingTools/pygmt-maintainers please give #1219 and #1218 a fast review and approval, otherwise, we won't finish the migration before the next 6.2.0rc2 or 6.2.0 release.

[maxrjones]

@GenericMappingTools/pygmt-maintainers please give #1219 and #1218 a fast review and approval, otherwise, we won't finish the migration before the next 6.2.0rc2 or 6.2.0 release.

I know we haven't updated the documentation for the maintainer review process yet, but I think these are good examples of when to use the 'request a review' option for pinging pygmt-maintainers. At least for me, I usually periodically skim the PRs for any that have been marked 'ready for review' or 'final review call' but will notice specific review requests immediately.

[seisman]

I've merged #1218, so now the CI tests are running with GMT 6.2.0rc1, and ~100 tests fail, due to the modern theme changes introduced in GMT 6.2.

What we need to do to make the tests pass again:

1. Install (or upgrade) 6.2.0rc1 in your local environment using conda install -c conda-forge/label/dev gmt=6.2.0rc1
2. Run tests in a specific test file, e.g., pytest pygmt/tests/test_basemap.py
3. If there are any failing tests, you should be able to see the baseline.png and result.png images in the results directory.
4. Compare the baseline.png and result.png images. Do the differences make sense to you?
5. Generate new baseline images using pytest --mpl-generate-path=baseline pygmt/tests/test_basemap.py
6. Copy new baseline images of the failing tests to the pygmt/tests/baseline directory
7. Add the baseline images to DVC and push the images following the contributing guides.
8. Open PRs

[seisman]

Here is a list of test files that contain failing tests. Anyone is welcome to work on them.

• pygmt/helpers/testing.py (Fix pygmt/helpers/testing.py doctest for GMT 6.2.0rc1 #1243 @weiji14)
• pygmt/tests/test_basemap.py (Update baseline images for basemap #1221 @seisman)
• pygmt/tests/test_coast.py (Update fig.coast baseline images for GMT 6.2.0rc1 #1222 @seisman)
• pygmt/tests/test_colorbar.py (Update fig.colorbar baseline images for GMT 6.2.0rc1 #1229 @seisman)
• pygmt/tests/test_config.py (Update pygmt.config baseline images for GMT 6.2.0rc1 #1235 @weiji14)
• pygmt/tests/test_contour.py (Update fig.contour baseline images for GMT 6.2.0rc1 #1234 @seisman)
• pygmt/tests/test_figure.py (Update fig.shift_origin baseline images for GMT 6.2.0rc1 #1237 @seisman)
• pygmt/tests/test_grd2cpt.py (Update fig.grd baseline images for GMT 6.2.0rc1 #1224 @weiji14)
• pygmt/tests/test_grdcontour.py (Update fig.grd baseline images for GMT 6.2.0rc1 #1224 @weiji14)
• pygmt/tests/test_grdfilter.py (Update fig.grd baseline images for GMT 6.2.0rc1 #1224 @weiji14)
• pygmt/tests/test_inset.py (Update fig.inset baseline images for GMT 6.2.0rc1 #1230 @seisman)
• pygmt/tests/test_legend.py (Update fig.legend baseline images for GMT 6.2.0rc1 #1233 @seisman)
• pygmt/tests/test_logo.py (Update fig.logo baseline images for GMT 6.2.0rc1 #1223 @seisman)
• pygmt/tests/test_makecpt.py (Update makecpt baseline images for GMT 6.2.0rc1 #1231 @meghanrjones)
• pygmt/tests/test_meca.py (Update fig.meca baseline images for GMT 6.2.0rc1 #1225 @seisman)
• pygmt/tests/test_plot.py (Update fig.plot baseline images for GMT 6.2.0rc1 #1227 @seisman)
• pygmt/tests/test_plot3d.py (Update fig.plot3d baseline images for GMT 6.2.0rc1 #1228 @seisman)
• pygmt/tests/test_rose.py (Update fig.rose baseline images for GMT 6.2.0rc1 #1226 @michaelgrund)
• pygmt/tests/test_solar.py (Update fig.solar baseline images for GMT 6.2.0rc1 #1232 @seisman)

Not failing, but wrong according to #1217 (comment)

• pygmt/tests/test_subplot.py
• pygmt/tests/test_text.py
• pygmt/tests/test_wiggle.py

[weiji14]

I'll handle all the test_grd*.py tests in a single PR if that's ok.

[seisman]

I'll handle all the test_grd*.py tests in a single PR if that's ok.

OK to me.

[weiji14]

Cool, can we also agree on a standardized PR title? E.g. "Update fig.??? baseline images for GMT 6.2.0rc1"?

[seisman]

Cool, can we also agree on a standardized PR title? E.g. "Update fig.??? baseline images for GMT 6.2.0rc1"?

Good.

[weiji14]

I think we may be encountering something flaky in PyGMT's test suite. There's 3 separate cases (opened by 3 different people) where using pytest pygmt/tests/test_somemodule.py passes on a specific module but make test fails for the whole test suite:

• fig.solar Update fig.solar baseline images for GMT 6.2.0rc1 #1232 (comment)
• fig.grdview Migrate Figure.grdview tests to dvc #1154 (comment)
• pygmt.makecpt Update makecpt baseline images for GMT 6.2.0rc1 #1231 (comment)

This reminds me of https://github.com/GenericMappingTools/pygmt/pull/476/files#r453429872 (which motivated the use of a @pytest.mark.runfirst decorator, @seisman will remember this). The 3 cases above may well be 3 separate issues, but they seem to be too much of a coincidence. We'll need to track down whether this is an upstream GMT issue, or something wrong with the test suite on PyGMT's side, and this will require quite a bit of detective work.

[maxrjones]

I think we may be encountering something flaky in PyGMT's test suite. There's 3 separate cases (opened by 3 different people) where using pytest pygmt/tests/test_somemodule.py passes on a specific module but make test fails for the whole test suite:

* `fig.solar` [#1232 (comment)](https://github.com/GenericMappingTools/pygmt/pull/1232#issuecomment-825230981)

* `fig.grdview` [#1154 (comment)](https://github.com/GenericMappingTools/pygmt/pull/1154#discussion_r618825105)

* `pygmt.makecpt` [#1231 (comment)](https://github.com/GenericMappingTools/pygmt/pull/1231#issuecomment-825259466)

This reminds me of https://github.com/GenericMappingTools/pygmt/pull/476/files#r453429872 (which motivated the use of a @pytest.mark.runfirst decorator, @seisman will remember this). The 3 cases above may well be 3 separate issues, but they seem to be too much of a coincidence. We'll need to track down whether this is an upstream GMT issue, or something wrong with the test suite on PyGMT's side, and this will require quite a bit of detective work.

For makecpt, make test fails if run after the examples in pygmt/clib/session.py are tested. Flaky indeed.

edit: I have not yet been able to figure out a solution. The two makecpt tests fail if there is a docstring example that imports pygmt and instantiates a figure (e.g., extract_region() in pygmt/clib/session.py and pygmt/src/grdfilter.py) and is tested before pygmt/tests/test_makecpt.py.

[seisman]

The two makecpt tests fail if there is a docstring example that imports pygmt and instantiates a figure (e.g., extract_region() in pygmt/clib/session.py and pygmt/src/grdfilter.py) and is tested before pygmt/tests/test_makecpt.py.

These flaky tests are difficult to debug. Perhaps we can use one of the "flaky plugins" to rerun the failing tests as a workaround. I think they should pass, no?

[seisman]

Just tried pytest-rerunfailures. Not lucky. It re-runs a test immediately after it fails, rather than re-running all failing tests after all tests finish.

[weiji14]

Yeah, from experience, I found the listed pytest flaky plugins to be a bit of a hit and miss in PyGMT's case. Also tried pytest-reverse and I'm getting lots of failures on fig.test too. The main problem (I think) is that PyGMT is using a single GMT session for everything (partly related to #867), and this can cause settings in one figure instance to pollute other figures if we're not careful (e.g. the bug in #733)

[maxrjones]

I understand the original logic behind a single GMT session for all tests in #327 (comment). Still, I don't expect that users will be attempting to use the entire PyGMT library in a single session, which is the goal of the test suite. So I think it would be worth revisiting this decision. Could it be possible to periodically test the examples/tutorials against baseline images to ensure that producing multiple plots in a single session is consistent and have the unit tests each use individual sessions?

[weiji14]

I understand the original logic behind a single GMT session for all tests in #327 (comment). Still, I don't expect that users will be attempting to use the entire PyGMT library in a single session, which is the goal of the test suite. So I think it would be worth revisiting this decision. Could it be possible to periodically test the examples/tutorials against baseline images to ensure that producing multiple plots in a single session is consistent and have the unit tests each use individual sessions?

Ok, I've opened up a separate issue to decide on whether PyGMT should have single/separate GMT session(s) at #1242. Let's continue the test flakiness discussion there.

For the purposes of quickly resolving this GMT 6.2.0rc1 baseline image update issue, specifically for makecpt at #1231 and fig.solar at #1232 (ignoring fig.grdview since that's about dvc migration), we'll need a quick workaround to get the tests green again. Do we want to 1) use @pytest.mark.xfail or 2) get the image that make test expects from https://github.com/GenericMappingTools/pygmt/actions/runs/787380280 under Artifacts and upload it to dvc or 3) something else?

[maxrjones]

I understand the original logic behind a single GMT session for all tests in #327 (comment). Still, I don't expect that users will be attempting to use the entire PyGMT library in a single session, which is the goal of the test suite. So I think it would be worth revisiting this
For the purposes of quickly resolving this GMT 6.2.0rc1 baseline image update issue, specifically for makecpt at #1231 and fig.solar at #1232 (ignoring fig.grdview since that's about dvc migration), we'll need a quick workaround to get the tests green again. Do we want to 1) use @pytest.mark.xfail or 2) get the image that make test expects from https://github.com/GenericMappingTools/pygmt/actions/runs/787380280 under Artifacts and upload it to dvc or 3) something else?

Since option 2 will reverse the problem (i.e., fail with pytest pygmt/tests/test_makecpt.py), I am in favor of option 1 as an interim solution.

[seisman]

Option 1 👍

[seisman]

A little progress on the flaky tests.

Similar to makecpt() tests in #1231, the Figure.soloar() tests (#1232) pass when running the single test, but fail when running with pygmt/tests/test_session_management.py. To reproduce it, you can check out the branch in #1232 and run:

pytest pygmt/tests/test_session_management.py pygmt/tests/test_solar.py

If I delete the file pygmt/tests/test_session_management.py and run the full tests with make test, the Figure.solar() tests pass, but some other tests fail (in a good way). Here are the list of new failing tests after deleting pygmt/tests/test_session_management.py:

FAILED ../pygmt/tests/test_subplot.py::test_subplot_basic_frame
FAILED ../pygmt/tests/test_subplot.py::test_subplot_direct
FAILED ../pygmt/tests/test_subplot.py::test_subplot_autolabel_margins_title
FAILED ../pygmt/tests/test_subplot.py::test_subplot_clearance_and_shared_xy_axis_layout
FAILED ../pygmt/tests/test_text.py::test_text_single_line_of_text
FAILED ../pygmt/tests/test_text.py::test_text_multiple_lines_of_text
FAILED ../pygmt/tests/test_text.py::test_text_input_single_filename
FAILED ../pygmt/tests/test_text.py::test_text_input_remote_filename
FAILED ../pygmt/tests/test_text.py::test_text_input_multiple_filenames
FAILED ../pygmt/tests/test_text.py::test_text_position
FAILED ../pygmt/tests/test_text.py::test_text_position_offset_with_line
FAILED ../pygmt/tests/test_text.py::test_text_angle_30
FAILED ../pygmt/tests/test_text.py::test_text_font_bold
FAILED ../pygmt/tests/test_text.py::test_text_fill
FAILED ../pygmt/tests/test_text.py::test_text_pen
FAILED ../pygmt/tests/test_text.py::test_text_round_clearance
FAILED ../pygmt/tests/test_text.py::test_text_justify_bottom_right_and_top_left
FAILED ../pygmt/tests/test_text.py::test_text_justify_parsed_from_textfile
FAILED ../pygmt/tests/test_text.py::test_text_angle_font_justify_from_textfile
FAILED ../pygmt/tests/test_text.py::test_text_transparency
FAILED ../pygmt/tests/test_text.py::test_text_varying_transparency
FAILED ../pygmt/tests/test_text.py::test_text_nonstr_text
FAILED ../pygmt/tests/test_wiggle.py::test_wiggle

I didn't check all the tests, but it seems some baseline images are incorrect, but the tests on master branch pass. For example, for the test test_text_transparency, the current baseline image is incorrect:

Current baseline	Expected baseline

As far as I understand it, test files like pygmt/tests/test_session_management.py or docstrings that call import pygmt (e.g., pygmt/src/grdfilter.py) create new sessions or nested sessions, which cause problems for us. Since we don't see these problems with GMT 6.1.1, perhaps some upstream bugs (e.g., memory leakages) in GMT 6.2.0rc1?

[seisman]

Actually, on the master branch, the Figure.text() tests fail when running pytest pygmt/tests/test_text.py, but pass with make test.

[seisman]

Here is a minimal example to reproduce the flaky tests we're having:

from pygmt import Figure
from pygmt.clib import Session
from pygmt.session_management import begin, end

# check the current value of MAP_FRAME_AXES
with Session() as lib:
    lib.call_module("get", "MAP_FRAME_AXES")
# plot the first image
fig = Figure()
fig.basemap(region=[0, 10, 10, 20], projection="X10c", frame=True)
fig.savefig("before.png")

# end the global session
end()
# start a new global session
begin()

# check the current value of MAP_FRAME_AXES
with Session() as lib:
    lib.call_module("get", "MAP_FRAME_AXES")
# plot the second image
fig = Figure()
fig.basemap(region=[0, 10, 10, 20], projection="X10c", frame=True)
fig.savefig("after.png")

Just in case you don't know it:

• import pygmt calls the begin() function, which is just the wrapper of gmt begin pygmt-session
• when the script exits, the end() function is called, which is the wrapper of gmt end

The output of the two "gmt get" calls is:

auto
WESNZ

The resulting images are:

Before	After

Apparently, these two images diff due to the differences of MAP_FRAME_AXES settings (auto vs WESNZ). I think it's an upstream GMT bug. Please see if the minimal example can help you debug GMT. @PaulWessel @meghanrjones

[maxrjones]

Thanks for the the minimal example @seisman. I will see what I can find about this.

[maxrjones]

No solution yet but it seems to be an issue with GMT_keyword_updated that was implemented as part of GenericMappingTools/gmt#3344. At the start of the second global session, GMT_keyword_updated is set to true for the options that were updated in gmtinit_conf_modern_override() in the first global session, which leads to a session-level gmt.conf file being written that contains the classic versions of the theme dependent parameters.

[maxrjones]

GenericMappingTools/gmt#5178 fixes the minimal example and make these tests fail (some now in a proper way):

FAILED ../pygmt/tests/test_basemap.py::test_basemap_polar
FAILED ../pygmt/tests/test_makecpt.py::test_makecpt_categorical
FAILED ../pygmt/tests/test_makecpt.py::test_makecpt_cyclic
FAILED ../pygmt/tests/test_solar.py::test_solar_terminators
FAILED ../pygmt/tests/test_solar.py::test_solar_set_terminator_datetime[terminator_datetime_string]
FAILED ../pygmt/tests/test_solar.py::test_solar_set_terminator_datetime[terminator_datetime1]
FAILED ../pygmt/tests/test_solar.py::test_solar_default_terminator
FAILED ../pygmt/tests/test_subplot.py::test_subplot_basic_frame
FAILED ../pygmt/tests/test_subplot.py::test_subplot_direct
FAILED ../pygmt/tests/test_subplot.py::test_subplot_autolabel_margins_title
FAILED ../pygmt/tests/test_subplot.py::test_subplot_clearance_and_shared_xy_axis_layout
FAILED ../pygmt/tests/test_text.py::test_text_single_line_of_text
FAILED ../pygmt/tests/test_text.py::test_text_multiple_lines_of_text
FAILED ../pygmt/tests/test_text.py::test_text_input_single_filename
FAILED ../pygmt/tests/test_text.py::test_text_input_remote_filename
FAILED ../pygmt/tests/test_text.py::test_text_input_multiple_filenames
FAILED ../pygmt/tests/test_text.py::test_text_position
FAILED ../pygmt/tests/test_text.py::test_text_position_offset_with_line
FAILED ../pygmt/tests/test_text.py::test_text_angle_30
FAILED ../pygmt/tests/test_text.py::test_text_font_bold
FAILED ../pygmt/tests/test_text.py::test_text_fill
FAILED ../pygmt/tests/test_text.py::test_text_pen
FAILED ../pygmt/tests/test_text.py::test_text_round_clearance
FAILED ../pygmt/tests/test_text.py::test_text_justify_bottom_right_and_top_left
FAILED ../pygmt/tests/test_text.py::test_text_justify_parsed_from_textfile
FAILED ../pygmt/tests/test_text.py::test_text_angle_font_justify_from_textfile
FAILED ../pygmt/tests/test_text.py::test_text_transparency
FAILED ../pygmt/tests/test_text.py::test_text_varying_transparency
FAILED ../pygmt/tests/test_text.py::test_text_nonstr_text
FAILED ../pygmt/tests/test_wiggle.py::test_wiggle

But, the two makecpt() tests still fail and I get a fatal error on pygmt/tests/test_grdtrack.py::test_grdtrack_input_dataframe_and_dataarray. I will merge GenericMappingTools/gmt#5178 since it is a needed correction. Then it will be necessary to sort out the remaining problems.

[weiji14]

Thanks @seisman and @meghanrjones for tracking this down and fixing the bug. Sounds like we might need a GMT 6.2.0rc2? Let's put an xfail for the fig.solar tests at #1232 for now @seisman to get the tests passing (and close this issue). We can then track/fix the false positive passing tests (for subplot, text, and wiggle) later in a separate issue/PR.

[maxrjones]

I agree 6.2.0rc2 will be needed soon. Even though the gmt-dev CI tests are passing, I am still concerned about my local fatal error in pygmt/tests/test_grdtrack.py::test_grdtrack_input_dataframe_and_dataarray and would like to figure that out before releasing rc2. It would be helpful to know if anyone else gets a similar issue when testing using the gmt-dev branch. This is my setup:

>>> pygmt.show_versions()
PyGMT information:
  version: v0.3.2.dev123+gb855310b.d20210429
System information:
  python: 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:20)  [Clang 11.0.1 ]
  executable: /Users/meghanj/Sandbox/anaconda-03/anaconda3/envs/pygmt-dev/bin/python
  machine: macOS-11.3-x86_64-i386-64bit
Dependency information:
  numpy: 1.20.2
  pandas: 1.2.4
  xarray: 0.17.0
  netCDF4: 1.5.6
  packaging: 20.9
  ghostscript: 9.53.3
  gmt: 6.2.0_a6980cb_2021.04.28

I plan to look into this more tomorrow, but in the meantime this is the specific error that I get when trying to run output = grdtrack(points=dataframe, grid=dataarray, newcolname="bathymetry"):
python grdtrack-test.py Assertion failed: (node < G->header->size), function gmt_bcr_get_z_fast, file ../src/gmt_bcr.c, line 256. Abort trap: 6

[weiji14]

Agree on finding out what is causing the grdtrack fatal error before GMT 6.2.0rc2, I've just built the latest GMT from master (GenericMappingTools/gmt@a6980cb) and will try and see what's up. I'll probably work on it in the grdrack refactor PR at #1189.

Edit: I can't reproduce the fatal error on Linux, and the 'macOS-11.0 - GMT master' test didn't seem to crash either at https://github.com/GenericMappingTools/pygmt/runs/2463360992?check_suite_focus=true#step:15:272.

[maxrjones]

Agree on finding out what is causing the grdtrack fatal error before GMT 6.2.0rc2, I've just built the latest GMT from master (GenericMappingTools/gmt@a6980cb) and will try and see what's up. I'll probably work on it in the grdrack refactor PR at #1189.

Edit: I can't reproduce the fatal error on Linux, and the 'macOS-11.0 - GMT master' test didn't seem to crash either at https://github.com/GenericMappingTools/pygmt/runs/2463360992?check_suite_focus=true#step:15:272.

Thanks for checking. It was a silly mistake on my part in continuing on in debug mode even after being done with debugging so I ran into an assert statement 🤦‍♀️

[weiji14]

With #1232 merged in, I think we can close this GMT 6.2.0rc1 issue. We should continue to track the test flakiness issue over at #1242, and open a separate "Bump to GMT 6.2.0rc2" issue later on to fix problematic tests.