grdimage shading fails with external matrix

[seisman]

The issue was first reported by @liamtoney in GenericMappingTools/pygmt#364. Using the PyGMT script in that issue, he got the following error:

grdimage [ERROR]: Passing zmax <= zmin prevents automatic CPT generation!
grdimage [ERROR]: Failed to read CPT geo.

After some debugging, it seems that GMT can correctly read the matrix (i.e., the earth relief data) and can plot it correctly if shading=False (i.e., no -I+d in the command line). However, if shading=True is used (i.e., grdimage -I+d), GMT reports the error above, possibly because the matrix in memory is modified/freed/re-initialized when calling grdgradient.

---

The issue can be partially reproduced using the following C codes. The C code below is modified from GMT's own test code testapi_matrix_plot.c. It reads a 2d matrix and plots the matrix using grdimage.

#include "gmt.h"

int main () {
	void *API = NULL;                /* The API control structure */
	struct GMT_MATRIX *M = NULL;    /* Structure to hold input matrix */
	char input[GMT_VF_LEN] = {""};	/* String to hold virtual input filename */
	char args[128] = {""};         	/* String to hold module command arguments */
	/* Initialize the GMT session */
	API = GMT_Create_Session ("test", 2U, GMT_SESSION_EXTERNAL, NULL);
	M = GMT_Read_Data (API, GMT_IS_MATRIX, GMT_IS_FILE, GMT_IS_SURFACE, GMT_READ_NORMAL, NULL, "2d-matrix.txt", NULL);
	/* Associate our matrix with a virtual file */
	GMT_Open_VirtualFile (API, GMT_IS_GRID|GMT_VIA_MATRIX, GMT_IS_SURFACE, GMT_IN, M, input);

	/* Prepare the module arguments */
	sprintf (args, "%s -JX6i -P -Baf -Cgeo", input);
	/* Call the grdimage module */
	GMT_Call_Module (API, "grdimage", GMT_MODULE_CMD, args);

	/* Close the virtual file */
	GMT_Close_VirtualFile (API, input);
	/* Destroy session */
	if (GMT_Destroy_Session (API)) return EXIT_FAILURE;
};

The data 2d-matrix.txt is the 60m earth relief data, converted from netCDF to matrix format using following Python code. The data is attached 2d-matrix.txt.

import numpy as np
import pygmt
er = pygmt.datasets.load_earth_relief("60m")
np.savetxt("2d-matrix.txt", np.flip(er.data, axis=0), fmt='%.1f')

The C code is compiled using

gcc testapi_matrix_plot.c $(gmt-config --cflags --libs)

The output image looks correct. The coordinates are incorrect, simply because I didn't pass the geographic information.

Changing the grdimage arguments to:

sprintf (args, "%s -JX6i -P -Baf -Cgeo -I+d", input);

to enable shading, then the output image is:

This is what I expect:

[joa-quim]

Hmmm, unless I'm missing something I think it's the user responsibility to pass arrays with order that GMT is expecting. I do that all the time in Julia and never had this problem. Well not counting the countless days working in a way that the memory layout can be set, and changed, to work with this.
Even netCDF can be TOPdown or BOTTOMup.

[seisman]

I didn't mean that the first figure is wrong. The flipping Y axis is simply because I didn't spend any time to re-sort the matrix. The first figure shows that the GMT_Read_Data can correctly read the matrix.

The main issue here is the second figure. After adding -I+d, the output figure is not what we expect.

[joa-quim]

Does it give the same thing if you create the ascii version with grd2xyz?

[seisman]

Sorry, I forgot to attach the data in my first comment 2d-matrix.txt.

[seisman]

Does it give the same thing if you create the ascii version with grd2xyz?

grd2xyz outputs XYZ triples or Z values, but GMT_Read_Data expects a matrix here (GMT_IS_MATRIX is passed to GMT_Read_Data).

[joa-quim]

Sorry but still fail to see that the problem is in GMT. As I said either in Mirone or Julia I pass grids back and forth and when such cases arise it was always my fault.
If the array has the correct the correct order, the illumination must have the same order. And speaking on orders, the header struct has a mem_layout member to deal with issues of this type.

[seisman]

@joa-quim It's not about the orders. I've updated the issue description. Hopefully it's more clear to read now.

[PaulWessel]

grdgradient does recycle the input grid and places gradients there except when we are not allowed:

`new_grid = gmt_set_outgrid (GMT, Ctrl->In.file, separate, Surf, &Out);	/* true if input is a read-only array */`

So since In.file is a @gmtapi@ memory reference it allocates a separate grid. So not immediately obvious why/where it overwrites the input grid. But clearly it does. @seisman, if you are able to walk through this in Xcode or similar you may be able to verify that it is not allocating a new grid. E.g., new_grid should be true in this case.

I can help with debug later but busy day with zoom with all Chairs and Provost. Shit-storm is brewing with Governor planning to take all vacant positions away (we have 5 with active searches all frozen...).

[seisman]

new_grid is true, but this line returns Grid_orig[0] with data=0.

gmt/src/grdimage.c

Line 891 in ddd829b

if (GMT_Read_Data (API, GMT_IS_GRID, GMT_IS_FILE, GMT_IS_SURFACE, GMT_DATA_ONLY, wesn, Ctrl->In.file[k], Grid_orig[k]) == NULL) { /* Get grid data */