Static server issues #7879

PaulWessel · 2023-10-07T07:21:56Z

These changes simplifies the handling of USERDIR and CACHEDIR and now things seems to work if

The environment sets GMT_DATA_SERVER=static
Or, cmake is configured to set iv via gmtest.in

Getting 32-33 failures if running one test at the time so that the downloads dont interfere.

Still stuff not working. Looks like setting expert GMT_DATA_SERVER=static works differently (better?) than having gmtest.in do it.

PaulWessel · 2023-10-07T07:24:20Z

Much harder to get --GMT_DATA_SERVER=static to reliably work since those options are parsed quite late in the initialisation. I will be testing this today a bit.

PaulWessel · 2023-10-08T12:57:23Z

Seems when tests are run in parallel there are hiccups where $GMT_DATA_SERVER is not detected and it defaults to oceania and it writes some files into .gmt/server and .gmt/cache. If I run the tests serially, one at the time then every remote file is saved under the static directory. I suspect something with sub-shells and some parts of scripts being run without $GMT_DATA_SERVER set. I dont know what might happen but clearly many files are being written to .gmt/server even though we have

[[ -z "${GMT_DATA_SERVER}" ]] && export GMT_DATA_SERVER=static

in gmtest[.in]. Example of error message in the log:

grdblend [NOTICE]: Remote data courtesy of GMT data server static [http://static.generic-mapping-tools.org]
grdblend [NOTICE]: SRTM15 Earth Relief at 05x05 arc minutes reduced by Gaussian Cartesian filtering (9.3 km fullwidth) [Tozer et al., 2019].
grdblend [NOTICE]:   -> Download 180x180 degree grid tile (earth_relief_05m_p): S90W180
grdconvert [ERROR]: File /Users/pwessel/.gmt/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.jp2 was not found

So not sure how it selects .gmt/server here since it clearly knows we are using static (the message). There has to be either a remaining bug in the code or something with the exports. If I look at the log and see that ex27 failed for the same reason, I can do chest -R ex27 and it passes...

PaulWessel · 2023-10-08T14:14:22Z

Running Cmake prints

* Found GMT data server : static

but when I run all the tests it only creates .gmt/server and .gmt/cache. No static dir is created.

I added some print statements to gmtremote_init_paths and nothing is written to the log. How is this possible?

Building again and this time I get a log with lots of

gmt [NOTICE]: Enter gmtremote_init_paths (0)
gmt [NOTICE]: GMT_DATA_SERVER is static

Yet things are saved in server and cache. Not sure what to think.

PaulWessel · 2023-10-08T14:59:43Z

Oh well, found my bug. Now I get consistently files in static.{server, cache} when either setting static in the environment or setting in in ConfigUserAdvanced.make. Please see if the latest update works for you, @seisman.

PaulWessel · 2023-10-08T15:05:20Z

I now get 30 failures only - all but two involve grid image differences.

seisman

Blindly approve because my Homebrew is broken so can't build GMT now.

PaulWessel · 2023-10-08T16:09:08Z

Conclusions so far

If I add set (GMT_DATA_SERVER "static") in ConfigUserAdvanced.cmake and build then the tests correctly see GMT_SERVER = static and the downloads land in .gmt/static.
If I leave that commented out but run export GMT_DATA_SERVER=static then files try to go to ./gmt/server and ./gmt/cache instead.

It seems to be that this new line in gmtest.in

[[ -z "${GMT_DATA_SERVER}" ]] && export GMT_DATA_SERVER=@GMT_DATA_SERVER@

which becomes (when not set explicitly)

[[ -z "${GMT_DATA_SERVER}" ]] && export GMT_DATA_SERVER=oceania

So that sets the default server probably because it is not set and thus defaults to oceania.

Maybe the simplest is for the CI to simply have set (GMT_DATA_SERVER "static") in its Advanced setting and not rely on any environment variables?

seisman · 2023-10-09T02:43:28Z

2. If I leave that commented out but run export GMT_DATA_SERVER=static then files try to go to ./gmt/server and ./gmt/cache instead.

It seems to be that this new line in gmtest.in

[[ -z "${GMT_DATA_SERVER}" ]] && export GMT_DATA_SERVER=@GMT_DATA_SERVER@

which becomes (when not set explicitly)

[[ -z "${GMT_DATA_SERVER}" ]] && export GMT_DATA_SERVER=oceania

So that sets the default server probably because it is not set and thus defaults to oceania.

[[ -z "${GMT_DATA_SERVER}" ]] && export GMT_DATA_SERVER=oceania. This command checks if GMT_DATA_SERVER is an empty string. If yes, then it runs export GMT_DATA_SERVER=oceania, otherwise it does nothing.

Following what you did, I also added gmt set GMT_VERBOSE d" to line 296 in gmtest.in, and ran a single test ctest -R doc/scripts/GMT_tut_11.sh. Here is the log file:
LastTest.log.

Some important logs are pasted here:

gmt [NOTICE]: GMT_DATA_SERVER is static
gmt [DEBUG]: Download remote file http://static.generic-mapping-tools.org/gmt_data_server.txt for the first time
gmt [INFORMATION]: Downloading file http://static.generic-mapping-tools.org/gmt_data_server.txt ...
gmt [DEBUG]: Delete /tmp/gmt_data_server.txt.download
gmt [DEBUG]: Load contents from /home/seisman/.gmt/gmt_data_server.txt
gmt [NOTICE]: GMT_DATA_SERVER is static
gmt [DEBUG]: Download remote file http://static.generic-mapping-tools.org/gmt_hash_server.txt for the first time
gmt [INFORMATION]: Downloading file http://static.generic-mapping-tools.org/gmt_hash_server.txt ...
gmt [DEBUG]: Delete /tmp/gmt_hash_server.txt.download

...
grdblend [DEBUG]: Look for file S90W180.earth_relief_05m_p.nc in /home/seisman/.gmt
grdblend [DEBUG]: Look for file S90W180.earth_relief_05m_p.nc in /home/seisman/OSS/gmt/gmt/doc/scripts
grdblend [DEBUG]: Look for file S90W180.earth_relief_05m_p.nc in /home/seisman/.gmt/cache
grdblend [DEBUG]: Look for file S90W180.earth_relief_05m_p.nc in /home/seisman/.gmt/server
grdblend [INFORMATION]: Downloading earth_relief_05m_p/ tile 1 of 1 [S90W180]
grdblend [DEBUG]: Get remote file http://static.generic-mapping-tools.org/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.jp2 and write to /home/seisman/.gmt/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.jp2
grdblend [DEBUG]: Download http://static.generic-mapping-tools.org/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.jp2 to /home/seisman/.gmt/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.jp2
grdblend [NOTICE]: Remote data courtesy of GMT data server static [http://static.generic-mapping-tools.org]

grdblend [NOTICE]: SRTM15 Earth Relief at 05x05 arc minutes reduced by Gaussian Cartesian filtering (9.3 km fullwidth) [Tozer et al., 2019].
grdblend [NOTICE]:   -> Download 180x180 degree grid tile (earth_relief_05m_p): S90W180
grdblend [INFORMATION]: Downloading file http://static.generic-mapping-tools.org/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.jp2 ...
grdblend [DEBUG]: Delete /tmp/S90W180.earth_relief_05m_p.nc.download
grdblend [INFORMATION]: Convert SRTM tile from JPEG2000 to netCDF grid [/home/seisman/.gmt/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.nc]
grdblend [DEBUG]: Running: grdconvert /home/seisman/.gmt/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.jp2 -G/home/seisman/.gmt/server/earth/earth_relief/earth_relief_05m_p/S90W180.earth_relief_05m_p.nc=ns+s0.5+o0 -Z+s0.5+o-0 -fg -Vq --IO_NC4_DEFLATION_LEVEL=9 --GMT_HISTORY=readonly

As you can see, GMT now knows it should use the static server, but somehow it still tries to find files in ~/.gmt/cache and ~/.gmt/server.

So, it's likely there are still bugs, but I'm OK to approve this PR if you think the bugs are not worth your coding time.

PaulWessel · 2023-10-09T12:22:58Z

I follow your example to run just tut_11 and I get no errors and all is static and files downloaded to static. This is with no GMT_DATA_SERVER set in my environment. How can you and I get different results? I also broke the [[ -z test in to and if - fi so it prints out

gmtest: export GMT_DATA_SERVER=static

So all seems good here?

seisman · 2023-10-09T12:46:59Z

This is with no GMT_DATA_SERVER set in my environment. How can you and I get different results?

With set (GMT_DATASERVER static), I'm getting the correct result.

If I comment out set (GMT_DATA_SERVER static) (thus default to oceania), run export GMT_DATA_SERVER=static in the terminal, then I get the error.

PaulWessel · 2023-10-09T12:59:44Z

Strange. Here is my terminal (I am in the build directory). I comment out stuff in gmtest

#if [[ -z "${GMT_DATA_SERVER}" ]]; then
#  export GMT_DATA_SERVER=static
#  echo "gmtest: export GMT_DATA_SERVER=static"
#fi

and static is set in the terminal environment via export.

(base) pwessel@MacAttack-2: echo $GMT_DATA_SERVER
static
(base) pwessel@MacAttack-2:ctest -R tut_11

Test project /Users/pwessel/UH/RESEARCH/CVSPROJECTS/GMTdev/gmt-dev/rbuild
    Start 168: doc/scripts/GMT_tut_11.sh
1/1 Test #168: doc/scripts/GMT_tut_11.sh ........***Failed    0.63 sec

The test fails because the gm compare finds an rms exceeding the threshold, but it runs fine and grabs data from static.

seisman · 2023-10-11T09:32:23Z

I feel it's just a minor issue and doesn't matter much. So maybe merge this PR and revisit the issue if it causes troubles.

PaulWessel · 2023-10-11T10:18:42Z

OK, merged.

Esteban82 · 2023-12-17T17:00:14Z

I got some errors with animation 2 because GMT searchs the remote data grids in the static sever. I think it is related to this PR. Later I can post more details.

joa-quim · 2023-12-17T17:07:40Z

Don't know if related but the new planetary grids are still only in candidate. Trying to access them leads to download errors.

PaulWessel · 2023-12-17T17:13:14Z

All the new stuff is still only on candidate. We release it to oceania and syncing hosts when 6.5 is released (same day/time). I did several of these bash aliases, e.g.,

alias set-static='export GMT_DATA_SERVER=static'
alias set-candidate='export GMT_DATA_SERVER=candidate'
etc.

and since I jump between servers for tests and try-outs I have to remember to do

set-static

before running tests and

set-candidate

before checking on Mars and mss etc. Works fine for me (?)

joa-quim · 2023-12-17T17:25:38Z

But since the new datasets are already visible in https://www.generic-mapping-tools.org/remote-datasets/ why waiting to official 6.5?

In my mix cmd-with-mingw those exports (set on Win) are not working and I've not yet found time/patience to find out why/fix-it.

Esteban82 · 2023-12-17T17:30:50Z

I usually use the candidate server. I suspect that GMT movie changes alone to the static server.

PaulWessel · 2023-12-17T18:26:23Z

Hm, yes those scripts started by system calls probably goes to the default server. So Oceania

PaulWessel added 2 commits October 6, 2023 12:10

WIP Fix issues when running tests on static

88dacf5

Still stuff not working. Looks like setting expert GMT_DATA_SERVER=static works differently (better?) than having gmtest.in do it.

Updates with simpler path management

08749f4

PaulWessel requested a review from seisman October 7, 2023 07:21

PaulWessel self-assigned this Oct 7, 2023

PaulWessel added the enhancement Improving an existing feature label Oct 7, 2023

PaulWessel added this to the 6.5.0 milestone Oct 7, 2023

PaulWessel added 3 commits October 8, 2023 13:50

Merge branch 'master' into static-server-issues

95876b9

Update gmtest.in

4341b4b

Merge branch 'master' into static-server-issues

7b6fb32

Merge branch 'master' into static-server-issues

1f40bec

Update gmt_remote.c

b46409a

seisman approved these changes Oct 8, 2023

View reviewed changes

Merge branch 'master' into static-server-issues

dceeb0a

Merge branch 'master' into static-server-issues

90ed583

PaulWessel changed the title ~~WIP Static server issues~~ Static server issues Oct 11, 2023

PaulWessel merged commit 3909151 into master Oct 11, 2023
6 checks passed

PaulWessel deleted the static-server-issues branch October 11, 2023 10:18

maxrjones added the add-changelog Add PR to the changelog label Dec 8, 2023

Esteban82 mentioned this pull request Dec 17, 2023

Movie automatically changes the dataserver to Oceania (or static)? #8209

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Static server issues #7879

Static server issues #7879

PaulWessel commented Oct 7, 2023

PaulWessel commented Oct 7, 2023

PaulWessel commented Oct 8, 2023

PaulWessel commented Oct 8, 2023

PaulWessel commented Oct 8, 2023

PaulWessel commented Oct 8, 2023

seisman left a comment

PaulWessel commented Oct 8, 2023

seisman commented Oct 9, 2023 •

edited

PaulWessel commented Oct 9, 2023

seisman commented Oct 9, 2023

PaulWessel commented Oct 9, 2023

seisman commented Oct 11, 2023

PaulWessel commented Oct 11, 2023

Esteban82 commented Dec 17, 2023

joa-quim commented Dec 17, 2023

PaulWessel commented Dec 17, 2023

joa-quim commented Dec 17, 2023

Esteban82 commented Dec 17, 2023

PaulWessel commented Dec 17, 2023

Static server issues #7879

Static server issues #7879

Conversation

PaulWessel commented Oct 7, 2023

PaulWessel commented Oct 7, 2023

PaulWessel commented Oct 8, 2023

PaulWessel commented Oct 8, 2023

PaulWessel commented Oct 8, 2023

PaulWessel commented Oct 8, 2023

seisman left a comment

Choose a reason for hiding this comment

PaulWessel commented Oct 8, 2023

seisman commented Oct 9, 2023 • edited

PaulWessel commented Oct 9, 2023

seisman commented Oct 9, 2023

PaulWessel commented Oct 9, 2023

seisman commented Oct 11, 2023

PaulWessel commented Oct 11, 2023

Esteban82 commented Dec 17, 2023

joa-quim commented Dec 17, 2023

PaulWessel commented Dec 17, 2023

joa-quim commented Dec 17, 2023

Esteban82 commented Dec 17, 2023

PaulWessel commented Dec 17, 2023

seisman commented Oct 9, 2023 •

edited