GMT tests fail on Windows: Failure to open virtual file

[niclasmattsson]

Continued from Discourse:
https://discourse.julialang.org/t/how-can-i-rasterize-the-polygons-of-a-shapefile-in-julia/15089/14

I have GMT6dev installed (x64 version) and have added GMT6\bin to my path. Running ]test GMT or the Hello World example in the GMT docs gets me this error:

6.0.0_r20419
ERROR: LoadError: GMT: Failure to open virtual file
Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] GMTJL_Set_Object(::Ptr{Nothing}, ::GMT.GMT_RESOURCE, ::Array{Float32,2}) at C:\Users\niclas\.julia\packages\GMT\SwD1H\src\gmt_main.jl:979
 [3] gmt(::String, ::Array{Float32,2}) at C:\Users\niclas\.julia\packages\GMT\SwD1H\src\gmt_main.jl:262
 [4] top-level scope at C:\Users\niclas\.julia\packages\GMT\SwD1H\test\runtests.jl:16
 [5] include at .\boot.jl:317 [inlined]
 [6] include_relative(::Module, ::String) at .\loading.jl:1038
 [7] include(::Module, ::String) at .\sysimg.jl:29
 [8] include(::String) at .\client.jl:388
 [9] top-level scope at none:0
in expression starting at C:\Users\niclas\.julia\packages\GMT\SwD1H\test\runtests.jl:13
ERROR: Package GMT errored during testing

[joa-quim]

That error message ERROR: LoadError: GMT: Failure to open virtual file is typical of when there is confusion from GMT5 and GMT6. Please double check that you have not GMT5 left behind.

The rest of the story is that I am the person who creates the GMT Windows installers (I'm a GMT developer too) and when I do the all my GMT.jl testings, examples, etc I am actually using the current GMT development version, which is not exactly equal to the one you installed (time has passed meanwhile). So I can't be 100% sure that that version works the I think (it should but ... you know). If that turns out to be the case, and if you want to be so kind to test it, I can prepare another installer that reflects current state of GMT development, and the one that I'm really using now.

[niclasmattsson]

No, GMT5 is completely uninstalled. There is nothing left where I installed it or in the PATH, and I restarted my computer. I now tried uninstalling and reinstalling GMT6dev. after the restart. This time I tried running the installer as Administrator. I can run psxy - from any folder and get the usage options. But I still get the same error when running the tests. Are you sure I only needed to add GMT6/bin to my PATH? My guess would be that other folders need to be there as well.

If you send me a link I'll try the updated installer for the current master.

Since you're the developer of the installer I'll also mention that the PATH too long error I get is very strange. There are only 12 folders in my PATH and I know Windows can handle a lot more than that. None of the 12 folder paths are very long, and I got the error even when I installed directly to C:.

[joa-quim]

Yes, I'm sure that only the bin needs to be in the path. The installer is created by nsis (comes with cmake). What I do is to build GMT on Windows as well as GDAL and a lot of its dependencies. But I have no real answer to the "path too long" thing.

I'll prepare a new installer with an updated GMT version and will try to test it here on the condition that GMT was installed under "Program Files", which is something that we advise against. That space in dir name tends to cause never ending problems.

[joa-quim]

OK, made a new installer and tested it. It works fine for me.

BUT, this time I tested the installer by installing it in the original computer where it was created and -- damn it -- got the same warning about path being too long. However my path is a bit long, larger than 2k characters. I think the limit is bytes, not in number on entries. Googling for that warning message reveals a lot of answers coming from cmake, so likely this is still a cmake issue and the solutions continues to be to add the gmt6\bin dir to the path manually.

[niclasmattsson]

With the new installer:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.0 (2018-08-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(v1.0) pkg> test GMT
   Testing GMT
    Status `C:\Users\niclas\AppData\Local\Temp\jl_75DE.tmp\Manifest.toml`
  [5752ebe1] GMT v0.5.0
  [2a0f44e3] Base64  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Base64`]
  [ade2ca70] Dates  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Dates`]
  [8bb1440f] DelimitedFiles  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\DelimitedFiles`]
  [8ba89e20] Distributed  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Distributed`]
  [b77e0a4c] InteractiveUtils  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\InteractiveUtils`]
  [76f85450] LibGit2  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\LibGit2`]
  [8f399da3] Libdl  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Libdl`]
  [37e2e46d] LinearAlgebra  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\LinearAlgebra`]
  [56ddb016] Logging  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Logging`]
  [d6f4376e] Markdown  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Markdown`]
  [a63ad114] Mmap  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Mmap`]
  [44cfe95a] Pkg  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Pkg`]
  [de0858da] Printf  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Printf`]
  [3fa0cd96] REPL  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\REPL`]
  [9a3f8284] Random  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Random`]
  [ea8e919c] SHA  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\SHA`]
  [9e88b42a] Serialization  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Serialization`]
  [6462fe0b] Sockets  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Sockets`]
  [8dfed614] Test  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Test`]
  [cf7118a7] UUIDs  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\UUIDs`]
  [4ec0a83e] Unicode  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Unicode`]
6.0.0_3196956
PSL: Warning: Selected font (15) out of range (0--1); reset to 0

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffcd70fa32c -- PSL_settransparencymode at C:\Program Files\gmt6\bin\postscriptlight_w64.dll (unknown line)
in expression starting at C:\Users\niclas\.julia\packages\GMT\q27Sj\test\runtests.jl:13
PSL_settransparencymode at C:\Program Files\gmt6\bin\postscriptlight_w64.dll (unknown line)
PSL_setfont at C:\Program Files\gmt6\bin\postscriptlight_w64.dll (unknown line)
gmt_setfont at C:\Program Files\gmt6\bin\gmt_w64.DLL (unknown line)
GMT_gmtlogo at C:\Program Files\gmt6\bin\gmt_w64.DLL (unknown line)
GMT_Call_Module at C:\Program Files\gmt6\bin\gmt_w64.DLL (unknown line)
GMT_Call_Module at C:\Users\niclas\.julia\packages\GMT\q27Sj\src\libgmt.jl:177
unknown function (ip: 00000000291B55FF)
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
gmt at C:\Users\niclas\.julia\packages\GMT\q27Sj\src\gmt_main.jl:266
finish_PS_module at C:\Users\niclas\.julia\packages\GMT\q27Sj\src\common_options.jl:936
finish_PS_module at C:\Users\niclas\.julia\packages\GMT\q27Sj\src\common_options.jl:964 [inlined]
#logo#35 at C:\Users\niclas\.julia\packages\GMT\q27Sj\src\gmtlogo.jl:79
unknown function (ip: 00000000291ECAA9)
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182 [inlined]
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1536 [inlined]
jl_invoke at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:56
#logo at .\none:0
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
do_call at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:324
eval_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:428
eval_stmt_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:363 [inlined]
eval_body at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:682
jl_interpret_toplevel_thunk_callback at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:799
unknown function (ip: FFFFFFFFFFFFFFFE)
unknown function (ip: 000000001001046F)
unknown function (ip: 0000000000000054)
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:787
jl_parse_eval_all at /home/Administrator/buildbot/worker/package_win64/build/src\ast.c:838
jl_load at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:821 [inlined]
jl_load_ at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:828
include at .\boot.jl:317 [inlined]
include_relative at .\loading.jl:1038
include at .\sysimg.jl:29
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
include at .\client.jl:388
jl_fptr_trampoline at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:1829
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
do_call at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:324
eval_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:428
eval_stmt_value at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:363 [inlined]
eval_body at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:682
jl_interpret_toplevel_thunk_callback at /home/Administrator/buildbot/worker/package_win64/build/src\interpreter.c:799
unknown function (ip: FFFFFFFFFFFFFFFE)
unknown function (ip: 000000000E33BACF)
unknown function (ip: FFFFFFFFFFFFFFFF)
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:787
jl_toplevel_eval_flex at /home/Administrator/buildbot/worker/package_win64/build/src\toplevel.c:734
jl_toplevel_eval_in at /home/Administrator/buildbot/worker/package_win64/build/src\builtins.c:622
eval at .\boot.jl:319
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
macro expansion at .\logging.jl:317 [inlined]
exec_options at .\client.jl:219
_start at .\client.jl:421
jl_apply_generic at /home/Administrator/buildbot/worker/package_win64/build/src\gf.c:2182
jl_apply at /home/Administrator/buildbot/worker/package_win64/build/src\julia.h:1536 [inlined]
true_main at /home/Administrator/buildbot/worker/package_win64/build/ui\repl.c:112
wmain at /home/Administrator/buildbot/worker/package_win64/build/ui\repl.c:233
__tmainCRTStartup at /usr/src/debug/mingw64-x86_64-runtime-5.0.3-1/crt\crtexe.c:329
mainCRTStartup at /usr/src/debug/mingw64-x86_64-runtime-5.0.3-1/crt\crtexe.c:212
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Allocations: 25823229 (Pool: 25816832; Big: 6397); GC: 56
ERROR: Package GMT errored during testing

[joa-quim]

It's proving harder than imagined but thanks to this we caught a bug in the GMT development. Very recently we had to make changes related to transparency in the conversion from PostScript to PDF because Ghostscript made a breaking change in its last version (9.25) driven by security reasons. Apparently our internal GMT tests were not able to catch this problem, and neither did I yesterday night when I did only quick tests. In fact I did not run test GMT, which when I do now make me reproduce the crash. That is good because we'll be able to trace it down.

But it's not all bad news. The part that you are more interested now, works. I mean, this works for me

coast(DCW="AR+p0.25p+gblue", frame=:a, show=1)

[joa-quim]

Fortunately your error message is much longer than mine

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffcff68aa30 -- unknown function (ip: 00007FFCFF68AA30)
in expression starting at C:\j\.julia\v0.7\GMT\test\runtests.jl:13
unknown function (ip: 00007FFCFF68AA2F)
Allocations: 65445506 (Pool: 65428662; Big: 16844); GC: 141
ERROR: Package GMT errored during testing

likely because you are on Julia v1.0 and I'm on v0.7. But this issue is more illusive that I thought. It only happens with GMT release builds. When I tested it with a debug build than it works.

[joa-quim]

I've been doing lots of test, among them installing a 'cleaner' computer. There I started to also the "Cannot open virtual file" error too. Then I remembered to update the GMT.jl version and started to work. I have no recall to have done breaking changes in master but the fact is that solved the problems, so please one more (and simpler) attempt. Go to GMT.jl master (]add GMT#master) and try again.

[niclasmattsson]

Sorry for disappearing for a few days, life intervened. But no, even with GMT#master I still get the same error as in my last post.

[niclasmattsson]

Now I get a different error when running ]test GMT:

(v1.0) pkg> test GMT
   Testing GMT
    Status `C:\Users\niclas\AppData\Local\Temp\jl_26A3.tmp\Manifest.toml`
  [5752ebe1] GMT v0.5.0+ #master (https://github.com/GenericMappingTools/GMT.jl.git)
  [2a0f44e3] Base64  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Base64`]
  [8bb1440f] DelimitedFiles  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\DelimitedFiles`]
  [8ba89e20] Distributed  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Distributed`]
  [b77e0a4c] InteractiveUtils  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\InteractiveUtils`]
  [8f399da3] Libdl  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Libdl`]
  [37e2e46d] LinearAlgebra  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\LinearAlgebra`]
  [56ddb016] Logging  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Logging`]
  [d6f4376e] Markdown  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Markdown`]
  [a63ad114] Mmap  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Mmap`]
  [de0858da] Printf  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Printf`]
  [9a3f8284] Random  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Random`]
  [9e88b42a] Serialization  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Serialization`]
  [6462fe0b] Sockets  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Sockets`]
  [8dfed614] Test  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Test`]
  [4ec0a83e] Unicode  [`C:\Julia\Julia-1.0.0\bin\..\share\julia\stdlib\v1.0\Unicode`]
6.0.0_3196956
psconvert [ERROR]: Error opening HKLM key
'gswin64c' is not recognized as an internal or external command,
operable program or batch file.
psconvert [ERROR]: Failed to parse response to GhostScript version query [n = -1 0 0].
ERROR: LoadError: Something went wrong when calling the module. GMT error number = 78
Stacktrace:
 [1] error(::String, ::Int32) at .\error.jl:42
 [2] gmt(::String) at C:\Users\niclas\.julia\packages\GMT\XBgvW\src\gmt_main.jl:271
 [3] #psconvert#128(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol},NamedTuple{(:adjust, :fmt),Tuple{Bool,String}}}, ::Function, ::String, ::Array{Any,1}) at C:\Users\niclas\.julia\packages\GMT\XBgvW\src\psconvert.jl:153
 [4] (::getfield(GMT, Symbol("#kw##psconvert")))(::NamedTuple{(:adjust, :fmt),Tuple{Bool,String}}, ::typeof(psconvert), ::String) at .\none:0
 [5] top-level scope at C:\Users\niclas\.julia\packages\GMT\XBgvW\test\runtests.jl:172
 [6] include at .\boot.jl:317 [inlined]
 [7] include_relative(::Module, ::String) at .\loading.jl:1038
 [8] include(::Module, ::String) at .\sysimg.jl:29
 [9] include(::String) at .\client.jl:388
 [10] top-level scope at none:0
in expression starting at C:\Users\niclas\.julia\packages\GMT\XBgvW\test\runtests.jl:13
ERROR: Package GMT errored during testing

Seems to be a problem with the Windows registry? As far as I know, the only thing that changed since last time was pointing GMT5_SHAREDIR to the right folder.

[joa-quim]

Ah, it's working. I just forgot to make it plain clear on the GMT install instructions (and probably try to come out with some test for this) that the ghostscript program is a mandatory dependency too. The thing is GMT produces PostScript only (with one exception for the CLI version) and we need ghostscript to to either visualize the figures or convert them to PDF, jpeg, png, etc. The conversion to other formats is done automatically by fmt=:pdf (likewise for other formats) keyword. For the CLI you also want the companion gsview, which is a better visualizer but uses ghostscript as the engine. GMT.jl does not use gsview.