new NetCDF_jll v400.702.402+0 broken on Windows

[Alexander-Barth]

Unfortunately, the new NetCDF_jll v400.702.402+0 does not work on Windows (as far as I know Linux x86_64 and apple M1 are fine).

The new version of NetCDF_jll was created in this commit:
#4481

The errors seem to be related to the upgrade of HDF5_jll v1.12.1+0.

Related bug reports:
JuliaGeo/NCDatasets.jl#164
meggart/NetCDF.jl#151

This is the full error message when a Windows user (on julia 1.7.2) creates a NetCDF (with HDF5 backend) as reported @visr is below.

Is there a way to test via CI natively the library before releasing them?

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x2374cb3 -- .text at C:\Users\visser_mn\.julia\artifacts\2b6e2ce84250e36811c3019c1ad253c1739c888f\bin\libnetcdf-18.dll (unknown line)
in expression starting at REPL[13]:1
.text at C:\Users\visser_mn\.julia\artifacts\2b6e2ce84250e36811c3019c1ad253c1739c888f\bin\libnetcdf-18.dll (unknown line)
NC4_create at C:\Users\visser_mn\.julia\artifacts\2b6e2ce84250e36811c3019c1ad253c1739c888f\bin\libnetcdf-18.dll (unknown line)
NC_create at C:\Users\visser_mn\.julia\artifacts\2b6e2ce84250e36811c3019c1ad253c1739c888f\bin\libnetcdf-18.dll (unknown line)
nc__create at C:\Users\visser_mn\.julia\artifacts\2b6e2ce84250e36811c3019c1ad253c1739c888f\bin\libnetcdf-18.dll (unknown line)
nc_create at C:\Users\visser_mn\.julia\artifacts\2b6e2ce84250e36811c3019c1ad253c1739c888f\bin\libnetcdf-18.dll (unknown line)
top-level scope at .\REPL[13]:1
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:876
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:830
jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:830
jl_toplevel_eval at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:894 [inlined]
jl_toplevel_eval_in at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:944
eval at .\boot.jl:373 [inlined]
eval_user_input at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:150
repl_backend_loop at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:246
start_repl_backend at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:231
#run_repl#47 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:364
run_repl at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:351
#930 at .\client.jl:394
jfptr_YY.930_36349.clone_1 at C:\Users\visser_mn\.julia\juliaup\julia-1.7.2+0~x64\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757
#invokelatest#2 at .\essentials.jl:716 [inlined]
invokelatest at .\essentials.jl:714 [inlined]
run_main_repl at .\client.jl:379
exec_options at .\client.jl:309
_start at .\client.jl:495
jfptr__start_21275.clone_1 at C:\Users\visser_mn\.julia\juliaup\julia-1.7.2+0~x64\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined]
true_main at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:559
jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:701
mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)
Allocations: 9681000 (Pool: 9675485; Big: 5515); GC: 13

[giordano]

Smells of upstream bug to me (and given the track record of troubles with this library I'm not even surprised)

[Alexander-Barth]

I retried to make a NetCDF 4.8.1 binary (with HDF5 1.12.1) and we got the same EXCEPTION_ACCESS_VIOLATION on Windows . However if we downgrade HDF5 to 1.12.0 (without Apple M1 support) NetCDF 4.8.1 and 4.7.4 works on Windows. For Linux, all tested combinations seem to work. Is it possible to release different versions for different platforms? Can we yank only the Windows version of NetCDF_jll v400.702.402+0?

[giordano]

Can we yank only the Windows version of NetCDF_jll v400.702.402+0?

There is no concept of platform-specificity in the registry nor the package manager, so that's unrealistic

[Alexander-Barth]

Should this release be yanked then from all platforms? Unfortunately, this will remove the Apple M1 binary, but Windows is pretty common ... (but the real solution we an updated working binary for all platforms).

[giordano]

That may be necessary, yes. But it'd also be great to understand why the Windows build is broken again. We haven't touched the mingw toolchain for quite some time, the hdf5 source is always the same (just a newer version maybe?), version of netcdf source is also the same as the last working version, no?

However, please make sure no dependents of netcdf_jll require the version you want to yank, otherwise you'll make those packages completely broken

[Alexander-Barth]

But it'd also be great to understand why the Windows build is broken again.

It might be related to the HDF5_jll update. HDF5.jl runs fine but maybe an issue with the header files in HDF5_jll?
The error does not occur in NetCDF when you do not use the HDF5 backend format.

version of netcdf source is also the same as the last working version, no?

yes, that is exactly the same version of NetCDF sources.

If we yank NetCDF_jll 400.702.402+0 we will fall-back to 400.702.400+0, I guess we will have a problem with
the latest version of TempestRemap_jll:

59d2816

I did not see any other package incompatible with

 grep -r  NetCDF_jll .
./T/TempestModel_jll/Compat.toml:NetCDF_jll = "400.701.400-400.799"
./T/TempestModel_jll/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./T/TempestRemap_jll/Compat.toml:NetCDF_jll = "400.701.400-400.799"
./T/TempestRemap_jll/Compat.toml:NetCDF_jll = "400.702.402-400.799"
./T/TempestRemap_jll/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./I/IOAPI_jll/Compat.toml:NetCDF_jll = "400.701.400-400.799"
./I/IOAPI_jll/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./N/NetcdfIO/Compat.toml:NetCDF_jll = "400.701.400-400.702.400"
./N/NetcdfIO/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./N/NetCDF/Compat.toml:NetCDF_jll = "4.7.4-4"
./N/NetCDF/Compat.toml:NetCDF_jll = "400.701.400-400"
./N/NetCDF/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./N/NCDatasets/Compat.toml:NetCDF_jll = "4.7.4-4"
./N/NCDatasets/Compat.toml:NetCDF_jll = "400.701.400-400"
./N/NCDatasets/Compat.toml:NetCDF_jll = ["400.701.400", "400.702.400"]
./N/NCDatasets/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./N/NetCDFF_jll/Compat.toml:NetCDF_jll = "400.701.400-400.799"
./N/NetCDFF_jll/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./N/NetCDF_jll/Package.toml:name = "NetCDF_jll"
./N/NetCDF_jll/Package.toml:repo = "https://github.com/JuliaBinaryWrappers/NetCDF_jll.jl.git"
./M/MDAL_jll/Compat.toml:NetCDF_jll = "400.701.400-400.799"
./M/MDAL_jll/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./V/VMEC_jll/Compat.toml:NetCDF_jll = "400.702.400-400.799"
./V/VMEC_jll/Deps.toml:NetCDF_jll = "7243133f-43d8-5620-bbf4-c2c921802cf3"
./Registry.toml:7243133f-43d8-5620-bbf4-c2c921802cf3 = { name = "NetCDF_jll", path = "N/NetCDF_jll" }

[giordano]

It might be related to the HDF5_jll update.

The funny thing is that Windows is the only platform for which we always used the same source: the msys2 libraries.

[Alexander-Barth]

Yes, indeed. I was wondering if there is reason to use msys2 over conda-forge on Windows for HDF5...

[giordano]

Msys2 is probably more consistent with the toolchain we use here (mingw) and we still need to provide the runtime dependencies. You'd need to investigate whether conda-forge build for Windows is compatible with our other libraries.

[Alexander-Barth]

You are right, conda-forge uses the Windows Visual C/C++ compiler (https://conda-forge.org/docs/maintainer/knowledge_base.html)

Given that TempestRemap_jll depends on NetCDF_jll 400.702.402+0, does this means that we out of luck with the yanking this NetCDF_jll version?

Unfortunately, the diffs in the headers file quite massive (for a patch release!):

diff  HDF5.v1.12.0.x86_64-w64-mingw32/include/  HDF5.v1.12.1.x86_64-w64-mingw32/include/ | wc -l
# 44843