Test failures with RDNAPTRANS2018 (Dutch National Grid)

[sebastic]

The Dutch national grids have been updated (but not in EPSG yet) and are available via:

https://www.nsgi.nl/geodetische-infrastructuur/producten/programma-rdnaptrans

Or directly from the proj-rdnap Debian package git repository:

https://salsa.debian.org/debian-gis-team/proj-rdnap

The license changed to CC-BY-4.0 so they can be included in proj-datumgrid.

Unfortunately some of the tests for the 2018 grids (Variant 2 - From ETRS89 to RD/NAP) fail with PROJ 6.2.0.

The following tarball contains a subset of the proj-rdnap Debian package to reproduce this issue:

https://linuxminded.nl/tmp/proj-rdnap-2018.tar.gz (6.6M)

Description of the content:

• RDNAPTRANS2018.pdf
  Upstream documentation about the grids, including cct commands to use the grids.
• testrdtrans2018.pl
  Script to execute all tests for both variants from ETRS89 to RD/NAP and vice versa.
• Z001_ETRS89andRDNAP.txt
  The self-validation dataset (ETRS89 & RD/NAP coordinates to test)
• Variant1/
  Directory with variant 1 of the grids.
• Variant2/
  DIrectory with variant 2 of the grids

The output of the test script for my run can be found at:

https://linuxminded.nl/tmp/testrdtrans2018.log.gz

The subset of coordinates from Z001_ETRS89andRDNAP.txt that fail can be found at:

https://linuxminded.nl/tmp/testrdtrans2018_FAILED.txt

According to Jochem Lesparre of Kadaster these failues are a known issue with PROJ due to points on the edge of the grid not being handled correctly, but he hadn't reported this issue yet. I'm taking the liberty to do so now.

[rouault]

The license changed to CC-BY-4.0 so they can be included in proj-datumgrid.

Perhaps worth filing a dedicated ticket to https://github.com/OSGeo/proj-datumgrid/issues

[sebastic]

I'll prepare a PR for proj-datumgrid once EPSG has been updated to use the new grids.

[Jochem-L]

Some additional information: There are 2 variants of the transformation. Variant 1 uses a NTv2 grid AND a 7 parameter transformation. Variant 2 uses only a NTv2 grid. The NTv2 grid has a course parent grid and a dense subgrid. In variant 1, the course grid and the edge of the dense grid only contain zero values, as the datum tranformation is done with a 7 parameter transformation and grid corrections are only needed upto 50 km outside the Netherlands. In variant 2, the course grid gives the same result as the 7 parameter transformation. The edge of the dense grid should also give the same results as the 7 parameter transformation. However, errors of about 1.5 cm occur on and near the edge of the dense grid.

Edit: The errors also occur on and near the edge of the course grid. So, it seems that it is a more general problem in the grid interpolation operation and it has nothing to do with the complications of having a subgrid nor of having two variants. The more complicated variant 1 next to the more straightforward variant 2 only reveil the error in the interpolation operation.

[rouault]

@Jochem-L Looking quickly at the PROJ pipelines in RDNAPTRANS2018.pdf, I've the feeling that some of the plays with egm96 and z_out=1e32 (at least this last one) to apparently compensate for the change of height due to the Helmert step could be more elegantly replaced by using the +step +proj=push +v_3 and +step +proj=pop +v_3 steps that respectively save and restore the Z value. See https://proj.org/operations/conversions/push.html

[Jochem-L]

@rouault Push and pop can be used for this problem, but only if one uses the real height in the Helmert transformation. Since we use a fixed height (0 m from RD Bessel to ETRS89 and 43 m from ETRS89 to RD Bessel) in the Helmert transformation (to get the same results as when using a horizontal grid shift file), this doesn't work. I think it is possible to do it slightly more elegant, by using +step proj=geogoffset next to push and pop. I only noticed the possibility of geogoffset after we published the PROJ. pipeline command. If this helps, I will update RDNAPTRANS2018.pdf with the more elegant command. However, it will still misuse some functionality. For a real solution, additional functionality would be needed: something like +step proj=set +v_3=43 or +step +proj=pop +v_3=43.

[rouault]

@Jochem-L I've investigated and fixed the issue in rouault@f36d237. This fix is queued in my rfc4_gtiff branch for convenience. It could theoretically be backported to 6.3 but this would require reworking internal interfaces... which I've done per RFC4, so I think this will be 7.0 only material

[rouault]

Actually backporting to 6.3 was not so bad: #1797

[Jochem-L]

Thanks! That will hopefully make my work much easier after 6.3...

[sebastic]

While this change was triggered by the 2018 Dutch grids, it seems to have affected the 2008 grids as well.

cs2cs did not report coordinated before:

PROJ version: 6.2.1
[...]

Test:   08 No_geoid
Exec:   cs2cs EPSG:7415 +to EPSG:4937 -f %.9f
Input:  100000.6700 350000.8900
Output: *	* inf
Expect: 51.136825197	4.601375361
Test OK: From RD/NAP to ETRS89 - 08 No_geoid (Expected output overriden: *	* ^-?(\d+\.\d+|inf)$)

Test:   09 No_rd
Exec:   cs2cs EPSG:7415 +to EPSG:4937 -f %.9f
Input:  79000.0100 500000.2300
Output: *	* inf
Expect: 52.482440839	4.268403889
Test OK: From RD/NAP to ETRS89 - 09 No_rd (Expected output overriden: *	* ^-?(\d+\.\d+|inf)$)

https://ci.debian.net/data/autopkgtest/testing/amd64/p/proj-rdnap/3743126/log.gz

It does now:

PROJ version: 6.3.0
[...]

Test:   08 No_geoid
Exec:   cs2cs EPSG:7415 +to EPSG:4937 -f %.9f
Input:  100000.6700 350000.8900
Output: 51.136825267	4.601375791 0.000000000
Expect: 51.136825197	4.601375361
Latitude exceeds 1e-08 degrees: 6.99999986863986e-08
Longitude exceeds 1e-08 degrees: 4.30000000051223e-07
Test FAILED: From RD/NAP to ETRS89 - 08 No_geoid (Expected output overriden: *	* ^-?(\d+\.\d+|inf)$)

Test:   09 No_rd
Exec:   cs2cs EPSG:7415 +to EPSG:4937 -f %.9f
Input:  79000.0100 500000.2300
Output: 52.482440838	4.268403950 0.000000000
Expect: 52.482440839	4.268403889
Longitude exceeds 1e-08 degrees: 6.09999997180921e-08
Test FAILED: From RD/NAP to ETRS89 - 09 No_rd (Expected output overriden: *	* ^-?(\d+\.\d+|inf)$)

https://ci.debian.net/data/autopkgtest/testing/amd64/p/proj-rdnap/3844406/log.gz

So while better, it still exceeds the allowed margin.

I'll update the test configuration to allow it for these edge cases, I'm mostly noting it here for the record.