Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NTV2 datum shifts fail for points between 10^-5 and 10^-11 pixels outside subgrid #209

proj4-bot opened this issue May 22, 2015 · 0 comments


Copy link

proj4-bot commented May 22, 2015

Reported by bpayne on 31 Jan 2013 18:15 UTC
There is a narrow range outside each subgrid in NTV2 in which a point to be transformed is determined to be inside the subgrid, but attempting to calculate the offset within that subgrid fails. This prevents the point from being correctly interpolated within the top level grid and results in a grid shift calculation failure.

To reproduce, attempt to calculate a datum shift (NAD27-Canada-NTV2) for point: 511075.11572697730 5899877.2372716544 (UTM Zone 12 N, METERS). This will result in a failure to calculate the datum shift.

It's a little complicated to explain, so I have included a potential fix here. Essentially, in pj_apply_gridshift_3, a tolerance (1/10,000th of a subgrid pixel)is used to determine that points close to a subgrid (in the example case ALvermil) are actually within the subgrid. However, when the interpolation point is calculated, in nad_intr, only pixels within a different tolerance (1/100000000000 of a subgrid pixel) are assigned a point within the subgrid. This means that an interpolation failure results, when in fact, the higher level grid should be used to calculate the shift. The proposed solution is to adjust the tolerances so that both are approximately 10^-8 (about halfway between the existing tolerance), meaning that the nearest pixels will use the subgrid, but all others will use the main grid.

Example code modifications:

The first half of the fix (accept fewer points outside the subgrid) is in pj_apply_gridshift, approximately line 164,
double epsilon =
should be changed to
double epsilon =
Note: there is one other example of the epsilon calculation around line 144. I think this should be left alone as the top level grids do require slightly different handling here (although I would be interested to hear if there are any counter arguments here).

The second half of the fix (allow more points outside the subgrid to be interpolated successfully) is in nad_intr.c, lines 19 and 32.

if (indx.lam == -1 && frct.lam > 0.99999999999) {
if (indx.phi == -1 && frct.phi > 0.99999999999) {
if (indx.lam == -1 && frct.lam > 0.9999999) {
if (indx.phi == -1 && frct.phi > 0.9999999) {

Upon request, I can submit the entire files with updates.


rouault added a commit to rouault/PROJ that referenced this issue Jan 11, 2020
rouault added a commit to rouault/PROJ that referenced this issue Jan 13, 2020
@kbevers kbevers added this to the 7.0.0 milestone Jan 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

2 participants