Update ESRI projection engine db to version 12.9 #2943

nyalldawson · 2021-11-16T05:26:48Z

This fixes #2929 by updating the projection engine db to version 12.9.

The first commit runs the older version of the build_db_from_esri.py script unchanged, so contains only the modifications made in version 12.9 But the 12.9 version also introduces a WKT2 definition of the objects, so I've updated the script to use these instead. The final commit is the resultant changes to the tables as a consequence of parsing WKT2 instead of WKT1.

Some notes:

I'd like to clean up the script, but for now I've only done the minimal changes to get the WKT2 parsing working
I stepped very cautiously here to ensure no breakage -- there was lots of asserts comparing new vs old results which I've stripped out in the final commit.
While there's very minimal actual changes to the definitions as a result of moving to wkt2 (basically some parameter values are specified in milliarcseconds instead of arcseconds), we do gain a whole bunch of extra security by the move (specifically, ensuring that the unit handling for parameters is correct)
There's one change in the db which resulted from using wkt2 which I'm not sure about. The Epoch units have been changed from Years (EPSG:1029) to "Unity" (EPSG:9201). This is coming straight from the Wkt 2 definitions of the operations from the ESRI files. I'm happy to hardcode an exception in here if this will impact proj negatively. I've hardcoded a conversion from Unity units to Years to satisfy the helmert_transformation table constraints
I can confirm that there's no changes resulting from rebuilding the db from scratch, as noted in ESRI Projection Engine 2.9 update available #2929 (comment)

nyalldawson · 2021-11-16T05:57:17Z

@rouault I'm a little unsure how to handle the "insert on projected_crs violates constraint: conversion must not be deprecated when projected_crs is not deprecated" errors. Should I force the CRS to be flagged as deprecated when it uses a deprecated conversion (even though it's not marked as deprecated into the ESRI files), or should I "undeprecate" the conversion?

…n engine db

nyalldawson · 2021-11-16T06:17:25Z

I'm a little unsure how to handle the "insert on projected_crs violates constraint: conversion must not be deprecated when projected_crs is not deprecated" errors. Should I force the CRS to be flagged as deprecated when it uses a deprecated conversion (even though it's not marked as deprecated into the ESRI files), or should I "undeprecate" the conversion?

For now I'm force-deprecating the CRSes. Let me know if this is the wrong approach.

a deprecated conversion for a non-deprecated CRS

nyalldawson · 2021-11-16T06:27:42Z

For now I'm force-deprecating the CRSes. Let me know if this is the wrong approach.

Actually -- ignore all that. I've found the real fix.

rouault

Looks great! Just a few questions

scripts/build_db_from_esri.py

rouault · 2021-11-16T11:59:36Z

data/sql/esri.sql

@@ -1705,6 +1705,24 @@ INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9379','IGb14_(3D)','ESRI');
 INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9469','SRGI2013_(3D)','ESRI');
 INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9546','LTF2004(G)_(3D)','ESRI');
 INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9695','REDGEOMIN_(3D)','ESRI');
+INSERT INTO "geodetic_datum" VALUES('ESRI','1305','ETRF2000_Poland','ETRF2000 Poland','EPSG','7019','EPSG','8901',NULL,NULL,NULL,NULL,0);


do you know why this is promoted as a plain geodetic_datum object, and not just an alias to EPSG:1305 "ETRF2000 Poland" ?
The EPSG corresponding record is pretty similar

INSERT INTO "geodetic_datum" VALUES('EPSG','1305','ETRF2000 Poland',NULL,'EPSG','7019','EPSG','8901','2012-11-15',NULL,NULL,NULL,0);

It's coming from this line: https://github.com/Esri/projection-engine-db-doc/blob/master/csv/pe_list_datum.csv#L164 , which lists the authority as ESRI.

I could add some logic here to test whether an equivalent EPSG record exists and if so, ignore the authority from the csv source file. What do you think?

I could add some logic here to test whether an equivalent EPSG record exists and if so, ignore the authority from the csv source file. What do you think?

yes that could make sense. One could argue too they should credit EPSG here.

I suggest we hold off on this for now and see if we get an answer in Esri/projection-engine-db-doc#34 -- it looks like a bunch of the new records incorrectly list esri as the authority.

@rouault not much action on the upstream ticket so far. Can I suggest we merge this as is, then I'll open atomic prs for adding the enhanced logic? I've come across a number of other things I think are issues and I'm concerned that this PR will become too large if I try to address them all here.

rouault · 2021-11-16T12:01:49Z

data/sql/esri.sql

@@ -6752,6 +6770,37 @@ INSERT INTO alias_name VALUES('projected_crs','EPSG','9680','WGS_84_TM_90_NE','E
 INSERT INTO alias_name VALUES('projected_crs','EPSG','9697','REDGEOMIN_UTM_zone_12S','ESRI');
 INSERT INTO alias_name VALUES('projected_crs','EPSG','9698','REDGEOMIN_UTM_zone_18S','ESRI');
 INSERT INTO alias_name VALUES('projected_crs','EPSG','9699','REDGEOMIN_UTM_zone_19S','ESRI');
+INSERT INTO "projected_crs" VALUES('ESRI','9709','NAD_1983_CSRS_UTM_Zone_23N',NULL,'EPSG','4400','EPSG','4617','EPSG','16023',NULL,0);


do you know why this is promoted as a plain projected_crs object, and not just an alias to EPSG:9709 ?

Again, it's coming from the ESRI source csvs which list ESRI as the authority for this: https://raw.githubusercontent.com/Esri/projection-engine-db-doc/master/csv/pe_list_projcs.csv

9709,9709,"PE_PCS_NAD83_CSRS_UTM_23N","NAD_1983_CSRS_UTM_Zone_23N","PROJCS[""NAD_1983_CSRS_UTM_Zone_23N"",GEOGCS[""GCS_North_American_1983_CSRS"",DATUM[""D_North_American_1983_CSRS"",SPHEROID[""GRS_1980"",6378137.0,298.257222101]],PRIMEM[""Greenwich"",0.0],UNIT[""Degree"",0.0174532925199433]],PROJECTION[""Transverse_Mercator""],PARAMETER[""False_Easting"",500000.0],PARAMETER[""False_Northing"",0.0],PARAMETER[""Central_Meridian"",-45.0],PARAMETER[""Scale_Factor"",0.9996],PARAMETER[""Latitude_Of_Origin"",0.0],UNIT[""Meter"",1.0]]","PROJCRS[""NAD_1983_CSRS_UTM_Zone_23N"",BASEGEOGCRS[""GCS_North_American_1983_CSRS"",DATUM[""D_North_American_1983_CSRS"",ELLIPSOID[""GRS_1980"",6378137.0,298.257222101,LENGTHUNIT[""Meter"",1.0]]],PRIMEM[""Greenwich"",0.0,ANGLEUNIT[""Degree"",0.0174532925199433]],CS[ellipsoidal,2],AXIS[""Latitude (lat)"",north,ORDER[1]],AXIS[""Longitude (lon)"",east,ORDER[2]],ANGLEUNIT[""Degree"",0.0174532925199433]],CONVERSION[""Transverse_Mercator"",METHOD[""Transverse_Mercator""],PARAMETER[""False_Easting"",500000.0,LENGTHUNIT[""Meter"",1.0]],PARAMETER[""False_Northing"",0.0,LENGTHUNIT[""Meter"",1.0]],PARAMETER[""Central_Meridian"",-45.0,ANGLEUNIT[""Degree"",0.0174532925199433]],PARAMETER[""Scale_Factor"",0.9996,SCALEUNIT[""Unity"",1.0]],PARAMETER[""Latitude_Of_Origin"",0.0,ANGLEUNIT[""Degree"",0.0174532925199433]]],CS[Cartesian,2],AXIS[""Easting (X)"",east,ORDER[1]],AXIS[""Northing (Y)"",north,ORDER[2]],LENGTHUNIT[""Meter"",1.0]]","NAD 1983 CSRS UTM Zone 23N","Esri","12.9.0","no","Canada - 48~~W to 42~~W",39.85,54.47,-48.0,-42.0

Want me to add logic to override this too?

Want me to add logic to override this too?

if that's not too complicated.

(by the way, I also noted a whole set of new projected CRSs, like ESRI:20904, being added under the ESRI authority where there were EPSG equivalent ones, with the difference that the EPSG ones had northing, easting axis order, whereas the ESRI WKT2 had an explicit easting, northing order. So having separated objects made sense. From what I see, they are new additions in the pe_list_projcs.csv, so that makes sense. Looks like Esri doesn't want to play with the axis order fun :-) Can't blame them... )

nyalldawson · 2021-11-16T22:42:37Z

For reference -- I've opened a ticket about the scale unit for reference time here Esri/projection-engine-db-doc#34, and also asked for clarification on the authority values. (as 9459 is another transformation which is listed with ESRI as the authority when it should be EPSG)

nyalldawson · 2021-11-17T00:42:08Z

@rouault

Looking at adding some extra projection method handling in here, I'm hitting a situation I can't explain.

This SQL always fails:

INSERT INTO "conversion" VALUES('ESRI','102022','unnamed',NULL,'EPSG','9822','Albers Equal Area','EPSG','8801','Latitude of natural origin',0.0,'EPSG','9102','EPSG','8802','Longitude of natural origin',25.0,'EPSG','9102','EPSG','8823','Latitude of 1st standard parallel',20.0,'EPSG','9102','EPSG','8824','Latitude of 2nd standard parallel',-23.0,'EPSG','9102','EPSG','8806','False easting',0.0,'EPSG','9001','EPSG','8807','False northing',0.0,'EPSG','9001',NULL,NULL,NULL,NULL,NULL,NULL,0);

with the error "Error: insert on conversion violates constraint: bad parameters for Albers Equal Area"

but if I change the final value (deprecated) from 0->1, it's happy and the insert works:

INSERT INTO "conversion" VALUES('ESRI','102022','unnamed',NULL,'EPSG','9822','Albers Equal Area','EPSG','8801','Latitude of natural origin',0.0,'EPSG','9102','EPSG','8802','Longitude of natural origin',25.0,'EPSG','9102','EPSG','8823','Latitude of 1st standard parallel',20.0,'EPSG','9102','EPSG','8824','Latitude of 2nd standard parallel',-23.0,'EPSG','9102','EPSG','8806','False easting',0.0,'EPSG','9001','EPSG','8807','False northing',0.0,'EPSG','9001',NULL,NULL,NULL,NULL,NULL,NULL,1);

I can't see any constraint which explains why the row is insert is rejected when deprecated is 0. Do you have any insights?

nyalldawson · 2021-11-17T01:02:58Z

I can't see any constraint which explains why the row is insert is rejected when deprecated is 0. Do you have any insights?

Ah got it -- it's all in conversion_triggers.sql... 👍

rouault · 2021-11-17T10:27:53Z

Ah got it -- it's all in conversion_triggers.sql... +1

yes, this is derived from the parameter names and orders as found in EPSG. Your entry has at least 2 differences in the parameters : 'Latitude of natural origin' vs 'Latitude of false origin' and similarly for longitude. I'm not aware of variants of Albers Equal Area so the semantics of the parameters are likely the same, just Esri using slightly different names.

rouault · 2021-11-20T21:56:30Z

Can I suggest we merge this as is, then I'll open atomic prs for adding the enhanced logic?

ok, just merging it

nyalldawson added 3 commits November 8, 2021 15:30

Update ESRI Projection Engine db to version 12.9

8a5240c

Update build_db_from_esri.py to use wkt2 definitions instead of wkt1

7641c59

Update ESRI projection db using wkt2 definitions

ee17e57

Force conversion of Unity units to Years for epochs in esri projectio…

d980257

…n engine db

When matching conversions from ESRI db, keep searching if we find

d72a09d

a deprecated conversion for a non-deprecated CRS

nyalldawson force-pushed the esri_12_9 branch from 1312cf7 to d72a09d Compare November 16, 2021 06:27

rouault reviewed Nov 16, 2021

View reviewed changes

rouault merged commit c7219b4 into OSGeo:master Nov 20, 2021

rouault added this to the 9.0.0 milestone Nov 20, 2021

agiudiceandrea mentioned this pull request Dec 13, 2021

Update internal srsdb and legacy mappings qgis/QGIS#43712

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ESRI projection engine db to version 12.9 #2943

Update ESRI projection engine db to version 12.9 #2943

nyalldawson commented Nov 16, 2021 •

edited

Loading

nyalldawson commented Nov 16, 2021

nyalldawson commented Nov 16, 2021

nyalldawson commented Nov 16, 2021

rouault left a comment

rouault Nov 16, 2021

nyalldawson Nov 16, 2021

rouault Nov 16, 2021

nyalldawson Nov 16, 2021

nyalldawson Nov 20, 2021

rouault Nov 16, 2021

nyalldawson Nov 16, 2021

rouault Nov 16, 2021 •

edited

Loading

nyalldawson commented Nov 16, 2021

nyalldawson commented Nov 17, 2021

nyalldawson commented Nov 17, 2021

rouault commented Nov 17, 2021

rouault commented Nov 20, 2021

Update ESRI projection engine db to version 12.9 #2943

Update ESRI projection engine db to version 12.9 #2943

Conversation

nyalldawson commented Nov 16, 2021 • edited Loading

nyalldawson commented Nov 16, 2021

nyalldawson commented Nov 16, 2021

nyalldawson commented Nov 16, 2021

rouault left a comment

Choose a reason for hiding this comment

rouault Nov 16, 2021

Choose a reason for hiding this comment

nyalldawson Nov 16, 2021

Choose a reason for hiding this comment

rouault Nov 16, 2021

Choose a reason for hiding this comment

nyalldawson Nov 16, 2021

Choose a reason for hiding this comment

nyalldawson Nov 20, 2021

Choose a reason for hiding this comment

rouault Nov 16, 2021

Choose a reason for hiding this comment

nyalldawson Nov 16, 2021

Choose a reason for hiding this comment

rouault Nov 16, 2021 • edited Loading

Choose a reason for hiding this comment

nyalldawson commented Nov 16, 2021

nyalldawson commented Nov 17, 2021

nyalldawson commented Nov 17, 2021

rouault commented Nov 17, 2021

rouault commented Nov 20, 2021

nyalldawson commented Nov 16, 2021 •

edited

Loading

rouault Nov 16, 2021 •

edited

Loading