Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ESRI projection engine db to version 12.9 #2943

Merged
merged 5 commits into from
Nov 20, 2021

Conversation

nyalldawson
Copy link
Contributor

@nyalldawson nyalldawson commented Nov 16, 2021

This fixes #2929 by updating the projection engine db to version 12.9.

The first commit runs the older version of the build_db_from_esri.py script unchanged, so contains only the modifications made in version 12.9 But the 12.9 version also introduces a WKT2 definition of the objects, so I've updated the script to use these instead. The final commit is the resultant changes to the tables as a consequence of parsing WKT2 instead of WKT1.

Some notes:

  1. I'd like to clean up the script, but for now I've only done the minimal changes to get the WKT2 parsing working
  2. I stepped very cautiously here to ensure no breakage -- there was lots of asserts comparing new vs old results which I've stripped out in the final commit.
  3. While there's very minimal actual changes to the definitions as a result of moving to wkt2 (basically some parameter values are specified in milliarcseconds instead of arcseconds), we do gain a whole bunch of extra security by the move (specifically, ensuring that the unit handling for parameters is correct)
  4. There's one change in the db which resulted from using wkt2 which I'm not sure about. The Epoch units have been changed from Years (EPSG:1029) to "Unity" (EPSG:9201). This is coming straight from the Wkt 2 definitions of the operations from the ESRI files. I'm happy to hardcode an exception in here if this will impact proj negatively. I've hardcoded a conversion from Unity units to Years to satisfy the helmert_transformation table constraints
  5. I can confirm that there's no changes resulting from rebuilding the db from scratch, as noted in ESRI Projection Engine 2.9 update available #2929 (comment)

@nyalldawson
Copy link
Contributor Author

@rouault I'm a little unsure how to handle the "insert on projected_crs violates constraint: conversion must not be deprecated when projected_crs is not deprecated" errors. Should I force the CRS to be flagged as deprecated when it uses a deprecated conversion (even though it's not marked as deprecated into the ESRI files), or should I "undeprecate" the conversion?

@nyalldawson
Copy link
Contributor Author

I'm a little unsure how to handle the "insert on projected_crs violates constraint: conversion must not be deprecated when projected_crs is not deprecated" errors. Should I force the CRS to be flagged as deprecated when it uses a deprecated conversion (even though it's not marked as deprecated into the ESRI files), or should I "undeprecate" the conversion?

For now I'm force-deprecating the CRSes. Let me know if this is the wrong approach.

a deprecated conversion for a non-deprecated CRS
@nyalldawson
Copy link
Contributor Author

For now I'm force-deprecating the CRSes. Let me know if this is the wrong approach.

Actually -- ignore all that. I've found the real fix.

Copy link
Member

@rouault rouault left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Just a few questions

scripts/build_db_from_esri.py Show resolved Hide resolved
@@ -1705,6 +1705,24 @@ INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9379','IGb14_(3D)','ESRI');
INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9469','SRGI2013_(3D)','ESRI');
INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9546','LTF2004(G)_(3D)','ESRI');
INSERT INTO alias_name VALUES('geodetic_crs','EPSG','9695','REDGEOMIN_(3D)','ESRI');
INSERT INTO "geodetic_datum" VALUES('ESRI','1305','ETRF2000_Poland','ETRF2000 Poland','EPSG','7019','EPSG','8901',NULL,NULL,NULL,NULL,0);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know why this is promoted as a plain geodetic_datum object, and not just an alias to EPSG:1305 "ETRF2000 Poland" ?
The EPSG corresponding record is pretty similar

INSERT INTO "geodetic_datum" VALUES('EPSG','1305','ETRF2000 Poland',NULL,'EPSG','7019','EPSG','8901','2012-11-15',NULL,NULL,NULL,0);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's coming from this line: https://github.com/Esri/projection-engine-db-doc/blob/master/csv/pe_list_datum.csv#L164 , which lists the authority as ESRI.

I could add some logic here to test whether an equivalent EPSG record exists and if so, ignore the authority from the csv source file. What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could add some logic here to test whether an equivalent EPSG record exists and if so, ignore the authority from the csv source file. What do you think?

yes that could make sense. One could argue too they should credit EPSG here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest we hold off on this for now and see if we get an answer in Esri/projection-engine-db-doc#34 -- it looks like a bunch of the new records incorrectly list esri as the authority.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rouault not much action on the upstream ticket so far. Can I suggest we merge this as is, then I'll open atomic prs for adding the enhanced logic? I've come across a number of other things I think are issues and I'm concerned that this PR will become too large if I try to address them all here.

@@ -6752,6 +6770,37 @@ INSERT INTO alias_name VALUES('projected_crs','EPSG','9680','WGS_84_TM_90_NE','E
INSERT INTO alias_name VALUES('projected_crs','EPSG','9697','REDGEOMIN_UTM_zone_12S','ESRI');
INSERT INTO alias_name VALUES('projected_crs','EPSG','9698','REDGEOMIN_UTM_zone_18S','ESRI');
INSERT INTO alias_name VALUES('projected_crs','EPSG','9699','REDGEOMIN_UTM_zone_19S','ESRI');
INSERT INTO "projected_crs" VALUES('ESRI','9709','NAD_1983_CSRS_UTM_Zone_23N',NULL,'EPSG','4400','EPSG','4617','EPSG','16023',NULL,0);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know why this is promoted as a plain projected_crs object, and not just an alias to EPSG:9709 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, it's coming from the ESRI source csvs which list ESRI as the authority for this: https://raw.githubusercontent.com/Esri/projection-engine-db-doc/master/csv/pe_list_projcs.csv

9709,9709,"PE_PCS_NAD83_CSRS_UTM_23N","NAD_1983_CSRS_UTM_Zone_23N","PROJCS[""NAD_1983_CSRS_UTM_Zone_23N"",GEOGCS[""GCS_North_American_1983_CSRS"",DATUM[""D_North_American_1983_CSRS"",SPHEROID[""GRS_1980"",6378137.0,298.257222101]],PRIMEM[""Greenwich"",0.0],UNIT[""Degree"",0.0174532925199433]],PROJECTION[""Transverse_Mercator""],PARAMETER[""False_Easting"",500000.0],PARAMETER[""False_Northing"",0.0],PARAMETER[""Central_Meridian"",-45.0],PARAMETER[""Scale_Factor"",0.9996],PARAMETER[""Latitude_Of_Origin"",0.0],UNIT[""Meter"",1.0]]","PROJCRS[""NAD_1983_CSRS_UTM_Zone_23N"",BASEGEOGCRS[""GCS_North_American_1983_CSRS"",DATUM[""D_North_American_1983_CSRS"",ELLIPSOID[""GRS_1980"",6378137.0,298.257222101,LENGTHUNIT[""Meter"",1.0]]],PRIMEM[""Greenwich"",0.0,ANGLEUNIT[""Degree"",0.0174532925199433]],CS[ellipsoidal,2],AXIS[""Latitude (lat)"",north,ORDER[1]],AXIS[""Longitude (lon)"",east,ORDER[2]],ANGLEUNIT[""Degree"",0.0174532925199433]],CONVERSION[""Transverse_Mercator"",METHOD[""Transverse_Mercator""],PARAMETER[""False_Easting"",500000.0,LENGTHUNIT[""Meter"",1.0]],PARAMETER[""False_Northing"",0.0,LENGTHUNIT[""Meter"",1.0]],PARAMETER[""Central_Meridian"",-45.0,ANGLEUNIT[""Degree"",0.0174532925199433]],PARAMETER[""Scale_Factor"",0.9996,SCALEUNIT[""Unity"",1.0]],PARAMETER[""Latitude_Of_Origin"",0.0,ANGLEUNIT[""Degree"",0.0174532925199433]]],CS[Cartesian,2],AXIS[""Easting (X)"",east,ORDER[1]],AXIS[""Northing (Y)"",north,ORDER[2]],LENGTHUNIT[""Meter"",1.0]]","NAD 1983 CSRS UTM Zone 23N","Esri","12.9.0","no","Canada - 48W to 42W",39.85,54.47,-48.0,-42.0

Want me to add logic to override this too?

Copy link
Member

@rouault rouault Nov 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Want me to add logic to override this too?

if that's not too complicated.

(by the way, I also noted a whole set of new projected CRSs, like ESRI:20904, being added under the ESRI authority where there were EPSG equivalent ones, with the difference that the EPSG ones had northing, easting axis order, whereas the ESRI WKT2 had an explicit easting, northing order. So having separated objects made sense. From what I see, they are new additions in the pe_list_projcs.csv, so that makes sense. Looks like Esri doesn't want to play with the axis order fun :-) Can't blame them... )

@nyalldawson
Copy link
Contributor Author

For reference -- I've opened a ticket about the scale unit for reference time here Esri/projection-engine-db-doc#34, and also asked for clarification on the authority values. (as 9459 is another transformation which is listed with ESRI as the authority when it should be EPSG)

@nyalldawson
Copy link
Contributor Author

@rouault

Looking at adding some extra projection method handling in here, I'm hitting a situation I can't explain.

This SQL always fails:

INSERT INTO "conversion" VALUES('ESRI','102022','unnamed',NULL,'EPSG','9822','Albers Equal Area','EPSG','8801','Latitude of natural origin',0.0,'EPSG','9102','EPSG','8802','Longitude of natural origin',25.0,'EPSG','9102','EPSG','8823','Latitude of 1st standard parallel',20.0,'EPSG','9102','EPSG','8824','Latitude of 2nd standard parallel',-23.0,'EPSG','9102','EPSG','8806','False easting',0.0,'EPSG','9001','EPSG','8807','False northing',0.0,'EPSG','9001',NULL,NULL,NULL,NULL,NULL,NULL,0);

with the error "Error: insert on conversion violates constraint: bad parameters for Albers Equal Area"

but if I change the final value (deprecated) from 0->1, it's happy and the insert works:

INSERT INTO "conversion" VALUES('ESRI','102022','unnamed',NULL,'EPSG','9822','Albers Equal Area','EPSG','8801','Latitude of natural origin',0.0,'EPSG','9102','EPSG','8802','Longitude of natural origin',25.0,'EPSG','9102','EPSG','8823','Latitude of 1st standard parallel',20.0,'EPSG','9102','EPSG','8824','Latitude of 2nd standard parallel',-23.0,'EPSG','9102','EPSG','8806','False easting',0.0,'EPSG','9001','EPSG','8807','False northing',0.0,'EPSG','9001',NULL,NULL,NULL,NULL,NULL,NULL,1);

I can't see any constraint which explains why the row is insert is rejected when deprecated is 0. Do you have any insights?

@nyalldawson
Copy link
Contributor Author

I can't see any constraint which explains why the row is insert is rejected when deprecated is 0. Do you have any insights?

Ah got it -- it's all in conversion_triggers.sql... 👍

@rouault
Copy link
Member

rouault commented Nov 17, 2021

Ah got it -- it's all in conversion_triggers.sql... +1

yes, this is derived from the parameter names and orders as found in EPSG. Your entry has at least 2 differences in the parameters : 'Latitude of natural origin' vs 'Latitude of false origin' and similarly for longitude. I'm not aware of variants of Albers Equal Area so the semantics of the parameters are likely the same, just Esri using slightly different names.

@rouault
Copy link
Member

rouault commented Nov 20, 2021

Can I suggest we merge this as is, then I'll open atomic prs for adding the enhanced logic?

ok, just merging it

@rouault rouault merged commit c7219b4 into OSGeo:master Nov 20, 2021
@rouault rouault added this to the 9.0.0 milestone Nov 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ESRI Projection Engine 2.9 update available
2 participants