Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add capability to get SQL statements to add custom CRS in the database #2577

Merged
merged 10 commits into from Mar 18, 2021

Conversation

rouault
Copy link
Member

@rouault rouault commented Mar 13, 2021

  • Add proj_insert_object_session_create(), proj_insert_object_session_destroy(), proj_get_insert_statements(), proj_suggests_code_for() to get SQL statements to add custom CRS in the database
  • projinfo: add -o SQL --output-id AUTH:NAME to get the SQL statements for a CRS object
  • Add proj_context_get_database_structure() to dump structure of empty valid auxiliary DB
  • projinfo: add a --dump-db-structure switch

Example:

# Create an auxiliary database with the definition of a custom CRS.
projinfo "+proj=merc +lat_ts=5 +datum=WGS84 +type=crs +title=my_crs" \
     --output-id HOBU:MY_CRS --dump-db-structure | sqlite3 aux.db

# Check that everything works OK
projinfo --aux-db-path aux.db HOBU:MY_CRS

outputs

PROJ.4 string:
+proj=merc +lat_ts=5 +lon_0=0 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs +type=crs

WKT2:2019 string:
PROJCRS["my_crs",
    BASEGEOGCRS["unknown",
        ENSEMBLE["World Geodetic System 1984 ensemble",
            MEMBER["World Geodetic System 1984 (Transit)"],
            MEMBER["World Geodetic System 1984 (G730)"],
            MEMBER["World Geodetic System 1984 (G873)"],
            MEMBER["World Geodetic System 1984 (G1150)"],
            MEMBER["World Geodetic System 1984 (G1674)"],
            MEMBER["World Geodetic System 1984 (G1762)"],
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]],
            ENSEMBLEACCURACY[2.0]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["HOBU","GEODETIC_CRS_MY_CRS"]],
    CONVERSION["unknown",
        METHOD["Mercator (variant B)",
            ID["EPSG",9805]],
        PARAMETER["Latitude of 1st standard parallel",5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    ID["HOBU","MY_CRS"]]

@rouault rouault added this to the 8.1.0 milestone Mar 13, 2021
@rouault
Copy link
Member Author

rouault commented Mar 13, 2021

CC @hobu

@kbevers
Copy link
Member

kbevers commented Mar 15, 2021

Interesting feature, @rouault. Do you mind sharing the motivation behind this?

I haven't really looked at the code but at least the docs looks alright to me.

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

Do you mind sharing the motivation behind this?

  • so that people can ingest custom WKTs and refer to them next with a code. @hobu might give more details about his intented use case
  • it should also make it easier to ingest later the IAU definitions that are provided as WKT.

@kbevers
Copy link
Member

kbevers commented Mar 15, 2021

so that people can ingest custom WKTs and refer to them next with a code

Yeah, so much was clear to me :-)

@hobu might give more details about his intented use case

That would be great.

Just to be clear, I think this is a great idea and a good replacement for custom init-files.

Thinking a bit further, it might also be nice to be able to do custom transformations, e.g. by running projinfo "+proj=helmert +x=12 +y=34 +z=56 ..." -s EPSG:xxxx -t EPSG:yyyy --output-id HOBU:MY_TRANSFORMATION --dump-db-structure .

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

Thinking a bit further, it might also be nice to be able to do custom transformations, e.g. by running projinfo "+proj=helmert +x=12 +y=34 +z=56 ..." -s EPSG:xxxx -t EPSG:yyyy --output-id HOBU:MY_TRANSFORMATION --dump-db-structure .

agreed, but not in my current scope of work. The "easy" solution would be to put them in the other_transformation table directly as PROJ strings. You might still need to customize the SQL or add switches to specify the accuracy, scope and extent.

@kbevers
Copy link
Member

kbevers commented Mar 15, 2021

but not in my current scope of work

I wasn't suggesting that you do it, just thinking out loud. One thing that comes to mind though is, how are the custom CRS added with this new feature usable without also registering transformations to this new CRS?

You might still need to customize the SQL or add switches to specify the accuracy, scope and extent.

Yeah, it could get complicated to capture everything in one projinfo call. Existing switches could be overridden as implied in my example above that uses -s and -t. Something similar could be done for --accuracy and --area which then only leaves the scope.

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

One thing that comes to mind though is, how are the custom CRS added with this new feature usable without also registering transformations to this new CRS?

For the use cases I was provided they use existing EPSG datums. They are typically projected CRS using us-ft as units but reyling on NAD83(2011). So the current set of EPSG transformations is fine to deal with them.

@kbevers
Copy link
Member

kbevers commented Mar 15, 2021

For the use cases I was provided they use existing EPSG datums.

I guess that works because of the +datum parameter? If I were to do this using WKT2 instead of a projstring, would it still work provided that a BASEGEOGCRS(?) is set?

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

I guess that works because of the +datum parameter?

the actual use cases are for CRS instanciated from WKT1 or WKT2

If I were to do this using WKT2 instead of a projstring, would it still work provided that a BASEGEOGCRS(?) is set?

Yes. It will try to re-use existing intermediate objects by using the CRS::identify() method as much as possible.
For example

$ projinfo  --output-id HOBU:MY_CRS -o SQL -q 'PROJCS["my CRS",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",4],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH]]' 

outputs

INSERT INTO conversion VALUES('HOBU','CONVERSION_MY_CRS','unnamed','','EPSG','9807','Transverse Mercator','EPSG','8801','Latitude of natural origin',0,'EPSG','9122','EPSG','8802','Longitude of natural origin',4,'EPSG','9122','EPSG','8805','Scale factor at natural origin',0.9996,'EPSG','9201','EPSG','8806','False easting',500000,'EPSG','9001','EPSG','8807','False northing',0,'EPSG','9001',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,0);
INSERT INTO usage VALUES('HOBU','USAGE_CONVERSION_MY_CRS','conversion','HOBU','CONVERSION_MY_CRS','PROJ','EXTENT_UNKNOWN','PROJ','SCOPE_UNKNOWN');
INSERT INTO projected_crs VALUES('HOBU','MY_CRS','my CRS','','EPSG','4400','EPSG','4326','HOBU','CONVERSION_MY_CRS',NULL,0);
INSERT INTO usage VALUES('HOBU','USAGE_PROJECTED_CRS_MY_CRS','projected_crs','HOBU','MY_CRS','PROJ','EXTENT_UNKNOWN','PROJ','SCOPE_UNKNOWN');

which shows that it re-uses EPSG:4326

@hobu
Copy link
Contributor

hobu commented Mar 15, 2021

@hobu might give more details about his intended use case

There are just too many situations that depend up on single codes where WKT of any flavor won't do. OGC services are a big culprit, but there are other cases as well.

In the US, many high precision engineering coordinate systems are either not in EPSG, are somewhere in the process of becoming registered with EPSG, or are registered EPSG CRSs that are bastardized by their users to modify things like parameters and transforms. /me shakes his fist at COMP_CS users of horizontal meters and vertical feet.

The old init file relief valve of sticking a code in there isn't such a good thing to do with PROJ 6+, and this approach gives us a more sustainable way to do such a thing. An alternative that would also be fine with me is some sort of ADDITIONAL custom proj.db being used with an environment variable denoting its location. The nice thing about the latter approach is your custom CRSs would survive software upgrades and such.

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

An alternative that would also be fine with me is some sort of ADDITIONAL custom proj.db being used with an environment variable denoting its location

The mechanics for that is already in place at the API level (we have a proj_context_set_database_path() function that can take a path to the main DB and auxiliary DBs). Maybe proj_context_create() could look at a PROJ_AUX_DB=/path/to/first.db:/path/to/second.db:... env varible to set that in a more user-friendly way ?

@kbevers
Copy link
Member

kbevers commented Mar 15, 2021

Thanks for the elaboration, Howard.

Maybe proj_context_create() could look at a PROJ_AUX_DB=/path/to/first.db:/path/to/second.db:... env varible to set that in a more user-friendly way ?

That would be quite useful, yes. It would certainly make it easier to distribute a set of custom CRS's to users, especially in cases where it is not fitting to put it in the main PROJ distribution.

@snowman2
Copy link
Contributor

One thought I had was that if you are not doing so already, it would be nice to prevent users from using the official authorities such as EPSG, ESRI, .. to help clarify that this was a custom CRS and is not from one of the official authorities.

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

One thought I had was that if you are not doing so already, it would be nice to prevent users from using the official authorities such as EPSG, ESRI, .. to help clarify that this was a custom CRS and is not from one of the official authorities.

I would not enforce that. This is the responsibility of the users. They might need to mess with official authorities as workarounds if some software is only able to support EPSG:XXXX codes.

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

look at a PROJ_AUX_DB=/path/to/first.db:/path/to/second.db:... env varible to set that in a more user-friendly way ?

done

@snowman2
Copy link
Contributor

They might need to mess with official authorities as workarounds if some software is only able to support EPSG:XXXX codes.

Since this is a new feature, they would likely need to update the underlying software anyway. I think helping direct the users towards intended usage patterns would be a good idea. Otherwise, it gets messy/confusing.

@hobu
Copy link
Contributor

hobu commented Mar 15, 2021

They might need to mess with official authorities as workarounds if some software is only able to support EPSG:XXXX codes

Is it PROJ's burden to support that? I vote no. They could still manually edit the database to do what they needed. We don't really want them adding EPSG codes. EPSG:900913 wasn't such a good thing 😉

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

Since this is a new feature, they would likely need to update the underlying software anyway. I think helping direct the users towards intended usage patterns would be a good idea. Otherwise, it gets messy/confusing.

How would we determine which authorities are allowed to have new definitions from those who aren't ? Currently in proj.db, we have EPSG, ESRI, IGNF, OGC, NKG and PROJ. And this new capability might potentially be used in the future for the ingestion scripts that populate proj.db with some of those official authorities. I think your concern would be best addressed by some note in the documentation. I'll let you follow-up in a later PR with such doc addition if you wish.

@snowman2
Copy link
Contributor

How would we determine which authorities are allowed to have new definitions from those who aren't?

I was actually thinking of only allowing one. Maybe just call it PROJ_CUSTOM and then only allow the user to enter a code for their CRS.

@kbevers
Copy link
Member

kbevers commented Mar 15, 2021

How would we determine which authorities are allowed to have new definitions from those who aren't ?

I'd say don't allow reuse of authorities already in proj.db. Authorities that are already in proj.db already knows how to get their stuff in the main database (or is sufficiently important that we do it on their behalf), so they can follow that path if they need to expand their data in the database. For any other use cases they can use a tempory AUTH like "NKG2". When it comes to proper integration in proj.db they auth-name is easily changed.

@hobu
Copy link
Contributor

hobu commented Mar 15, 2021

I was actually thinking of only allowing one. Maybe just call it PROJ_CUSTOM and then only allow the user to enter a code for their CRS.

There's still too much opportunity for clashing in this case. Your PROJ_CUSTOM:4326 and my PROJ_CUSTOM:4326 might have flipped XY axes but different units 😛

@snowman2
Copy link
Contributor

snowman2 commented Mar 15, 2021

There's still too much opportunity for clashing in this case. Your PROJ_CUSTOM:4326 and my PROJ_CUSTOM:4326 might have flipped XY axes but different units

That's true. Maybe use it as a prefix: PROJ_CUSTOM_HOBU

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

The last comments are really about debating how a tool might be misused. There are so many ways that PROJ can be misused. In ancient times, people could easily add a new line to the 'epsg' file... And we won't prevent them to edit the existing proj.db with new EPSG definitions, whatever technological barrier we put, if they are sufficiently determined to do so

@hobu
Copy link
Contributor

hobu commented Mar 15, 2021

Let's go with this for now:

  • Document what is expected for adding custom registries (what is possible and what is not)
  • State that custom additions should not be common registries such as ESRI, EPSG, or IAU (but don't enforce that in any way)
  • Guide people toward using their own PROJ_AUX_DB instead of updating proj.db directly to prevent their data getting stomped on by a software update
  • Point to the EPSG process and tell people that they should really register real world coordinate systems that are in use.

@snowman2
Copy link
Contributor

There are so many ways that PROJ can be misused.

Two things can be true at the same time. It is true someone can hack the proj.db and true that it would be good for PROJ to try to get users to use it correctly. 😄

@rouault
Copy link
Member Author

rouault commented Mar 15, 2021

Let's go with this for now:

See my last commit in this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants