Remove data/epsg, IGNF and esri.* files / support legacy +init=epsg:XXXX syntax #1182

rouault · 2018-11-26T15:38:55Z

This is an attempt at resolving the issues raised in #1178

compatibility emulation of +init=epsg:XXXX / +init=IGNF:XXXX syntax using the database to do the resolution, and with the assumptions we traditionnaly did on axis order and units. This emulation is enabled in the following cases:
- using cs2cs,
- using proj_api.h API: pj_init(), pj_init_ctx(), pj_init_plus(), pj_init_plux_ctx()
- or setting PROJ_USE_PROJ4_INIT_RULES to YES / proj_context_use_proj4_init_rules(ctx, TRUE)
cs2cs modified to use proj_create_crs_to_crs()
cs2cs now accepts syntax like "cs2cs EPSG:4326 +to EPSG:32631" in which case, it will assume EPSG axis order and unit. If using "cs2cs +init=epsg:4326 ...", it will assume traditional order.
proj_create_crs_to_crs() modified to use the C++ code to compute the best transformation pipeline.
proj_create_crs_to_crs() can accept as source and target values, and string accepted by the C++ createFromUserInput() API, that is a AUTH:CODE, a PROJ string or a WKT string.
note: so, as cs2cs is using proj_create_crs_to_crs(), one can use cs2cs with WKT strings as well: "cs2cs 'PROJ_CS[....]' +to 'PROJ_CS[...]'"
cs2cs extended to support cs2cs [-options] <SOURCE> <TARGET> (without +to), but in that mode no input files may be specified (only from standard input)
if using proj_create_crs_to_crs() with "EPSG:XXXX" values, it will assume EPSG axis order and unit (so typically lat-long degree for geodetic CRS): this is a breaking change. If using it with "+init=epsg:XXXX" values, it will assume traditional order and unit (long-lat radians for geodetic CRS)
and all needed changes to be able to remove the data/epsg, IGNF and esri.* legacy files

…ing authname

…instead

…nd units

…e area of use argument, and make createFromUserInput() recognize init=epsg: / init=IGNF: in legacy mode, that is when proj_context_get_use_proj4_init_rules() is used

kbevers

This PR seems important enough that I (and others) should give a proper review. This is just to say that I haven't done so and please don't merge before I (and others) have had a change to look it over. I am travelling the next couple of days so there's a good chance I'll have a few hours of uninterrupted quality review time in an airport some where :-)

kbevers

Overall, I think you have done a good job finding a nice solution to a very difficult problem and I have only minor details to discuss.

I don't have that many specific comments to the code, only a few in cs2cs.cpp.

Now that proj_create_crs_to_crs() accepts proj-strings, EPSG-like codes and WKT(2) strings the documentation should be adjusted accordingly. I also would suggest renaming the arguments srid_from/srid_to to from/to to make it clear that more than just an EPSG-code can be supplied.

Concerning the input to cs2cs, I think that using +to looks weird when used with EPSG-codes and WKT strings. How about dropping the +to, so that calls to cs2cs are on the form cs2cs <FROM> <TO>? The +to will still have to be used when using proj-strings as input, unless they are in quotes: cs2cs "+proj=latlong +ellps=GRS80" "+proj=utm +zone=32". This also opens the door for a removal of the +to syntax in the future.

In case there are several transformations available between two crs'es, is it possible with cs2cs to select which one to use? Or would that require a combination of projinfo and cct?

Before this is merged the cs2cs docs should be updated.

kbevers · 2018-11-28T21:53:01Z

src/cs2cs.cpp

-            emess(3, "projection initialization failure\ncause: %s",
-                  pj_strerrno(pj_errno));
+    PJ_OBJ *dst = nullptr;
+    if (!toStr.empty()) {


This block looks more or less identical to the one above (lines 418-448). Maybe refactor into a function?

kbevers · 2018-11-28T21:54:08Z

src/cs2cs.cpp

+                  "missing target CRS and source CRS is not a projected CRS");
+        }
+    } else if (fromStr.empty()) {
+        assert(dst);


Similar to the above comment. These looks like to identical blocks that can be refactored into a function.

kbevers · 2018-11-28T21:56:25Z

src/cs2cs.cpp

+    pipeline = proj_create_crs_to_crs(nullptr, fromStr.c_str(), toStr.c_str(),
+                                      nullptr);
+    if (!pipeline) {
+        emess(3, "cannot initialize pipeline\ncause: %s",


Is "pipeline" the best word to use here? I think "transformation" would be more familiar to the users. I know that it is not necessarily correct, depending on the input, but I don't think "operation" is any better than "pipeline"...

kbevers · 2018-11-28T21:59:53Z

src/pj_fwd.c

+
+        if( P->is_long_wrap_set ) {
+            if( coo.lpz.lam != HUGE_VAL ) {
+                double val = fmod(coo.lpz.lam , M_TWOPI);


Isn't this doing the same thing as adjlon() does? Or am I missing something?

Similar indeed. adjlon() reframe in [-PI,PI], whereas the long_wrap mechanism reframe in [long_wrap - PI, long_wrap + PI]. I just copied this snippet from pj_transform.c. I've modified this as P->long_wrap_center + adjlon(coo.lpz.lam - P->long_wrap_center);

kbevers · 2018-11-28T22:01:19Z

src/proj_4D_api.c

@@ -794,7 +794,12 @@ PJ  *proj_create_crs_to_crs (PJ_CONTEXT *ctx, const char *srid_from, const char
        return NULL;
    }

-    P = proj_create(ctx, proj_string);
+    if( proj_string[0] == '\0' ) {
+        /* Null transform ? */


Maybe we should add a proper null operation? We can call it noop.

Traced as #1185 . Low priority

rouault · 2018-11-28T22:46:51Z

I also would suggest renaming the arguments srid_from/srid_to to from/to

ok

I think that using +to looks weird when used with EPSG-codes and WKT strings. How about dropping the +to, so that calls to cs2cs are on the form cs2cs <FROM> <TO>?

Hum, I initially had the same idea, but I discovered while refactoring it that cs2cs could accept input files : cs2cs +proj= .. +to +proj= ... first_file second_file ... . And it can also support omission of the source CRS before +to or the omission of the target CRS after the +to. So the +to is central to determine which if and which of the CRS have been specified and distinguish them from input files. That said if we have only 2 arguments (ie not prefixed by '-'), and none of them is +to, then we can assume they are <FROM> and <TO>

In case there are several transformations available between two crs'es, is it possible with cs2cs to select which one to use?

No, only the one that is sorted first by create_operations() will be used.

Or would that require a combination of projinfo and cct?

Yes

Before this is merged the cs2cs docs should be updated.

If you don't mind, I'll rather create a ticket about that. I'd like to advance on the coding side of GDAL, and then libgeotiff, to check that the PROJ side of things has no holes.

…remove the data/epsg and data/IGNF files

I've found that if building PROJ with lto, but GDAL without lto, exceptions thrown in PROJ are not properly captured by try {} blocks in PROJ C API, and unexpectedly go back up to GDAL. Might be a defect of the particular compiler I use (gcc 5.4 Ubuntu 16.04)

rouault · 2018-11-28T23:43:40Z

That said if we have only 2 arguments (ie not prefixed by '-'), and none of them is +to, then we can assume they are <FROM> and <TO>

Done

- proj_obj_create_projected_XXXXX() are renamed to proj_obj_create_conversion_snake_case() and just instanciate a Conversion object - Advanced manipulation functions are moved to a dedicated section at bottom of proj.h - New C API needed for GDAL OGRSpatialReference

kbevers · 2018-11-29T11:11:50Z

If you don't mind, I'll rather create a ticket about that. I'd like to advance on the coding side of GDAL, and then libgeotiff, to check that the PROJ side of things has no holes.

Sure, it can wait, as long as you are confident you can remember the detail in a couple of months when you get around to doing it. Also, I am a bit unsure, how does the workings of cs2cs affect GDAL and libgeotiff?

rouault · 2018-11-29T11:17:38Z

as long as you are confident you can remember the detail in a couple of months

The intro text of this PR onto which I've pointed #1186 captures well the changes

Also, I am a bit unsure, how does the workings of cs2cs affect GDAL and libgeotiff?

Wasn't specific to cs2cs, but more a general remark on being sure PROJ is ready for GDAL and libgeotiff needs. I feel I'm behind the schedule I anticipated and want to be sure that the PROJ 6 release will be OK for their needs. PROJ received all love until now, so time to give some to the other projects :-)

I've just pushed the changes regarding function renaming and moving them at bottom ofin proj.h

… test cases

rouault · 2018-11-29T14:40:37Z

@kbevers I believe I've addressed your comments (or created tickets for defered actions)

busstoptaktik · 2018-11-30T14:06:02Z

A high level vision of PJ is it is just a mathematical formula to express a coordinate transformation. It doesn't capture the nature of objects

Actually, to me a PJ is just a bag of bits with a few embedded function pointers defining how to interpret the bits (actually a kind of limited RTTI implemented in C). At this level of (lack of) abstraction, it can represent anything.

The "one API entry per operation type" helps enforcing correct initialization, even though a sufficiently clever user can ram garbage into any parameter anyway if not understanding it properly.

But at least the individual API entries provide a reference frame for providing the relevant documentation, so as I said: You may very well convince me that this is the better choice.

On the other hand it was also you, a few years ago, who convinced me that the first versions of proj.h/proj_4D_api.c included too much material into the "proj_..." API namespace, so I learned this API surface skepticism from a highly experienced and competent schoolmaster :-)

So yes the current state of PROJ master is a hybrid beast between the "pure" mathematical historical C code and all the new C++ metadata stuff

Which is of course A-OK in an evolutionary world. And having taken the step and accepting C++11 code into the PROJ code base really opens up for some major code reductions and improvements in the historical department of the code base.

One approach could be that the Conversion class could do the math itself (or actually be sub-classed in as many classes as they are conversion methods), instead of generating a PROJ string that is instanciated as a PJ object later.

Yes, that would probably be a superior design, but would require a re-implementation of most of the library.

Entirely dropping the PROJ syntax on the other hand, would probably not be good either: Despite all its quirkiness, to me it really relates to WKT as JSON or YAML relates to XML: The less strict, but typing friendly, and human-readable version of the formal uncle. Instantiating in any syntax, and letting the object serialize itself in whatever syntax preferred is a nice feature.

rouault · 2018-11-30T14:19:23Z

I learned this API surface skepticism from a highly experienced and competent schoolmaster :-)

"Do what I say, not what I do" :-) That said, the API, even if large, is still an abstraction. For example, it does not expose that the database implementation is SQLite (and even in the C++ code, the interaction with SQLite has been carefully isolated in the DatabaseContext class). The C++ API itself tries to limit what is visible as public exported symbols.

Entirely dropping the PROJ syntax on the other hand, would probably not be good either

Wasn't suggesting that either. There are a number of data formats using PROJ string to express the CRS of the data. It has the benefit of being compact (with the associated drawback of lacking some usefu metadata). My view is that PROJ strings are good to express coordinate transformation, especially since the pipeline introduction, but inappropriate to express a CRS object.

…experimental.h header

…no match

rouault · 2018-12-03T11:14:00Z

@kbevers @busstoptaktik How do we go forward with this PR ? (my recent commits aren't related. fixes there and there)

kbevers · 2018-12-03T12:28:53Z

How do we go forward with this PR ?

I still have a few things that I am not completely happy with. Most of them because I still don't fully understand the complexity of all the various aspects of the new code. I also think they go deeper than just this PR so I can create individual tickets for those. So disregarding those, I would say that this PR is ready for merging. You have found a nice way to solve the problem in #1178. I hope this is the last big PR in relation to GDALbarn so reviews will be simpler from now on :-)

…base when possible

rouault added 7 commits November 21, 2018 21:49

createFromUserInput("authname:code"): make it case insensitive regard…

afb37da

…ing authname

rtodms(): make it output decimal point when locale has comma decimal …

fe59ae4

…instead

Move 'builtins' test of src/gie.c to test/unit/gie_self_tests.cpp

85a4e91

Fix transformation between geographic CRS that differ by axis order a…

b9d5024

…nd units

Make proj_create_crs_to_crs() use proj_obj_create_operations() and us…

549268f

…e area of use argument, and make createFromUserInput() recognize init=epsg: / init=IGNF: in legacy mode, that is when proj_context_get_use_proj4_init_rules() is used

Rename cs2cs.c to cs2cs.cpp with minimal changes to make it compile

a7f696c

Run reformat_cpp.sh on cs2cs.cpp

d48f971

kbevers requested changes Nov 26, 2018

View reviewed changes

rouault mentioned this pull request Nov 26, 2018

Missing datum shift for DB_REF coordinate systems #1167

Closed

kbevers requested changes Nov 28, 2018

View reviewed changes

rouault mentioned this pull request Nov 28, 2018

Document cs2cs changes done per https://github.com/OSGeo/proj.4/pull/1182 #1186

Closed

rouault force-pushed the plug_new_code branch from ea95dad to 02bdfd0 Compare November 28, 2018 23:28

rouault added 6 commits November 29, 2018 00:35

cs2cs: upgrade to use proj_create_crs_to_crs()

a66c122

Reformat test .cpp files

c57e293

Add unit test for pj_tranform() now that cs2cs no longer use it

7e05bd0

Redirect epsg:XXXX and IGNF:XXXX CRS expansions to the database, and …

67758b2

…remove the data/epsg and data/IGNF files

Remove data/esri and data/esri.extra files

1a4a056

Update proj_symbol_rename.h

47cada7

rouault force-pushed the plug_new_code branch from 02bdfd0 to 47cada7 Compare November 28, 2018 23:35

rouault added 3 commits November 29, 2018 00:39

importFromWKT: check we have a valid unit where we need one

ff7f5da

exportToWKT WKT1_GDAL: export axis by default for GeocentricCRS

d87573c

rouault added 2 commits November 29, 2018 02:52

Preserve EPSG code when importFromWKT WKT1_GDAL of EPSG:3857

6d9a1a9

proj_create_crs_to_crs(): rename arguments, update doc, and add a few…

7b3ab52

… test cases

API: move all advanced PJ_OBJ creation functions in a dedicated proj_…

b6a5c44

…experimental.h header

rouault force-pushed the plug_new_code branch 2 times, most recently from ce473d0 to 2e37e1a Compare November 30, 2018 21:26

importFromWKT: morph GDAL_WKT1 datum names into their EPSG spelling

9d19d55

rouault force-pushed the plug_new_code branch from 2e37e1a to 270d044 Compare December 1, 2018 16:36

rouault added 3 commits December 1, 2018 17:53

projinfo: add a --single-line option

872751c

Add ESRI 'Central_Parallel' as an alias of other similar parameters

fa11a21

importFromWKT: deal with uncommon formulation of EPSG:3857

18dbc00

rouault force-pushed the plug_new_code branch 2 times, most recently from 6d3c0bd to 75739f3 Compare December 1, 2018 17:21

Rename test/old/ to test/cli/ to better reflect their nature

23bf8bf

rouault force-pushed the plug_new_code branch from 75739f3 to 03c7263 Compare December 1, 2018 18:15

rouault added 2 commits December 1, 2018 20:21

Fix PROJ_GRID_AVAILABILITY_IGNORED

0a74607

Add testing of projinfo utility

c81ad96

rouault force-pushed the plug_new_code branch from 03c7263 to c81ad96 Compare December 1, 2018 19:22

C API: rename parameter to reflect its output status

ccae82e

rouault force-pushed the plug_new_code branch from 5bca86a to f294346 Compare December 2, 2018 18:44

improve identify() for projected and bound CRS

0ab1867

rouault force-pushed the plug_new_code branch from f294346 to 0ab1867 Compare December 2, 2018 20:31

rouault added 2 commits December 2, 2018 22:27

identify: improve GeographicCRS identification when the CRS name has …

9577f8e

…no match

importFromWKT: add another alias of WebMercator

98ebc46

kbevers approved these changes Dec 3, 2018

View reviewed changes

rouault added 2 commits December 3, 2018 13:51

WKTParser: fix to avoid creation of empty nodes

2d8f295

Export to ESRI WKT: make it use verbatim WKT definition from the data…

ba111ac

…base when possible

rouault merged commit d0506e1 into OSGeo:master Dec 3, 2018

rouault mentioned this pull request Dec 20, 2018

Integration of RFC2 work with existing PROJ code: axis order and unit issues #1178

Closed

Remove data/epsg, IGNF and esri.* files / support legacy +init=epsg:XXXX syntax #1182

Remove data/epsg, IGNF and esri.* files / support legacy +init=epsg:XXXX syntax #1182

Uh oh!

Conversation

rouault commented Nov 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kbevers left a comment

Choose a reason for hiding this comment

Uh oh!

kbevers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rouault commented Nov 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rouault commented Nov 28, 2018

Uh oh!

kbevers commented Nov 29, 2018

Uh oh!

rouault commented Nov 29, 2018

Uh oh!

rouault commented Nov 29, 2018

Uh oh!

busstoptaktik commented Nov 30, 2018

Uh oh!

rouault commented Nov 30, 2018

Uh oh!

rouault commented Dec 3, 2018

Uh oh!

kbevers commented Dec 3, 2018

Uh oh!

Uh oh!

rouault commented Nov 26, 2018 •

edited

Loading

rouault commented Nov 28, 2018 •

edited

Loading