New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove data/epsg, IGNF and esri.* files / support legacy +init=epsg:XXXX syntax #1182

Merged
merged 38 commits into from Dec 3, 2018

Conversation

Projects
None yet
3 participants
@rouault
Member

rouault commented Nov 26, 2018

This is an attempt at resolving the issues raised in #1178

  • compatibility emulation of +init=epsg:XXXX / +init=IGNF:XXXX syntax using the database to do the resolution, and with the assumptions we traditionnaly did on axis order and units. This emulation is enabled in the following cases:
    - using cs2cs,
    - using proj_api.h API: pj_init(), pj_init_ctx(), pj_init_plus(), pj_init_plux_ctx()
    - or setting PROJ_USE_PROJ4_INIT_RULES to YES / proj_context_use_proj4_init_rules(ctx, TRUE)
  • cs2cs modified to use proj_create_crs_to_crs()
  • cs2cs now accepts syntax like "cs2cs EPSG:4326 +to EPSG:32631" in which case, it will assume EPSG axis order and unit. If using "cs2cs +init=epsg:4326 ...", it will assume traditional order.
  • proj_create_crs_to_crs() modified to use the C++ code to compute the best transformation pipeline.
  • proj_create_crs_to_crs() can accept as source and target values, and string accepted by the C++ createFromUserInput() API, that is a AUTH:CODE, a PROJ string or a WKT string.
  • note: so, as cs2cs is using proj_create_crs_to_crs(), one can use cs2cs with WKT strings as well: "cs2cs 'PROJ_CS[....]' +to 'PROJ_CS[...]'"
  • cs2cs extended to support cs2cs [-options] <SOURCE> <TARGET> (without +to), but in that mode no input files may be specified (only from standard input)
  • if using proj_create_crs_to_crs() with "EPSG:XXXX" values, it will assume EPSG axis order and unit (so typically lat-long degree for geodetic CRS): this is a breaking change. If using it with "+init=epsg:XXXX" values, it will assume traditional order and unit (long-lat radians for geodetic CRS)
  • and all needed changes to be able to remove the data/epsg, IGNF and esri.* legacy files

rouault added some commits Nov 21, 2018

Make proj_create_crs_to_crs() use proj_obj_create_operations() and us…
…e area of use argument, and make createFromUserInput() recognize init=epsg: / init=IGNF: in legacy mode, that is when proj_context_get_use_proj4_init_rules() is used
@kbevers

This PR seems important enough that I (and others) should give a proper review. This is just to say that I haven't done so and please don't merge before I (and others) have had a change to look it over. I am travelling the next couple of days so there's a good chance I'll have a few hours of uninterrupted quality review time in an airport some where :-)

@kbevers

Overall, I think you have done a good job finding a nice solution to a very difficult problem and I have only minor details to discuss.

I don't have that many specific comments to the code, only a few in cs2cs.cpp.

Now that proj_create_crs_to_crs() accepts proj-strings, EPSG-like codes and WKT(2) strings the documentation should be adjusted accordingly. I also would suggest renaming the arguments srid_from/srid_to to from/to to make it clear that more than just an EPSG-code can be supplied.

Concerning the input to cs2cs, I think that using +to looks weird when used with EPSG-codes and WKT strings. How about dropping the +to, so that calls to cs2cs are on the form cs2cs <FROM> <TO>? The +to will still have to be used when using proj-strings as input, unless they are in quotes: cs2cs "+proj=latlong +ellps=GRS80" "+proj=utm +zone=32". This also opens the door for a removal of the +to syntax in the future.

In case there are several transformations available between two crs'es, is it possible with cs2cs to select which one to use? Or would that require a combination of projinfo and cct?

Before this is merged the cs2cs docs should be updated.

emess(3, "projection initialization failure\ncause: %s",
pj_strerrno(pj_errno));
PJ_OBJ *dst = nullptr;
if (!toStr.empty()) {

This comment has been minimized.

@kbevers

kbevers Nov 28, 2018

Member

This block looks more or less identical to the one above (lines 418-448). Maybe refactor into a function?

This comment has been minimized.

@rouault
"missing target CRS and source CRS is not a projected CRS");
}
} else if (fromStr.empty()) {
assert(dst);

This comment has been minimized.

@kbevers

kbevers Nov 28, 2018

Member

Similar to the above comment. These looks like to identical blocks that can be refactored into a function.

This comment has been minimized.

@rouault
pipeline = proj_create_crs_to_crs(nullptr, fromStr.c_str(), toStr.c_str(),
nullptr);
if (!pipeline) {
emess(3, "cannot initialize pipeline\ncause: %s",

This comment has been minimized.

@kbevers

kbevers Nov 28, 2018

Member

Is "pipeline" the best word to use here? I think "transformation" would be more familiar to the users. I know that it is not necessarily correct, depending on the input, but I don't think "operation" is any better than "pipeline"...

This comment has been minimized.

@rouault
if( P->is_long_wrap_set ) {
if( coo.lpz.lam != HUGE_VAL ) {
double val = fmod(coo.lpz.lam , M_TWOPI);

This comment has been minimized.

@kbevers

kbevers Nov 28, 2018

Member

Isn't this doing the same thing as adjlon() does? Or am I missing something?

This comment has been minimized.

@rouault

rouault Nov 28, 2018

Member

Similar indeed. adjlon() reframe in [-PI,PI], whereas the long_wrap mechanism reframe in [long_wrap - PI, long_wrap + PI]. I just copied this snippet from pj_transform.c. I've modified this as P->long_wrap_center + adjlon(coo.lpz.lam - P->long_wrap_center);

@@ -794,7 +794,12 @@ PJ *proj_create_crs_to_crs (PJ_CONTEXT *ctx, const char *srid_from, const char
return NULL;
}
P = proj_create(ctx, proj_string);
if( proj_string[0] == '\0' ) {
/* Null transform ? */

This comment has been minimized.

@kbevers

kbevers Nov 28, 2018

Member

Maybe we should add a proper null operation? We can call it noop.

This comment has been minimized.

@rouault

rouault Nov 28, 2018

Member

Traced as #1185 . Low priority

@rouault

This comment has been minimized.

Member

rouault commented Nov 28, 2018

I also would suggest renaming the arguments srid_from/srid_to to from/to

ok

I think that using +to looks weird when used with EPSG-codes and WKT strings. How about dropping the +to, so that calls to cs2cs are on the form cs2cs <FROM> <TO>?

Hum, I initially had the same idea, but I discovered while refactoring it that cs2cs could accept input files : cs2cs +proj= .. +to +proj= ... first_file second_file ... . And it can also support omission of the source CRS before +to or the omission of the target CRS after the +to. So the +to is central to determine which if and which of the CRS have been specified and distinguish them from input files. That said if we have only 2 arguments (ie not prefixed by '-'), and none of them is +to, then we can assume they are <FROM> and <TO>

In case there are several transformations available between two crs'es, is it possible with cs2cs to select which one to use?

No, only the one that is sorted first by create_operations() will be used.

Or would that require a combination of projinfo and cct?

Yes

Before this is merged the cs2cs docs should be updated.

If you don't mind, I'll rather create a ticket about that. I'd like to advance on the coding side of GDAL, and then libgeotiff, to check that the PROJ side of things has no holes.

rouault added some commits Nov 23, 2018

Redirect epsg:XXXX and IGNF:XXXX CRS expansions to the database, and …
…remove the data/epsg and data/IGNF files

@rouault rouault force-pushed the rouault:plug_new_code branch from 02bdfd0 to 47cada7 Nov 28, 2018

rouault added some commits Nov 28, 2018

Build: change back link-time-optimization default to off
I've found that if building PROJ with lto, but GDAL without lto, exceptions
thrown in PROJ are not properly captured by try {} blocks in PROJ C API,
and unexpectedly go back up to GDAL.
Might be a defect of the particular compiler I use (gcc 5.4 Ubuntu 16.04)
@rouault

This comment has been minimized.

Member

rouault commented Nov 28, 2018

That said if we have only 2 arguments (ie not prefixed by '-'), and none of them is +to, then we can assume they are <FROM> and <TO>

Done

rouault added some commits Nov 29, 2018

C API extensions and renaming
- proj_obj_create_projected_XXXXX() are renamed to
  proj_obj_create_conversion_snake_case() and just instanciate
  a Conversion object
- Advanced manipulation functions are moved to a dedicated
  section at bottom of proj.h
- New C API needed for GDAL OGRSpatialReference
@kbevers

This comment has been minimized.

Member

kbevers commented Nov 29, 2018

If you don't mind, I'll rather create a ticket about that. I'd like to advance on the coding side of GDAL, and then libgeotiff, to check that the PROJ side of things has no holes.

Sure, it can wait, as long as you are confident you can remember the detail in a couple of months when you get around to doing it. Also, I am a bit unsure, how does the workings of cs2cs affect GDAL and libgeotiff?

@rouault

This comment has been minimized.

Member

rouault commented Nov 29, 2018

as long as you are confident you can remember the detail in a couple of months

The intro text of this PR onto which I've pointed #1186 captures well the changes

Also, I am a bit unsure, how does the workings of cs2cs affect GDAL and libgeotiff?

Wasn't specific to cs2cs, but more a general remark on being sure PROJ is ready for GDAL and libgeotiff needs. I feel I'm behind the schedule I anticipated and want to be sure that the PROJ 6 release will be OK for their needs. PROJ received all love until now, so time to give some to the other projects :-)

I've just pushed the changes regarding function renaming and moving them at bottom ofin proj.h

@rouault

This comment has been minimized.

Member

rouault commented Nov 29, 2018

@kbevers I believe I've addressed your comments (or created tickets for defered actions)

@busstoptaktik

This comment has been minimized.

Member

busstoptaktik commented Nov 30, 2018

A high level vision of PJ is it is just a mathematical formula to express a coordinate transformation. It doesn't capture the nature of objects

Actually, to me a PJ is just a bag of bits with a few embedded function pointers defining how to interpret the bits (actually a kind of limited RTTI implemented in C). At this level of (lack of) abstraction, it can represent anything.

The "one API entry per operation type" helps enforcing correct initialization, even though a sufficiently clever user can ram garbage into any parameter anyway if not understanding it properly.

But at least the individual API entries provide a reference frame for providing the relevant documentation, so as I said: You may very well convince me that this is the better choice.

On the other hand it was also you, a few years ago, who convinced me that the first versions of proj.h/proj_4D_api.c included too much material into the "proj_..." API namespace, so I learned this API surface skepticism from a highly experienced and competent schoolmaster :-)

So yes the current state of PROJ master is a hybrid beast between the "pure" mathematical historical C code and all the new C++ metadata stuff

Which is of course A-OK in an evolutionary world. And having taken the step and accepting C++11 code into the PROJ code base really opens up for some major code reductions and improvements in the historical department of the code base.

One approach could be that the Conversion class could do the math itself (or actually be sub-classed in as many classes as they are conversion methods), instead of generating a PROJ string that is instanciated as a PJ object later.

Yes, that would probably be a superior design, but would require a re-implementation of most of the library.

Entirely dropping the PROJ syntax on the other hand, would probably not be good either: Despite all its quirkiness, to me it really relates to WKT as JSON or YAML relates to XML: The less strict, but typing friendly, and human-readable version of the formal uncle. Instantiating in any syntax, and letting the object serialize itself in whatever syntax preferred is a nice feature.

@rouault

This comment has been minimized.

Member

rouault commented Nov 30, 2018

I learned this API surface skepticism from a highly experienced and competent schoolmaster :-)

"Do what I say, not what I do" :-) That said, the API, even if large, is still an abstraction. For example, it does not expose that the database implementation is SQLite (and even in the C++ code, the interaction with SQLite has been carefully isolated in the DatabaseContext class). The C++ API itself tries to limit what is visible as public exported symbols.

Entirely dropping the PROJ syntax on the other hand, would probably not be good either

Wasn't suggesting that either. There are a number of data formats using PROJ string to express the CRS of the data. It has the benefit of being compact (with the associated drawback of lacking some usefu metadata). My view is that PROJ strings are good to express coordinate transformation, especially since the pipeline introduction, but inappropriate to express a CRS object.

@rouault rouault force-pushed the rouault:plug_new_code branch 2 times, most recently from ce473d0 to 2e37e1a Nov 30, 2018

@rouault rouault force-pushed the rouault:plug_new_code branch from 2e37e1a to 270d044 Dec 1, 2018

rouault added some commits Nov 30, 2018

@rouault rouault force-pushed the rouault:plug_new_code branch 2 times, most recently from 6d3c0bd to 75739f3 Dec 1, 2018

@rouault rouault force-pushed the rouault:plug_new_code branch from 75739f3 to 03c7263 Dec 1, 2018

rouault added some commits Dec 1, 2018

@rouault rouault force-pushed the rouault:plug_new_code branch from 03c7263 to c81ad96 Dec 1, 2018

@rouault rouault force-pushed the rouault:plug_new_code branch from 5bca86a to f294346 Dec 2, 2018

@rouault rouault force-pushed the rouault:plug_new_code branch from f294346 to 0ab1867 Dec 2, 2018

rouault added some commits Dec 2, 2018

@rouault

This comment has been minimized.

Member

rouault commented Dec 3, 2018

@kbevers @busstoptaktik How do we go forward with this PR ? (my recent commits aren't related. fixes there and there)

@kbevers

This comment has been minimized.

Member

kbevers commented Dec 3, 2018

How do we go forward with this PR ?

I still have a few things that I am not completely happy with. Most of them because I still don't fully understand the complexity of all the various aspects of the new code. I also think they go deeper than just this PR so I can create individual tickets for those. So disregarding those, I would say that this PR is ready for merging. You have found a nice way to solve the problem in #1178. I hope this is the last big PR in relation to GDALbarn so reviews will be simpler from now on :-)

@kbevers

kbevers approved these changes Dec 3, 2018

rouault added some commits Dec 3, 2018

@rouault rouault merged commit d0506e1 into OSGeo:master Dec 3, 2018

1 check passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment