Remove proj_api.h

[kbevers]

Removes proj_api.h from the public API. The contents of the header file
has been moved to proj_internal.h verbatim and any references to
proj_api.h has been changed to proj_internal.h.

The documentation of proj_api.h has been removed from both sphinx and
the ancient pj_init.3 man-file. The only exception to this is the API
migration guides which still mention the old API.

This is a first low impact attempt at getting rid of the old API. More can be done, like removing pj_transform() and other old API functions that are not in use internally. I may do that in later commits. Speak up if you think that is a bad idea.

• Closes Remove proj_api.h as a public header file #837
• Tests modified
• Added clear title that can be used to generate release notes
• Fully documented, including updating docs/source/*.rst for new API

[rouault]

I know it has been announced for years, but I'm still a bit uneasy with this removal in PROJ 7.0. Are there major cleanups & developments that will go in 7.0 and depend on proj_api being no longer in our way ? Bas mentioned a couple of blocker packages, like osmium, and I can indeed see in https://github.com/osmcode/libosmium/blob/master/include/osmium/geom/projection.hpp that they are still using proj_api.h exclusively (although it doesn't seem it would be a huge work to port it to proj.h). It will thus delay PROJ 7 deployment in a number of *nix distributions for months. The benefits vs drawbacks for us as PROJ developers doesn't seem obvious to me.

I can see that if we remove it now, then PROJ master can be considered as a 7.x, leading to 7.1, 7.2 etc incremental releases. If we remove it later, at the point where we'll remove it, PROJ master will be a 8.0dev

So all in all, my gut feeling is close to -0, but we should probably still proceed according to the plan, and packages still using proj_api.h will now have a obvious incentive to do migration to proj.h

[kbevers]

I know it has been announced for years, but I'm still a bit uneasy with this removal in PROJ 7.0

Yeah, I know.

Are there major cleanups & developments that will go in 7.0 and depend on proj_api being no longer in our way ?

Nothing major, I think. I am currently looking into removing the old fileapi functions and context stuff specific to proj_api.h and after that probably try to remove pj_transform(). So far it looks like fairly clean cuts. So the clean up for now just consists of removing code that is not necessary when proj_api.h is not a public header anymore.

It will thus delay PROJ 7 deployment in a number of *nix distributions for months. The benefits vs drawbacks for us as PROJ developers doesn't seem obvious to me.

This is why I have agreed to issue maintenance releases of the 6.3 branch for the next year. I don't expect PROJ 7 to be adopted right away by the big packaging systems. They do however mostly seem to be caught up with PROJ 6 which is a good bridge between old and new PROJ API's. I hope that be extended the life-time of the 6.3-branch we can ease the transition.

I can see that if we remove it now, then PROJ master can be considered as a 7.x, leading to 7.1, 7.2 etc incremental releases. If we remove it later, at the point where we'll remove it, PROJ master will be a 8.0dev

That's the thing, when would be the right time to put out 8.0? 7.0 will be the third major version release in three years. I would like to introduce some stability after 7.0 and hopefully wait a few years before introducing breaking changes again.

So all in all, my gut feeling is close to -0, but we should probably still proceed according to the plan, and packages still using proj_api.h will now have a obvious incentive to do migration to proj.h

I really don't want to end in a Python2 type of situation, so I'd rather we make life a bit more difficult now rather than pushing the problem ahead of us.

[rouault]

I really don't want to end in a Python2 type of situation, so I'd rather we make life a bit more difficult now rather than pushing the problem ahead of us.

Fair enough.

[hobu]

I'm also -0 leaning hard toward -1 and would prefer delaying until 8.0. I think we are one tick-tock too early. The issue isn't simply getting unreleased software updated to the new API. It is also filtration of those releases into packaging systems, the interlocking of those packages with ones that depend upon them, and any bug hashing activity that might result from the migration.

We don't owe the old API users significant fixes, backports, or architectural refactorings that would take a lot of effort. I think we have done enough of that with fair warning. We also shouldn't pull the rug out from under them just for what we perceive as their own good. My proposal for 7.0 would be something like this:

• Leave proj_api.h in with a deprecation warning for 7.0+
• Do not backport any bug fixes or changes to it, if something breaks significantly due to the rest of development moving forward, so be it.
• Kill it at 8.0

We have made a lot of changes to PROJ in the past four years, and people are just now starting to catch up to what has happened, what it means for their software, how things have improved, and how things might have gotten more complicated. PROJ was never software that was in a hurry, and users interacted with it under that assumption. It seems like we are now (even though it has been a four year process!) in relation to the 40 year history of the project.

If the audience making noise in #1552 steps forward with the resources, time, or code to provide a DB indirection, it is possible there will be more API churn. In short, the house remodel is almost done, but there's still some painting and trim to do in a few places.

What development does the old API prevent from taking place? Must it be removed before any of that can happen? The case for it needs to be stronger, IMO.

[kbevers]

Kill it at 8.0

Your plan fails to mention a time of death.

In short, the house remodel is almost done, but there's still some painting and trim to do in a few places.

My experience with home improvement is that if you don't get the finishing done right away you'll still be looking at those unpainted panels ten years down the line. So following your analogy we will never get rid of proj_api.h :-)

What development does the old API prevent from taking place? Must it be removed before any of that can happen? The case for it needs to be stronger, IMO.

I don't think that it stands in the way of anything. Anything is possible, but leaving it in does add extra friction when cleaning up the internals of the code base. Error-handling is one example of that which is made more complicated by pj_transform logic that we would have to find a solution for.

But since you guys aren't really on board with this I can't merge this PR. I am not happy about that but that is the way it goes.

[hobu]

Your plan fails to mention a time of death.

Do you mean to ask when 8.0 would be released? I had assumed another year, March 2021.

We've moved fast, especially as measured by the geodetic drift pace that PROJ had previously kept. We had a few earthquakes, and now our users deformation models need to catch up 😄

[kbevers]

Do you mean to ask when 8.0 would be released? I had assumed another year, March 2021.

Yes, that's what I meant. What will be different in a year from now? I fear we will end up having the same discussion and then ultimately post-poning to 2022.

We've said for two years now that proj_api.h would be removed with this upcoming release. I like to keep my promises. I don't want to announce a new time of removal if that can't be kept.

[hobu]

What will be different in a year from now?

All of the softwares that have recently migrated to proj.h (MapServer, PostGIS, spatialite, etc) will have had one more year and probably one or two release cycles to matriculate any repercussions. Any softwares stuck in an LTS cycle will have had another year to make the step forward. It helps eliminates the chance that any public effort to fork the old code into a new project for convenience.

I am enthusiastically +1 to kill it next year at this time.

[kbevers]

My impressions is that the main problems for packagers is the less popular packages that don't see as much work as the ones you mention. But of course it will be a big help for the packagers get more time to transition.

I am enthusiastically +1 to kill it next year at this time.

I'm glad to hear that. Unfortunately my enthusiasm for 7.0 has dropped significantly.

[hobu]

Unfortunately my enthusiasm for 7.0 has dropped significantly.

What if we simply delay 7.0 for 9 months and release it right around FOSS4G? Then you can keep your 7.0 promise and we can let the laggards catch up and the packaging situation mature?

[kbevers]

What if we simply delay 7.0 for 9 months and release it right around FOSS4G? Then you can keep your 7.0 promise and we can let the laggards catch up and the packaging situation mature?

That would just be breaking another promise (release date of 7.0). No, we need to get RFC4 functionality out in the wild. We can't sit on that for 9 months. That's one thing I'm still enthusiastic about :-)

[sebastic]

While it's been decided already, I still think it's fine to remove proj_api.h in 7.0 as initially planned and communicated.

I second this:

What will be different in a year from now? I fear we will end up having the same discussion and then ultimately post-poning to 2022.

As I've mentioned on the list before, when there is no release in which proj_api.h has actually been removed some project lack the incentive to adopt proj.h.

I have no problem shipping the next Debian stable release with 6.x, and using the following development cycle to adopt 7.x even if that means removing SpatiaLite.

[hobu]

@sebastic I generally agree about ripping the band aid off quickly, but the API migration isn't simply switching a header with a different name. The internals of how things work are now little bit different too (see today's thread for an example https://lists.osgeo.org/pipermail/gdal-dev/2020-January/051500.html ). A little more time to matriculate isn't such a big cost.

Its removal at 8.0 is clearly enough deference to any long-time PROJ users who haven't been paying attention recently. 7.x => 8.x isn't going to be nearly the leap that 6.x => 7.x was either.