Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build: require SQLite 3.11 (refs #1718) #1721

Merged
merged 2 commits into from Nov 9, 2019
Merged

Conversation

@rouault
Copy link
Member

rouault commented Nov 7, 2019

PROJ can build and run against older version (3.7 for example), but it has
been found that performance is horrible. With 3.11 (Ubuntu 16.04), it is
fine, so sets this as the minimum version.

PROJ can build and run against older version (3.7 for example), but it has
been found that performance is horrible. With 3.11 (Ubuntu 16.04), it is
fine, so sets this as the minimum version.
@kbevers

This comment has been minimized.

Copy link
Member

kbevers commented Nov 7, 2019

Thanks. Just a reminder from RFC3:

A change in programming language standard can only be introduced with a new major version release of PROJ. Changes for software package dependencies can be introduced with minor version releases of PROJ. Changing the version requirements for a dependency needs to be approved by the PSC.

So, we can introduce this with PROJ 6.3 provided that the PSC accepts this change. SQLite 3.11 was released in 2016 so we are clear of the older-than-two-years requirement for package dependencies.

I would suggest that RFC3 be amended with the new requirements of SQLite.

@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 7, 2019

RFC3 be amended with the new requirements of SQLite.

As I read it, RFC3 is general policy and bootstrapping. Does it need to be a living document ? Probably not the best place to capture requirements that might be per PROJ version

@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 7, 2019

Motion sent to the list

@kbevers

This comment has been minimized.

Copy link
Member

kbevers commented Nov 7, 2019

Does it need to be a living document ?

It doesn't have to but I figure that it might be useful. Maybe it's too much admin work to keep up to date, I dont know... I was just thinking a history section that keeps track of when new requirements are introduced.

Where do we keep the list of requirements now, that isn't RFC3? The README? Somewhere on the website? I think this is a place where we can improve things a bit at least.

@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 7, 2019

Where do we keep the list of requirements now

In the checks of build systems :-)

@kbevers

This comment has been minimized.

Copy link
Member

kbevers commented Nov 7, 2019

In the checks of build systems :-)

I'll go out on a limb here and say that that isn't good enough :)

@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 7, 2019

I'll go out on a limb here and say that that isn't good enough :)

it turns out I discovered there was already more or less something for that. I've tweaked it in b2873d7

docs/source/install.rst Outdated Show resolved Hide resolved
@rouault rouault force-pushed the rouault:require_sqlite_3_11 branch from b2873d7 to 589f210 Nov 7, 2019
@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 9, 2019

Motion passed

@rouault rouault merged commit 25c4f55 into OSGeo:master Nov 9, 2019
4 checks passed
4 checks passed
FreeBSD Task Summary
Details
Travis CI - Pull Request Build Passed
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
coverage/coveralls Coverage remained the same at 85.815%
Details
@devrimgunduz

This comment has been minimized.

Copy link

devrimgunduz commented Nov 14, 2019

Can you please give me a hint about compiling Proj with a non-standard sqlite version (I mean, which is outside $PATH, etc?)

I'm having trouble doing it. Proj successfully finds my new sqlite, but then I get:

libtool: link: g++ -std=c++11 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -Wdeprecated -Wzero-as-null-pointer-constant -g -O2 -fvisibility=hidden -Wl,-rpath -Wl,/usr/proj62/lib64 -o .libs/geod apps/geod.o apps/geod_set.o apps/geod_interface.o apps/emess.o  ./.libs/libproj.so -lpthread -lm -Wl,-rpath -Wl,/usr/proj62/lib
./.libs/libproj.so: undefined reference to `sqlite3_column_text'
./.libs/libproj.so: undefined reference to `sqlite3_close'

...(and this goes on a bit more)

Any ideas? Thanks!

@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 14, 2019

Any ideas? Thanks!

How do you invoke ./configure ? Do you use pkg-config or explicitly set SQLITE3_CFLAGS and SQLITE3_LIBS ?

@devrimgunduz

This comment has been minimized.

Copy link

devrimgunduz commented Nov 14, 2019

Argh, thanks! I was using them, but apparently in a wrong way. Here is what I did, which worked. %{sqlite33dir } is an RPM macro, and it's replaced by /usr/sqlite330 in my case:

export SQLITE3_LIBS="-L%{sqlite33dir}/lib -lsqlite3"
export SQLITE3_INCLUDE_DIR='%{sqlite33dir}/include'
export SQLITE3_CFLAGS="$CFLAGS -I%{sqlite33dir}/include"
export PATH=%{sqlite33dir}/bin/:$PATH

Thanks! Just a FYI: I'm working on custom sqlite3 packages on RHEL7, and Proj packages in the PostgreSQL RPM repository will depend of this sqlite version, which will solve the issues.

@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 14, 2019

Just a FYI: I'm working on custom sqlite3 packages on RHEL7, and Proj packages in the PostgreSQL RPM repository will depend of this sqlite version, which will solve the issues.

Yes, @hobu made me aware of that. Much appreciated ! I've posted about your work on the PROJ mailing list too in https://lists.osgeo.org/pipermail/proj/2019-November/009002.html

So, you're no replacing the system libsqlite3 ? How does that work with packages such as GDAL which themselves link to sqlite3 (do you build it againt that /usr/sqlite330 too ?). Otherwise that might potentially cause runtime clashes

@devrimgunduz

This comment has been minimized.

Copy link

devrimgunduz commented Nov 14, 2019

Hi @rouault ,

I'm not replacing the system sqlite. The package I created install things under /usr/sqlite330 directory. I rebuilt proj, and with rpath magic, made sure that libproj.so is linked to the new sqlite:
-bash-4.2$ ldd /usr/proj62/lib/libproj.so|grep sqlite
libsqlite3.so.0 => /usr/sqlite330/lib/libsqlite3.so.0 (0x00007f6c46200000)
I'll do the same for GDAL, libspatialite, libgeotiff, and then rebuild PostGIS against these.

Do you see any potential problems with this plan?

Thanks!

Regards, Devrim

@rouault

This comment has been minimized.

Copy link
Member Author

rouault commented Nov 14, 2019

Do you see any potential problems with this plan?

No, that sounds good. Downstream users of those packages having an explicit sqlite3 dependency should make sure to link against sqlite330 too.

Before you attempted this, I tried a quick & dirty approach by having an internal copy of sqlite inside PROJ, but I also tried to avoid any potential symbol clash by having a #define sqlite330_sqlite3_XXXX sqlite3_XXXX for each sqlite3 symbol. That way you can link both against PROJ and the system sqlite3 without risks
A solution to do that easily is from the amalgation sqlite3.c to

gcc -shared -fPIC sqlite3.c -shared -o temp.so
(for i in `objdump -T temp.so | grep sqlite3 | awk '{print $7}'`; do echo "#define $i sqlite330_$i"; done) > rename.h
cat rename.h sqlite3.h > sqlite3_with_rename.h
mv sqlite3_with_rename.h sqlite3.h

That way only the PROJ package would have to been built against this hacked sqlite3 lib, and all other packages could link against the regular sqlite3 without risk of conflict

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.