New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

library(sf) loads wrong version of GEOS #844

Closed
dbaston opened this Issue Sep 11, 2018 · 21 comments

Comments

Projects
None yet
4 participants
@dbaston
Copy link
Contributor

dbaston commented Sep 11, 2018

I get the following error when building sf master against GEOS 3.7.0:

 unable to load shared object '/home/dan/R/x86_64-pc-linux-gnu-library/3.4/sf/libs/sf.so':
  /home/dan/R/x86_64-pc-linux-gnu-library/3.4/sf/libs/sf.so: undefined symbol: GEOSFrechetDistanceDensify_r

I verified that the symbol is present in libgeos_c.so:

nm -g /usr/local/lib/libgeos_c.so | grep Frechet
0000000000010210 T GEOSFrechetDistance
0000000000010230 T GEOSFrechetDistanceDensify
00000000000141e0 T GEOSFrechetDistanceDensify_r
0000000000014040 T GEOSFrechetDistance_r

The following command is used to link sf:

g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o sf.so RcppExports.o bbox.o gdal.o gdal_geom.o gdal_read.o gdal_utils.o gdal_write.o geos.o hex.o proj.o raster2sf.o sfg.o signed_area.o stars.o wkb.o -lproj -L/usr/local/lib -lgdal -L/usr/local/lib -lgeos_c -L/usr/lib/R/lib -lR

which looks right to me.

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 11, 2018

I've seen the same thing when testing against 3.7.0rc2, after Brian Ripley warned @rsbivand (rgeos) and me about it a few days ago. Will try again with 3.7.0. Big mystery!

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 11, 2018

I reproduced it again (in a container) with GEOS 3.7.0, and got the same error. In earlier attempts I tried to remove GEOSFrechetDistanceDensify_r, but then it would break on the next symbol, and so on; as you showed, the symbols are there. Did they do something with GEOS?

@dbaston

This comment has been minimized.

Copy link
Contributor

dbaston commented Sep 11, 2018

@edzer I'm able to build and run a dummy program using GEOSFrechetDistanceDensify_r:

#include <geos_c.h>
#include <iostream>

int main(int argc, char** argv) {
	auto handle = initGEOS_r(nullptr, nullptr);

	GEOSGeometry* geom1_ = GEOSGeomFromWKT_r(handle, "LINESTRING (0 0, 100 0)");
	GEOSGeometry* geom2_ = GEOSGeomFromWKT_r(handle, "LINESTRING (0 0, 50 50, 100 0)");

	double dist;
	int ret = GEOSFrechetDistanceDensify_r(handle, geom1_, geom2_, 0.5, &dist);

	std::cout << dist << std::endl;
}

with

g++ frechet.cpp -std=c++11 -o frechet -L /usr/local/lib -lgeos_c

@dbaston

This comment has been minimized.

Copy link
Contributor

dbaston commented Sep 12, 2018

Running ldd sf.so also shows correct output:

libgeos_c.so.1 => /usr/local/lib/libgeos_c.so.1 (0x00007f4621c23000)

Yet, running the following:

LD_DEBUG=all LD_DEBUG_OUTPUT=/tmp/debug Rscript -e "dyn.load('/home/dan/dev/sf/src/sf.so')"
then

grep Frechet /tmp/debug*
shows that the wrong library is being loaded at runtime

/tmp/debug.28543: 28543: symbol=GEOSFrechetDistanceDensify_r; lookup in file=/usr/lib/x86_64-linux-gnu/libgeos_c.so.1

edzer added a commit that referenced this issue Sep 12, 2018

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 12, 2018

The link step of rgeos is

g++ -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -Wl,-z,relro -o rgeos.so dummy.o init.o local_stubs.o rgeos.o rgeos_R2geos.o rgeos_R2geosMP.o rgeos_bbox.o rgeos_buffer.o rgeos_coord.o rgeos_geos2R.o rgeos_linearref.o rgeos_misc.o rgeos_poly2nb.o rgeos_predicate_binary.o rgeos_predicate_unary.o rgeos_topology.o rgeos_topology_binary.o rgeos_validate.o rgeos_wkt.o -L/usr/local/lib -lgeos-3.7.0 -L/usr/local/lib -lgeos_c -L/usr/lib/R/lib -lR

which includes -lgeos-3.7.0, not in sf.

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 12, 2018

adding this didn't solve the problem.

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 12, 2018

After moving /usr/lib/x86_64-linux-gnu/libgeos_c.so.1.9.1 and /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 out of the way, sf installed fine.

@rsbivand

This comment has been minimized.

Copy link
Member

rsbivand commented Sep 12, 2018

Yes, multiple libgeos will probably cause havoc. I haven't seen any rgeos issues on Fedora, but always delete earlier /usr/local/lib/libgeos* etc., and never let any packager install GDAL, PROJ or GEOS. The warning from Brian Ripley may relate to static builds (libgeos.a), but configure.ac presupposes dynamic (libgeos-3.7.0):

[rsb@localhost geos-3.7.0]$ geos-config --cclibs
-L/usr/local/lib -lgeos
[rsb@localhost geos-3.7.0]$ geos-config --clibs
-L/usr/local/lib -lgeos_c
[rsb@localhost geos-3.7.0]$ geos-config --libs
-L/usr/local/lib -lgeos-3.7.0
[rsb@localhost geos-3.7.0]$ geos-config --static-clibs
-L/usr/local/lib -lgeos_c -lgeos -lm
[rsb@localhost geos-3.7.0]$ geos-config --static-cclibs
-L/usr/local/lib -lgeos -lm

and configure.ac has:

GEOS_LIBS=`${GEOS_CONFIG} --libs`
GEOS_CLIBS=`${GEOS_CONFIG} --clibs`

Please advise on any need to change configure.ac.

@rsbivand

This comment has been minimized.

Copy link
Member

rsbivand commented Sep 12, 2018

rgeos configure.ac committed with:

GEOS_LIBS=`${GEOS_CONFIG} --cclibs`
GEOS_CLIBS=`${GEOS_CONFIG} --clibs`
@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 12, 2018

configure controls compilation, but the error occurs at runtime. I can't think of a way in which configure could change this.

@rsbivand

This comment has been minimized.

Copy link
Member

rsbivand commented Sep 12, 2018

But it removes the version suffix from the rgeos link step. Beyond that, the loader will use the first libgeom it finds. I'm not seeing any issues on Fedora, feels like a platform issue. Time someone made a Fedora rocker for improved stability?

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 13, 2018

I think the issue Can't build against GEOS 3.7.0 is not true; the issue Can't build against GEOS 3.7.0 if there is also another GEOS installation on my system is, depending on where it is. Some diagnostics:

When hacking geos.cpp such that it doesn't use symbols provided only by GEOS 3.7.0 or 3.6.1, it compiles AND loads on R when GEOS 3.7.0 is in /usr/local/lib and GEOS 3.5.1 (or so) is in /usr/lib/something. Then:

root@313b7b08657b:/# ldd /usr/local/lib/R/site-library/sf/libs/sf.so  | grep geos
	libgeos_c.so.1 => /usr/local/lib/libgeos_c.so.1 (0x00007fb39962a000)
	libgeos-3.7.0.so => /usr/local/lib/libgeos-3.7.0.so (0x00007fb3964d2000)
root@313b7b08657b:/# echo $LD_LIBRARY_PATH
/usr/local/lib:

but after starting R:

> system("ldd /usr/local/lib/R/site-library/sf/libs/sf.so | grep geos")
	libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007fdad7954000)
	libgeos-3.5.1.so => /usr/lib/x86_64-linux-gnu/libgeos-3.5.1.so (0x00007fdad481d000)
> Sys.getenv("LD_LIBRARY_PATH")
[1] "/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/jre/lib/amd64/server:/usr/local/lib:"

A hacky solution to this might be

> Sys.setenv(LD_LIBRARY_PATH="/usr/local/lib:/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/jre/lib/amd64/server:")
> system("ldd /usr/local/lib/R/site-library/sf/libs/sf.so | grep geos")
	libgeos_c.so.1 => /usr/local/lib/libgeos_c.so.1 (0x00007f4909bc1000)
	libgeos-3.7.0.so => /usr/local/lib/libgeos-3.7.0.so (0x00007f4906a69000)

but I'm not sure if I would go as far as recommending this to anyone.

It would actually be good (sane, secure) to do a run-time check that the GEOS version linked to corresponds to the one found at compile time.

@rsbivand

This comment has been minimized.

Copy link
Member

rsbivand commented Sep 13, 2018

In rgdal this is done on attach (maybe should be done on load?) using GDALCheckVersion(); does GEOS provide a similar function?

edzer added a commit that referenced this issue Sep 13, 2018

edzer added a commit that referenced this issue Sep 13, 2018

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 13, 2018

Gives me:

> library(sf)
Linking to GEOS 3.7.0, GDAL 2.4.0dev-358fa65, proj.4 5.1.0
WARNING: different compile-time and runtime versions for GEOS

I'll try to improve this: the linking to GEOS info is wrong.

edzer added a commit that referenced this issue Sep 13, 2018

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 13, 2018

Gives:

> library(sf)
Linking to GEOS 3.5.1, GDAL 2.4.0dev-358fa65, PROJ 5.1.0
WARNING: different compile-time and runtime versions for GEOS found:
Linked against: 3.5.1-CAPI-1.9.1 r4246 compiled against: 3.7.0-CAPI-1.11.0
@rsbivand

This comment has been minimized.

Copy link
Member

rsbivand commented Sep 13, 2018

I've committed to rgeos as r576, but found that my loader itself stopped the package loading when the versions differed:

> library(rgeos)
Error: package or namespace load failed for ‘rgeos’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/rsb/lib/r_libs/rgeos/libs/rgeos.so':
  libgeos-3.7.0.so: cannot open shared object file: No such file or directory

but had replaced 3.7.0 with 3.6.3. In your case, I assume that both were present but libgeos was linked to another version?

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 13, 2018

Yes, it is more likely to get a load error than a version mismatch warning, but nevertheless useful to check IMO. In my case both are present and in LD_LIBRARY_PATH. R changed LD_LIBRARY_PATH by putting stuff in front of the user-set value, and that caused the havoc.

@dbaston

This comment has been minimized.

Copy link
Contributor

dbaston commented Sep 13, 2018

@edzer Yes, it seems it's really an issue with R itself, not sf or GEOS.

@dbaston dbaston changed the title Can't build against GEOS 3.7.0 library(sf) loads wrong version of GEOS Sep 13, 2018

@dbaston

This comment has been minimized.

Copy link
Contributor

dbaston commented Sep 13, 2018

Everything works ok if I explicitly load GEOS before sf, with:

dyn.load('/usr/local/lib/libgeos_c.so', local=FALSE)
library(sf)

But putting a dyn.load into the .onLoad function seems to be "too late."

@edzer

This comment has been minimized.

Copy link
Member

edzer commented Sep 13, 2018

Thanks! Now that we know all this, I'm thinking about writing a short wiki entry, and make a link to it from the README.md file.

@mccarthyryanc

This comment has been minimized.

Copy link

mccarthyryanc commented Nov 20, 2018

@edzer I'm running into similar issues, any progress on the wiki entry to help describe this issue or how to avoid it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment