strange segfault when using pyproj

[jmarca]

Not sure what I'm doing wrong here, but it seems importing both pyproj and pyosmium leads to a segfault.

Environment:

python --version
Python 3.7.4
 pip list
Package            Version    
------------------ -----------
...
osmium             2.15.3     
...
pyproj             2.4.0      

Code that segfaults

import osmium
import pyproj

crs = pyproj.CRS.from_epsg(4326)
print(crs)

python pyproj_test.py
Segmentation fault

But if I switch the order of imports, no problem.

import pyproj
import osmium

crs = pyproj.CRS.from_epsg(4326)
print(crs)

Obviously, the simple answer is to import osmium last! But segfaults are bad, and I can't figure out why the order might matter.

[lonvia]

What system are you running on? Both pyosmium and pyproj use the libproj library and I suspect we are running into version issues here.

[jmarca]

I am using a Docker image that is based off the python 3.7 image. So whatever Ubuntu is inside of that. I can compile libproj et al from source. Would that be better? All I really want to do is transform geometries. Is there a call inside osmium to do that? James

…

On Oct 4, 2019, at 13:41, Sarah Hoffmann ***@***.***> wrote: What system are you running on? Both pyosmium and pyproj use the libproj library and I suspect we are running into version issues here. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[lonvia]

It might help if you use the source wheel of osmium instead of the binary wheel.

[jmarca]

I'm sorry, I'm not familiar with what you mean by "source" versus "binary wheel". Just educated my self a little bit:

my Dockerfile currently just says:

pip install --upgrade --no-cache-dir osmium sqlalchemy geoalchemy2 pyproj shapely

Should I change that to

pip install --upgrade --no-cache-dir https://github.com/osmcode/pyosmium/archive/v2.15.3.tar.gz
&& ...

etc for those libraries that link to proj?

I'll try it and see.

[jmarca]

Yep, looks like library versions are the problem. Trying to build pyproj from source, it complains that it needs proj v6.2.0, but only 5.2.0 is installed (python 3.7 image is based on buster). So my guess is the binary pip install of pyproj included proj 6.2 or something. I'll close this when I get everything compiled with same versions of proj and have verified no crashing.

[lonvia]

Thanks for testing. The Proj library had a major breaking release with version 6. So I'm not surprised that causes issues. pyosmium >= 2.15.3 can compile against Proj >= 6.0 (see osmcode/libosmium#277) but the binary wheels are built against proj 5. So I suspect it will pollute your python with an old proj when you load osmium before pyproj.

I need to check where we include proj and if we can do without it. Please leave the issue open. I'd consider using source wheels really only as a temporary workaround.

NB: the source wheels are also available via pip installation. You can force to use the source over the binary wheel with the option --no-binary. I.e. to install pyproj and osmium from source, run something like pip install --no-binary :all: pyproj osmium. You still need the proper dependencies, just like when using the archive from github.

[jmarca]

Okay, I fixed my Dockerfile to do the following:

Download proj 6.2.0 from source, build it;
pip install from source for both osmium and pyproj

Bug goes away...I can include libraries in whatever order.

Relevant bits of Dockerfile (ultimately based on canonical python 3.7 image)

ENV PROJ_VERSION 6.2.0
RUN apt-get install -y sqlite3 \
        && wget -O proj.tgz "https://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz" \
        && tar -xvf proj.tgz \
        && wget "https://download.osgeo.org/proj/proj-datumgrid-1.8.zip" \
        && unzip -o proj-datumgrid-1.8.zip -d proj-${PROJ_VERSION}/data/ \
        && cd proj-${PROJ_VERSION} \
        && mkdir build \
        && cd build \
        && cmake -DCMAKE_INSTALL_PREFIX=/usr .. \
        && make -j6 \
        && make install \
        && cd ../../ && rm -rf proj*

RUN pip install --upgrade --no-cache-dir --no-binary :all: pyproj osmium

# test the install
USER user

RUN python -c "import osmium as o; import pyproj as pp; crs = pp.CRS.from_epsg(4326); print (crs)"
RUN python -c "import pyproj as pp; import osmium as o; crs = pp.CRS.from_epsg(4326); print (crs)"