Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

strange segfault when using pyproj #116

Open
jmarca opened this issue Oct 4, 2019 · 7 comments

Comments

@jmarca
Copy link

@jmarca jmarca commented Oct 4, 2019

Not sure what I'm doing wrong here, but it seems importing both pyproj and pyosmium leads to a segfault.

Environment:

python --version
Python 3.7.4
 pip list
Package            Version    
------------------ -----------
...
osmium             2.15.3     
...
pyproj             2.4.0      

Code that segfaults

import osmium
import pyproj

crs = pyproj.CRS.from_epsg(4326)
print(crs)
python pyproj_test.py
Segmentation fault

But if I switch the order of imports, no problem.

import pyproj
import osmium

crs = pyproj.CRS.from_epsg(4326)
print(crs)

Obviously, the simple answer is to import osmium last! But segfaults are bad, and I can't figure out why the order might matter.

@lonvia

This comment has been minimized.

Copy link
Member

@lonvia lonvia commented Oct 4, 2019

What system are you running on? Both pyosmium and pyproj use the libproj library and I suspect we are running into version issues here.

@jmarca

This comment has been minimized.

Copy link
Author

@jmarca jmarca commented Oct 4, 2019

@jmarca jmarca changed the title strangle segfault when using pyproj strange segfault when using pyproj Oct 6, 2019
@lonvia

This comment has been minimized.

Copy link
Member

@lonvia lonvia commented Oct 9, 2019

It might help if you use the source wheel of osmium instead of the binary wheel.

@jmarca

This comment has been minimized.

Copy link
Author

@jmarca jmarca commented Oct 9, 2019

I'm sorry, I'm not familiar with what you mean by "source" versus "binary wheel". Just educated my self a little bit:

my Dockerfile currently just says:

pip install --upgrade --no-cache-dir osmium sqlalchemy geoalchemy2 pyproj shapely

Should I change that to

pip install --upgrade --no-cache-dir https://github.com/osmcode/pyosmium/archive/v2.15.3.tar.gz
&& ...

etc for those libraries that link to proj?

I'll try it and see.

@jmarca

This comment has been minimized.

Copy link
Author

@jmarca jmarca commented Oct 9, 2019

Yep, looks like library versions are the problem. Trying to build pyproj from source, it complains that it needs proj v6.2.0, but only 5.2.0 is installed (python 3.7 image is based on buster). So my guess is the binary pip install of pyproj included proj 6.2 or something. I'll close this when I get everything compiled with same versions of proj and have verified no crashing.

@lonvia

This comment has been minimized.

Copy link
Member

@lonvia lonvia commented Oct 10, 2019

Thanks for testing. The Proj library had a major breaking release with version 6. So I'm not surprised that causes issues. pyosmium >= 2.15.3 can compile against Proj >= 6.0 (see osmcode/libosmium#277) but the binary wheels are built against proj 5. So I suspect it will pollute your python with an old proj when you load osmium before pyproj.

I need to check where we include proj and if we can do without it. Please leave the issue open. I'd consider using source wheels really only as a temporary workaround.

NB: the source wheels are also available via pip installation. You can force to use the source over the binary wheel with the option --no-binary. I.e. to install pyproj and osmium from source, run something like pip install --no-binary :all: pyproj osmium. You still need the proper dependencies, just like when using the archive from github.

@jmarca

This comment has been minimized.

Copy link
Author

@jmarca jmarca commented Oct 14, 2019

Okay, I fixed my Dockerfile to do the following:

Download proj 6.2.0 from source, build it;
pip install from source for both osmium and pyproj

Bug goes away...I can include libraries in whatever order.

Relevant bits of Dockerfile (ultimately based on canonical python 3.7 image)

ENV PROJ_VERSION 6.2.0
RUN apt-get install -y sqlite3 \
        && wget -O proj.tgz "https://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz" \
        && tar -xvf proj.tgz \
        && wget "https://download.osgeo.org/proj/proj-datumgrid-1.8.zip" \
        && unzip -o proj-datumgrid-1.8.zip -d proj-${PROJ_VERSION}/data/ \
        && cd proj-${PROJ_VERSION} \
        && mkdir build \
        && cd build \
        && cmake -DCMAKE_INSTALL_PREFIX=/usr .. \
        && make -j6 \
        && make install \
        && cd ../../ && rm -rf proj*

RUN pip install --upgrade --no-cache-dir --no-binary :all: pyproj osmium

# test the install
USER user

RUN python -c "import osmium as o; import pyproj as pp; crs = pp.CRS.from_epsg(4326); print (crs)"
RUN python -c "import pyproj as pp; import osmium as o; crs = pp.CRS.from_epsg(4326); print (crs)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.