Seg fault with python 2.7 rasterio 1.1.0 wheel

[glostis]

Hi,

I have encountered failing Travis builds on one of my projects following the latest rasterio release (1.1.0).
My Travis build tests both python 3.7 and 2.7, and only the build on 2.7 started failing.
It was an end-to-end test, but I have managed to narrow it down to a very minimal example, shown below.

TL;DR: importing rasterio in a script that executes pyproj crashes with a Segmentation fault with the rasterio-1.1.0-cp27-cp27mu-manylinux1_x86_64.whl wheel.

Steps to reproduce the problem.

I have included a .zip with the Dockerfile and the two python scripts needed to reproduce the problem.

rio_bug.zip

For easier readability, here is the Dockerfile and the two python scripts:

expand Dockerfile

FROM ubuntu:latest

RUN apt-get update && apt-get install -y python-pip python3-pip

WORKDIR home

RUN for py_ver in 2 3; do \
        pip${py_ver} install virtualenv; \
        \
        for rio_ver in 1.0.28 1.1.0; do \
            venv_name=py_${py_ver}_rio_${rio_ver}; \
            \
            # Create virtual environment
            virtualenv -p python${py_ver} ${venv_name}; \
            \
            # Install packages
            ${venv_name}/bin/pip install "rasterio[s3]==${rio_ver}" pyproj; \
            \
            # Add lines to test script
            for with_out in with_rasterio without_rasterio; do \
                echo "echo ${venv_name} ${with_out}" >> test.sh; \
                echo ${venv_name}/bin/python ${with_out}.py >> test.sh; \
            done; \
        done; \
    done

WORKDIR /

COPY with_rasterio.py without_rasterio.py home/

WORKDIR home

CMD ["bash", "test.sh"]

expand with_rasterio.py

import pyproj
import rasterio


pyproj.Proj(init="epsg:4326")
print("ok")

expand without_rasterio.py

import pyproj


pyproj.Proj(init="epsg:4326")
print("ok")

The Dockerfile basically runs the two python scripts with a matrix of python versions (2/3) and rasterio versions (1.0.28/1.1.0).

Building and running this Dockerfile gives the following output:

py_2_rio_1.0.28 with_rasterio
ok
py_2_rio_1.0.28 without_rasterio
ok
py_2_rio_1.1.0 with_rasterio
test.sh: line 6:    13 Segmentation fault      py_2_rio_1.1.0/bin/python with_rasterio.py
py_2_rio_1.1.0 without_rasterio
ok
py_3_rio_1.0.28 with_rasterio
ok
py_3_rio_1.0.28 without_rasterio
ok
py_3_rio_1.1.0 with_rasterio
ok
py_3_rio_1.1.0 without_rasterio
ok

As you can see, there is one failing case among the 8 configurations: python2.7 with rasterio 1.1.0.

I didn't see anything obvious in the changelog from 1.0.28 to 1.1.0 that could explain this error, especially due to the fact that it only occurs in python2.7.

[vincentsarago]

@glostis what happens if you import rasterio after pyproj ?

[glostis]

@vincentsarago What exactly do you mean by importing after?

The with_rasterio.py script already imports rasterio after pyproj:

import pyproj
import rasterio


pyproj.Proj(init="epsg:4326")
print("ok")

I have also tried it the other way around:

import rasterio
import pyproj


pyproj.Proj(init="epsg:4326")
print("ok")

and it produces the same result.

[vincentsarago]

oh yeah sorry I meant the opposite 🤦‍♂

and it produces the same result.

just to be sure, when rasterio is imported before pyproj you get the same Segfault ?

I guess the problem is that pyproj built from wheel will use GDAL3 while rasterio wheel is shipped with GDAL 2.4

when working with shapely/pyproj/rasterio together I alway try to avoid using wheels to avoid the segfault, I think there is an old issue in rasterio or fiona repo about that

[glostis]

just to be sure, when rasterio is imported before pyproj you get the same Segfault ?

Yes it produces the same Segfault.

I guess the problem is that pyproj built from wheel will use GDAL3 while rasterio wheel is shipped with GDAL 2.4

I see. However the thing that surprises me is that my experiment works with the rasterio 1.0.28 wheel, which is quite similar to the rasterio 1.1.0 wheel in that regard if I am not mistaken.

when working with shapely/pyproj/rasterio together I alway try to avoid using wheels to avoid the segfault, I think there is an old issue in rasterio or fiona repo about that

Yes I guess installing the sdists is safer. But I find wheels so convenient (fast to install, and they come with all of their dependencies)...

[sgillies]

@vincentsarago yes, I think your diagnosis is right, except that I'm sure that the library causing us trouble is proj, not gdal.

The difference between Python 2.7 and 3.6 is curious. In rasterio 1.1b1 we had a dataset reference counting bug that only caused segfaults for Python 2.7. A module compiled with the 3.6 Python SDK is more fault tolerant for reasons I don't understand.

[glostis]

Thanks for looking into this @sgillies and @vincentsarago!

Do you think this issue could be solved by changing something in the lastest rasterio wheels, or is the only way to go what @vincentsarago described, by not working with wheels to avoid the problem?
I still don't understand why this behavior "appeared" with rasterio==1.1.0. Did the version of proj in that wheel change?

How could I go about debugging this issue?

[sgillies]

@glostis the rasterio wheels aren't built any differently for 1.1.0. PROJ and GDAL are built the same way. Rasterio 1.1.0 did add a new _transform module but I don't see anything out of the ordinary there. Did your pyproj version change between your builds?

I'm pretty much finished with Python 2.7 and will be ignoring problems that only crop up there starting in 2020. Rasterio is going to stop testing against Python 2.7 soon. That's not to say that I don't want to know why this is happening.

[snowman2]

What versions of pyproj for each scenario?

[glostis]

Sorry for the late response.

As you can see in my Dockerfile, I was testing the bug on 4 different python virtualenvs:

1. python 2 with rasterio 1.0.28
2. python 2 with rasterio 1.1.0
3. python 3 with rasterio 1.0.28
4. python 3 with rasterio 1.1.0

In each virtualenv, I was simply requiring the latest version of pyproj through pip. I hadn't considered that the pyproj version could be different, but I checked following your question, and realized that my assumption was false because pyproj dropped support for py2 after version 2.2.2. The consequence is that in envs 1 and 2, pyproj==2.2.2 was installed, whereas in envs 3 and 4, pyproj==2.4.0 was installed.

I have rerun my Dockerfile by forcing pyproj==2.2.2 in all envs, and it gave more logical results: the bug occurs in envs 2 and 4, and not in envs 1 and 3.

To sum up, this bug occurs on python environments with rasterio==1.1.0 and pyproj==2.2.2, regardless of the python version.

I hope this clarifies things a little.

[snowman2]

There were some fixes in the latest pyproj and PROJ in the wheels in (2.4.0) that prevents core dumping in specific scenarios. So, I bet it is related. Not sure why importing rasterio would impact it though. Interesting 🤔

[sgillies]

@snowman2 can you link to the changes? I'd like to take a look. After that, I'm going to close this issue. The solutions are outside the scope of the rasterio project.

[snowman2]

I think this is definitely related to the fix in pyproj/PROJ. Does importing rasterio set the PROJ_LIB environment variable? If so, then that is what causes the problem.

More details in related issue: OSGeo/PROJ#1574

[sgillies]

@snowman2 yes, when we import rasterio it peeks in the process environment and if PROJ_LIB is not set, and the rasterio package contains PROJ data (from a wheel), it patches the environment https://github.com/mapbox/rasterio/blob/059ad3e01c0da363c99cd1f2ca6c4e3353c1485c/rasterio/env.py#L648-L654. The behavior is the same in 1.0.28 and 1.1.0.

[snowman2]

The behavior is the same in 1.0.28 and 1.1.0

That throws a wrench in the PROJ only theory. Any difference in the PROJ/GDAL versions in the rasterio wheels?

[sgillies]

No. It's PROJ 4.9.3 and GDAL 2.4.2 in each.

[snowman2]

It is definitely related to this issue OSGeo/PROJ#1574. I created and environment with rasterio==1.1.0 and pyproj==2.2.0. Then, using gdb it generates the same traceback in the issue:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff630e63d in ?? ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
(gdb) bt
#0  0x00007ffff630e63d in ?? ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
#1  0x00007ffff63107b0 in osgeo::proj::io::createFromUserInput(std::string const&, projCtx_t*) ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
#2  0x00007ffff637a965 in proj_create ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
#3  0x00007ffff63efd40 in ?? ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
#4  0x00007ffff63f010a in ?? ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
#5  0x00007ffff63f017d in ?? ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
#6  0x00007ffff6322b51 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::string const&) ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
---Type <return> to continue, or q <return> to quit---
#7  0x00007ffff630ea45 in ?? ()
   from /home/snowal/miniconda/envs/rioproj/lib/python3.7/site-packages/pyproj/.libs/libproj-1b1cfd8e.so.15.1.0
#8  0x00007ffff63107b0 in osgeo::proj::io::createFromUserInput(std::string const&, projCtx_t*) ()
...

I then tested out unsetting PROJ_LIB and no segfault:

>>> import rasterio
>>> import pyproj
>>> import os
>>> proj_lib = os.environ.pop("PROJ_LIB")
>>> pyproj.Proj(init="epsg:4326")
Proj('+proj=longlat +datum=WGS84 +no_defs', preserve_units=True)
>>> os.environ["PROJ_LIB"] = proj_lib
>>> pyproj.Proj(init="epsg:4326")
Segmentation fault (core dumped)

[snowman2]

However, with rasterio==1.0.28:

>>> import rasterio, os
>>> os.environ["PROJ_LIB"]
Traceback (most recent call last):
 ...
    raise KeyError(key) from None
KeyError: 'PROJ_LIB'

[glostis]

@snowman2 yes, when we import rasterio it peeks in the process environment and if PROJ_LIB is not set, and the rasterio package contains PROJ data (from a wheel), it patches the environment

https://github.com/mapbox/rasterio/blob/059ad3e01c0da363c99cd1f2ca6c4e3353c1485c/rasterio/env.py#L648-L654

. The behavior is the same in 1.0.28 and 1.1.0.

@sgillies if I am not mistaken, the lines you refer to, which set the PROJ_LIB env var if PROJ data is found, were introduced in version 1.1b1 (059ad3e#diff-bcec099fb5186938da74b5bfc04c1476).

@snowman2 has shown above that it is the presence of this env var that is clashing with pyproj.
This would explain why the bug occurs on rasterio version 1.1.0 and not on version 1.0.28.

[sgillies]

@glostis no, 1.0.28 did set PROJ_LIB, but as I look more closely I remember that we removed these two lines in 1.1: https://github.com/mapbox/rasterio/blob/1.0.28/rasterio/env.py#L644-L646. The reason: PROJ data at a system path might be for a different version of PROJ than the one needed by rasterio in a wheel.

[snowman2]

Well, this is a PROJ/pyproj issue that has been resolved in pyproj>=2.3.1 and rasterio shouldn't need to do anything as it is working as expected in rasterio==1.1.0. In summary, need to upgrade to Python 3.