Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PROJ core dumps when setting data directory on specific context and it is not set on global context #1574

Closed
snowman2 opened this issue Aug 31, 2019 · 18 comments

Comments

@snowman2
Copy link
Contributor

@snowman2 snowman2 commented Aug 31, 2019

pyproj4/pyproj#419

This is a wierd edge case that has taken me some time to track down as the cause has been elusive for quite some time due to the core dumping. However, I believe the symptoms and the solution should help find a solution in PROJ I believe.

In the case where the data directory is not able to be determined by PROJ in the default locations on the global context, it will core dump when attempting to use proj_create with a perfectly valid custom context that has the proper data directory set. However, this does not happen in all instances. It only happens when using the +init= syntax.

For example:

>>> from pyproj import CRS
>>> cc = CRS("+proj=utm +zone=33 +ellps=WGS84 +datum=WGS84 +units=m +no_defs")
>>> cc = CRS(init="epsg:4326")
Segmentation fault (core dumped)

My guess is it has to do with:

PROJ/src/iso19111/c_api.cpp

Lines 437 to 440 in f059b90

// Only connect to proj.db if needed
if (strstr(text, "proj=") == nullptr || strstr(text, "init=") != nullptr) {
getDBcontextNoException(ctx, __FUNCTION__);
}

The fix I applied in pyproj was to set the data directory for the global PROJ context along with the custom context.

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Aug 31, 2019

The tricky part is to have the data directory in a place where PROJ cannot find it and then use proj_context_set_search_paths to set the data directory on a different context.

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Aug 31, 2019

Note: This issue began in PROJ 6.1 and still persists in 6.2

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Aug 31, 2019

This is also problematic for pyproj wheels when the PROJ_LIB is set to an old PROJ data directory. The global context finds the old data directory without the proj.db and proceeds to core dump even though the custom context has the data directory paths set properly..

@rouault

This comment has been minimized.

Copy link
Member

@rouault rouault commented Sep 4, 2019

I tried to reproduce, but couldn't manage. A bit confused by your explanations. Could you give an exact way to reproduce (possibly with a standalone C file) or perhaps display the stack trace where it crashes ?

gdb python
[ set args myscript.py ]
run
[ python code and wait for it to crash ]
bt
@kbevers

This comment has been minimized.

Copy link
Member

@kbevers kbevers commented Sep 4, 2019

possibly with a standalone C file

As a side note, I generally find it quite difficult to debug problems presented in the form of pyproj code. If the problem can be reproduced using PROJ apps or a bit of C code that it is preferable.

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 4, 2019

I definitely see that the explanation is confusing. Code is definitely a good way to communicate. Hopefully this will help:

Step 1:

Move the PROJ data directory to a non-standard location where it cannot be auto detected and is not set by PROJ_LIB.

Step 2:

Create a new context and set the search paths without setting it in the global context. Then, attempt to create a CRS using the init= syntax:

std::string dirname = "/path/to/proj_data/share234/proj";
auto ctx = proj_context_create();

const char *path = dirname.c_str();
proj_context_set_search_paths(ctx, 1, &path);
proj_context_use_proj4_init_rules(ctx, 1);
auto P = proj_create(ctx, "+init=epsg:4326 +type=crs");

Updated to allow init rules ^^

@rouault

This comment has been minimized.

Copy link
Member

@rouault rouault commented Sep 4, 2019

I've tested your above (updated) script, and can't make it crash

$ PROJ_DEBUG=5 LD_LIBRARY_PATH=src/.libs ./mytest 
pj_open_lib(proj.db): call fopen(/path/to/proj_data/share234/proj/proj.db) - failed
internal_proj_create: Cannot find proj.db
pj_open_lib(proj.db): call fopen(/path/to/proj_data/share234/proj/proj.db) - failed
pj_open_lib(epsg): call fopen(/home/even/proj/install-proj-master/share/proj/epsg) - failed
internal_proj_create: no database context specified

The hardcoded data path is (/home/even/proj/install-proj-master/share/proj and I renamted it to (/home/even/proj/install-proj-master/share/proj.disabed

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 4, 2019

One step closer. Can you try with:

std::string dirname = "/home/even/proj/install-proj-master/share/proj.disabed";

The reason is that the new context needs the path to the PROJ data directory in its moved form while the main/global PROJ context cannot find it.

@rouault

This comment has been minimized.

Copy link
Member

@rouault rouault commented Sep 4, 2019

One step closer. Can you try with:

std::string dirname = "/home/even/proj/install-proj-master/share/proj.disabed";

Still no crash. New output is

PROJ_DEBUG=5 LD_LIBRARY_PATH=src/.libs ./mytest 
pj_open_lib(proj.db): call fopen(/home/even/proj/install-proj-master/share/proj.dis/proj.db) - succeeded
pj_open_lib(epsg): call fopen(/home/even/proj/install-proj-master/share/proj/epsg) - failed
@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 5, 2019

Hmmm, strange. Wonder what the difference is on my end. That's where it core dumps in the pyproj code. I wonder if running as a C program instead of Cpp makes any difference?

@rouault

This comment has been minimized.

Copy link
Member

@rouault rouault commented Sep 5, 2019

I wonder if running as a C program instead of Cpp makes any difference?

I doubt so. Why not running your python code under gdb and displaying the stack trace when it crashes as I suggested above in #1574 (comment) ?

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 5, 2019

The output is here: pyproj4/pyproj#368 (comment)

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 5, 2019

Another one is here: pyproj4/pyproj#351

@rouault

This comment has been minimized.

Copy link
Member

@rouault rouault commented Sep 5, 2019

In pyproj4/pyproj#368, there's not the full backtrace and no line number, so hard to do something with it.
pyproj4/pyproj#351 is a bit more interesting as it shows an infinite loop.
Commit 149bd81 was supposed to fix that, or something rather similar.
I'm afraid I can't do more without an exact reproducer.

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 5, 2019

Makes sense. I will see if I can get better debugging information later.

@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 5, 2019

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6a169ac in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
(gdb) bt
#0  0x00007ffff6a169ac in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#1  0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#2  0x00007ffff6a45d22 in proj_create ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#3  0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#4  0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int)
    () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#5  0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#6  0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) ()
   from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
----Type <return> to continue, or q <return> to quit---
#31 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#32 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#33 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#34 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#35 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#36 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#37 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#38 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#39 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#40 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#41 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#42 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#43 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#44 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#45 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#46 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#47 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#48 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<osgeo::proj::io::DatabaseContext> const&, bool, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#49 0x00007ffff6a1a318 in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, projCtx_t*) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#50 0x00007ffff6a45d22 in proj_create () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#51 0x00007ffff6a94f0e in get_init(projCtx_t*, char const*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#52 0x00007ffff6a95182 in pj_expand_init_internal(projCtx_t*, ARG_list*, int) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#53 0x00007ffff6a0e313 in osgeo::proj::io::PROJStringParser::createFromPROJString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/snowal/miniconda3/envs/proj62/lib/libproj.so.15
#54 0x00007ffff6a16a9e in osgeo::proj::io::createFromUserInput(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&---Type <return> to continue, or q <return> to quit---


@snowman2

This comment has been minimized.

Copy link
Contributor Author

@snowman2 snowman2 commented Sep 5, 2019

Seems to be the infinite loop problem.

@rouault

This comment has been minimized.

Copy link
Member

@rouault rouault commented Sep 5, 2019

oh! I can finally reproduce with your procedure #1574 (comment) + #1574 (comment) . The extra needed step is to have a 'epsg' file coming from PROJ 5.2 in the default PROJ resource directory. Or equivalently have PROJ_LIB pointing to the resource directory of PROJ 5.2

@rouault rouault closed this in 9c38bab Sep 6, 2019
rouault added a commit that referenced this issue Sep 6, 2019
PROJStringParser::createFromPROJString(): avoid potential infinite recursion (fixes #1574)
backporting bot pushed a commit that referenced this issue Sep 6, 2019
…cursion (fixes #1574)

The exact circumstances are a bit difficult to explain, but they involve
using a non-default context, enabling proj_context_use_proj4_init_rules() on it,
using proj_create(ctxt, "+init=epsg:XXXX +type=crs"), whereas PROJ_LIB is
defined to a directory that has a 'epsg' file in it.
rouault added a commit that referenced this issue Sep 6, 2019
[Backport 6.2] PROJStringParser::createFromPROJString(): avoid potential infinite recursion (fixes #1574)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.