Segmentation fault if pandas imported before ortools

[simon-b]

What version of OR-Tools and what language are you using?
Version: 9.3.10497
Language: python 3.8.12

Which solver are you using (e.g. CP-SAT, Routing Solver, GLOP, BOP, Gurobi)
CP-SAT

What operating system (Linux, Windows, ...) and version?
macOS 12.3.1

What did you do?
Import pandas (1.3.4) in a codebase that uses ortools. E.g. "solve_with_time_limit_sample_sat" sample, with import pandas added to the top)
https://github.com/google/or-tools/blob/stable/ortools/sat/samples/solve_with_time_limit_sample_sat.py

What did you expect to see
No segmentation fault

What did you see instead?
A segmentation fault

Small failing example:

import pandas
from ortools.sat.python import cp_model

model = cp_model.CpModel()
model.NewIntVar(0, 1, "x")
model.ExportToFile("model.txt")

But this works fine:

from ortools.sat.python import cp_model

model = cp_model.CpModel()
model.NewIntVar(0, 1, "x")
model.ExportToFile("model.txt")

And so does this:

from ortools.sat.python import cp_model
import pandas

model = cp_model.CpModel()
model.NewIntVar(0, 1, "x")
model.ExportToFile("model.txt")

Stack backtrace
From lldb :

  * frame #0: 0x000000016f7aab3d libortools.9.dylib`operations_research::sat::CpModelProto::_InternalSerialize(unsigned char*, google::protobuf::io::EpsCopyOutputStream*) const + 877
    frame #1: 0x000000016c45ff70 libarrow.600.dylib`google::protobuf::MessageLite::ParseFromString(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 320
    frame #2: 0x0000000152b5b554 swig_helper.so`___lldb_unnamed_symbol89$$swig_helper.so + 52
    frame #3: 0x0000000152b5f5ba swig_helper.so`___lldb_unnamed_symbol96$$swig_helper.so + 90
    frame #4: 0x0000000152b5643a swig_helper.so`___lldb_unnamed_symbol44$$swig_helper.so + 5034
    frame #5: 0x00000001000cd884 python`cfunction_call + 52
    frame #6: 0x0000000100071c48 python`_PyObject_MakeTpCall + 136
    frame #7: 0x00000001001a12b7 python`call_function + 311
    frame #8: 0x000000010019ed5f python`_PyEval_EvalFrameDefault + 44175
    frame #9: 0x0000000100072e78 python`_PyFunction_Vectorcall + 248
    frame #10: 0x00000001001a122c python`call_function + 172
    frame #11: 0x000000010019ecb9 python`_PyEval_EvalFrameDefault + 44009
    frame #12: 0x0000000100192c6d python`_PyEval_EvalCode + 557
    frame #13: 0x00000001002020b1 python`pyrun_file + 321
    frame #14: 0x000000010020190a python`pyrun_simple_file + 394
    frame #15: 0x000000010020172d python`PyRun_SimpleFileExFlags + 109
    frame #16: 0x00000001002294d9 python`pymain_run_file + 329
    frame #17: 0x0000000100228a7b python`pymain_run_python + 443
    frame #18: 0x0000000100228865 python`Py_RunMain + 37
    frame #19: 0x0000000100229f11 python`pymain_main + 49
    frame #20: 0x0000000100001e98 python`main + 56
    frame #21: 0x00000001003ff51e dyld`start + 462

[lperron]

Does it works with 1.4.2 ?

[simon-b]

I have two python environments (python 3.8.12, 3.9.7) and on both I see this problem, and also if I update pandas to 1.4.2.
However with a fresh python environment and only ortools==9.3.10497 and pandas==1.4.2 installed, I do not observe this.

However I notice that there are differences in dependencies pulled -- e.g. I have protobuf==3.19.4 (which appears in backtrace) in the failing cases, whereas the fresh env has grabbed protobuf==3.20.1 as a dependency of ortools.

I will investigate further.

[Mizux]

FYI last ortools v9.3.10497 was built against protobuf 3.19.4 so if you import ortools first then symbols will be read from the libortools.so aka will be 3.19.4 while if you import panda first you may end up with protobuf 3.20.1 symbols (according to your investigation).
note1: dlopen() won't overwrite symbols already present in the runtime (ed this is why LD_PRELOAD can work).
note2: python load everything in its process so you can't have several versions (i.e. symbols) of the same lib since everything is in the same [unix] process.

[simon-b]

I did a little more investigation and managed to get the following minimal environment that fails (python 3.8.12):

pandas==1.4.2
ortools==9.3.10497
pyarrow==7.0.0

Without pyarrow, it works.

With the above, currently pip installs:

absl-py==1.0.0
certifi==2022.5.18.1
numpy==1.22.4
ortools==9.3.10497
pandas==1.4.2
protobuf==4.21.0
pyarrow==7.0.0
python-dateutil==2.8.2
pytz==2022.1
six==1.16.0

[EricPrescottGagnon]

If that helps investigating, I am also seeing this issue with the following:

What version of OR-Tools and what language are you using?
Version: 9.4.1874
Language: python 3.9.5
pandas: 1.4.1

Which solver are you using (e.g. CP-SAT, Routing Solver, GLOP, BOP, Gurobi)
Routing Solver

What operating system (Linux, Windows, ...) and version?
macOS 11.6.6

Running the exact same code and package versions inside a docker container running Ubuntu 20.04.5, I don't get the problem.

[lperron]

Duplicate of #3230.

Incompatible protobuf versions being pulled.

[qemtek]

@lperron is there a fix for this? I seem to be getting this issue when running unit tests that use ORTools but not otherwise.

[lperron]

Not until tensorflow upgrades to the latest version of protobuf. Laurent Perron | Operations Research | ***@***.*** | (33) 1 42 68 53 00 Le lun. 30 janv. 2023 à 14:42, Christopher Collins ***@***.***> a écrit :

…

@lperron <https://github.com/lperron> is there a fix for this? I seem to be getting this issue when running unit tests that use ORTools but not otherwise. — Reply to this email directly, view it on GitHub <#3308 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUPL3J6LKMS542XD3C2Z2LWU7ALJANCNFSM5XCLGREQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>