LLVM symbolizer error with LogisticRegression example

[caoganyuan]

Hello,

I am having an error with the LLVM symbolizer

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):

I have already tried this

echo 'export LLVM_SYMBOLIZER_PATH=/opt/Homebrew/opt/llvm/bin/llvm-symbolizer' >> ~/.zshrc

But it still doesnt work. Any idea?

My settings are

concrete-ml:         1.5.0
concrete-python:     2.6.0rc1
Python:              3.9.6
OS:                  MacOS
Arch:                arm64e

I am running the example on https://docs.zama.ai/concrete-ml/guides/prediction_with_fhe. The line that triggers the error

fhe_circuit = model.compile(x_train)

The full log

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  _mlir.cpython-39-darwin.so              0x000000016e9c5480 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  _mlir.cpython-39-darwin.so              0x000000016e9c43d8 llvm::sys::RunSignalHandlers() + 72
2  _mlir.cpython-39-darwin.so              0x000000016e9c5b20 llvm::sys::PrintStackTraceOnErrorSignal(llvm::StringRef, bool) + 784
3  libsystem_platform.dylib                0x000000018102f584 _sigtramp + 56
4  libConcretelangBindingsPythonCAPI.dylib 0x000000032b161ad8 mlir::SymbolTable::getSymbolName(mlir::Operation*) + 28
5  libConcretelangBindingsPythonCAPI.dylib 0x000000032b161ad8 mlir::SymbolTable::getSymbolName(mlir::Operation*) + 28
6  _concretelang.cpython-39-darwin.so      0x000000016f80ce8c mlir::concretelang::createTFHECircuitSolutionParametrizationPass(std::__1::optional<concrete_optimizer::dag::CircuitSolution>) + 48664
7  _concretelang.cpython-39-darwin.so      0x000000016f80c844 mlir::concretelang::createTFHECircuitSolutionParametrizationPass(std::__1::optional<concrete_optimizer::dag::CircuitSolution>) + 47056
8  libConcretelangBindingsPythonCAPI.dylib 0x000000032b16eb48 mlir::detail::walk(mlir::Operation*, llvm::function_ref<mlir::WalkResult (mlir::Operation*)>, mlir::WalkOrder) + 232
9  libConcretelangBindingsPythonCAPI.dylib 0x000000032b16eb0c mlir::detail::walk(mlir::Operation*, llvm::function_ref<mlir::WalkResult (mlir::Operation*)>, mlir::WalkOrder) + 172
10 _concretelang.cpython-39-darwin.so      0x000000016f802794 mlir::concretelang::createTFHECircuitSolutionParametrizationPass(std::__1::optional<concrete_optimizer::dag::CircuitSolution>) + 5920
11 _concretelang.cpython-39-darwin.so      0x000000016f801cd0 mlir::concretelang::createTFHECircuitSolutionParametrizationPass(std::__1::optional<concrete_optimizer::dag::CircuitSolution>) + 3164
12 libConcretelangBindingsPythonCAPI.dylib 0x000000032b0414ec mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) + 412
13 libConcretelangBindingsPythonCAPI.dylib 0x000000032b041bc4 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) + 312
14 libConcretelangBindingsPythonCAPI.dylib 0x000000032b043764 mlir::PassManager::run(mlir::Operation*) + 1124
15 _concretelang.cpython-39-darwin.so      0x000000016f5ded88 mlir::concretelang::pipeline::parametrizeTFHE(mlir::MLIRContext&, mlir::ModuleOp&, std::__1::optional<mlir::concretelang::V0FHEContext>&, std::__1::function<bool (mlir::Pass*)>) + 1172
16 _concretelang.cpython-39-darwin.so      0x000000016f5ef218 mlir::concretelang::CompilerEngine::compile(mlir::ModuleOp, mlir::concretelang::CompilerEngine::Target, std::__1::optional<std::__1::shared_ptr<mlir::concretelang::CompilerEngine::Library>>) + 3736
17 _concretelang.cpython-39-darwin.so      0x000000016f5f28ec mlir::concretelang::CompilerEngine::compile(mlir::ModuleOp, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, bool, bool, bool) + 420
18 _concretelang.cpython-39-darwin.so      0x000000016f5f27c4 mlir::concretelang::CompilerEngine::compile(mlir::ModuleOp, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, bool, bool, bool) + 124
19 _concretelang.cpython-39-darwin.so      0x000000016f4c17ec PyInit__concretelang + 95960
20 _concretelang.cpython-39-darwin.so      0x000000016f4c130c PyInit__concretelang + 94712
21 _concretelang.cpython-39-darwin.so      0x000000016f5143cc PyInit__concretelang + 434872
22 _concretelang.cpython-39-darwin.so      0x000000016f4b4594 PyInit__concretelang + 42112
23 Python3                                 0x0000000103a1fb58 PyCFunction_GetFlags + 736
24 Python3                                 0x00000001039dc78c _PyObject_MakeTpCall + 360
25 Python3                                 0x00000001039df110 PyMethod_New + 400
26 Python3                                 0x0000000103aaea38 _PyEval_EvalFrameDefault + 34668
27 Python3                                 0x0000000103aabb78 _PyEval_EvalFrameDefault + 22700
28 Python3                                 0x0000000103aaf7ec _PyEval_EvalFrameDefault + 38176
29 Python3                                 0x00000001039dcf18 _PyFunction_Vectorcall + 236
30 Python3                                 0x0000000103aaea38 _PyEval_EvalFrameDefault + 34668
31 Python3                                 0x0000000103aabb54 _PyEval_EvalFrameDefault + 22664
32 Python3                                 0x0000000103aaf7ec _PyEval_EvalFrameDefault + 38176
33 Python3                                 0x00000001039dcf18 _PyFunction_Vectorcall + 236
34 Python3                                 0x0000000103aaea38 _PyEval_EvalFrameDefault + 34668
35 Python3                                 0x0000000103aabc70 _PyEval_EvalFrameDefault + 22948
36 Python3                                 0x00000001039dcfd0 _PyFunction_Vectorcall + 420
37 Python3                                 0x0000000103aaea38 _PyEval_EvalFrameDefault + 34668
38 Python3                                 0x0000000103aabb54 _PyEval_EvalFrameDefault + 22664
39 Python3                                 0x0000000103aaf7ec _PyEval_EvalFrameDefault + 38176
40 Python3                                 0x00000001039dcf18 _PyFunction_Vectorcall + 236
41 Python3                                 0x00000001039dc5b4 _PyObject_FastCallDictTstate + 272
42 Python3                                 0x00000001039dd31c _PyObject_Call_Prepend + 160
43 Python3                                 0x0000000103a41434 _PyType_Lookup + 37636
44 Python3                                 0x0000000103a38a20 _PyType_Lookup + 2288
45 Python3                                 0x00000001039dc78c _PyObject_MakeTpCall + 360
46 Python3                                 0x0000000103aaea98 _PyEval_EvalFrameDefault + 34764
47 Python3                                 0x0000000103aabbf4 _PyEval_EvalFrameDefault + 22824
48 Python3                                 0x0000000103aaf7ec _PyEval_EvalFrameDefault + 38176
49 Python3                                 0x00000001039dcf18 _PyFunction_Vectorcall + 236
50 Python3                                 0x00000001039df0e0 PyMethod_New + 352
51 Python3                                 0x0000000103aaea38 _PyEval_EvalFrameDefault + 34668
52 Python3                                 0x0000000103aabc70 _PyEval_EvalFrameDefault + 22948
53 Python3                                 0x0000000103aaf7ec _PyEval_EvalFrameDefault + 38176
54 Python3                                 0x00000001039dcf18 _PyFunction_Vectorcall + 236
55 Python3                                 0x00000001039df0e0 PyMethod_New + 352
56 Python3                                 0x0000000103aaea38 _PyEval_EvalFrameDefault + 34668
57 Python3                                 0x0000000103aabb78 _PyEval_EvalFrameDefault + 22700
58 Python3                                 0x0000000103aaf7ec _PyEval_EvalFrameDefault + 38176
59 Python3                                 0x0000000103aa6204 PyEval_EvalCode + 80
60 Python3                                 0x0000000103aea2f0 PyParser_ASTFromStringObject + 500
61 Python3                                 0x0000000103aea4c4 PyRun_FileExFlags + 340
62 Python3                                 0x0000000103ae85ec PyRun_SimpleFileExFlags + 820
63 Python3                                 0x0000000103b057cc Py_RunMain + 1604
64 Python3                                 0x0000000103b05c3c Py_Main + 372
65 Python3                                 0x0000000103b05cdc Py_BytesMain + 40
66 dyld                                    0x0000000180c760e0 start + 2360
zsh: segmentation fault  python3 test-concreteML.py

Thank you very much!

[RomanBredehoft]

Hello @caoganyuan ,
Is it a bit hard to know what is going on yet. Could you tell us a bit more about your issue ? In particular, we would be interested in knowing :

• your Concrete ML and Python version
• your OS and architecture
• the code that triggered this crash (especially which line)
• the full traceback if there is one

Thanks !

[caoganyuan]

Hello @caoganyuan , Is it a bit hard to know what is going on yet. Could you tell us a bit more about your issue ? In particular, we would be interested in knowing :

• your Concrete ML and Python version
• your OS and architecture
• the code that triggered this crash (especially which line)
• the full traceback if there is one

Thanks !

Hi. Thank you for updating! I have posted the full logs etc.

[bcm-at-zama]

I would rename the issue: the problem is not in the LLVM symbolication but more that a Concrete ML example fails for you.

[caoganyuan]

I would rename the issue: the problem is not in the LLVM symbolication but more that a Concrete ML example fails for you.

Hi. I changed the issue name a little bit. But I think it should be still with the LLVM symbolizer since when I try the concrete-Python example, it still fails at the step of circuit compilation.

[bcm-at-zama]

thx for changing. Are you able to run the simplest Concrete example, ie https://github.com/zama-ai/concrete?tab=readme-ov-file#a-simple-example?

[caoganyuan]

thx for changing. Are you able to run the simplest Concrete example, ie https://github.com/zama-ai/concrete?tab=readme-ov-file#a-simple-example?

No. That is actually the example I am referring to. The it fails at circuit = compiler.compile(inputset).

[bcm-at-zama]

Not clear to me sorry:

• in the first comment, you mention it crashes on Concrete ML FHE training example ( https://docs.zama.ai/concrete-ml/guides/prediction_with_fhe. )
• my question is for something much easier, with Concrete e https://github.com/zama-ai/concrete?tab=readme-ov-file#a-simple-example

I want to know if the installation is completely broken on your side, or if it's related to the more complex FHE training Concrete ML.

[caoganyuan]

Not clear to me sorry:

• in the first comment, you mention it crashes on Concrete ML FHE training ( https://docs.zama.ai/concrete-ml/guides/prediction_with_fhe. )
• my question is for something much easier, with Concrete e https://github.com/zama-ai/concrete?tab=readme-ov-file#a-simple-example

I want to know if the installation is completely broken on your side, or if it's related to the more complex FHE training.

Hi. Sorry for being confusing. It actually fails in both of the examples.

• In the Concrete-ML example, it fails at fhe_circuit = model.compile(x_train).
• In the Concrete example. it fails at circuit = compiler.compile(inputset).

Hope that's clearer. Thank you!

[RomanBredehoft]

Hello again @caoganyuan,
We recently faced several issues with MacOS M1/M2 configurations, some of which were fixed but it's possible that some others remain. Unfortunately, I was not able to reproduce your issue with Concrete ML 1.5.0 on my mac M2 in a python 3.9 virtual environment.

Therefore, we need more information on your current configuration. Would you be able to run the following commands and share the output ? :

• system_profiler SPHardwareDataType (mostly : Chip: )
• otool --version (mostly: disassmbler: LLVM version)
• xcode-select -v

Thanks a lot !

[caoganyuan]

Hello again @caoganyuan, We recently faced several issues with MacOS M1/M2 configurations, some of which were fixed but it's possible that some others remain. Unfortunately, I was not able to reproduce your issue with Concrete ML 1.5.0 on my mac M2 in a python 3.9 virtual environment.

Therefore, we need more information on your current configuration. Would you be able to run the following commands and share the output ? :

• system_profiler SPHardwareDataType (mostly : Chip: )
• otool --version (mostly: disassmbler: LLVM version)
• xcode-select -v

Thanks a lot !

Hi Roman, thank you for your answer! I am actually using M3 chip. Anyway, here is the info you need

Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: Mac15,6
      Model Number: Z1AV000MQSM/A
      Chip: Apple M3 Pro
      Total Number of Cores: 12 (6 performance and 6 efficiency)
      Memory: 36 GB
      System Firmware Version: 10151.121.1
      OS Loader Version: 10151.121.1
      Serial Number (system): JFQ44RFY7H
      Hardware UUID: 41DF510B-059F-5828-A263-9871A8E5A680
      Provisioning UDID: 00006030-000228E00C84001C
      Activation Lock Status: Disabled

llvm-otool(1): Apple Inc. version cctools-1010.6
otool(1): Apple Inc. version cctools-1010.6
disassmbler: LLVM version 15.0.0

xcode-select version 2408

[RomanBredehoft]

Thanks a lot @caoganyuan, we'll check on our side if we spot anything wrong with this config and we'll get back to you when we know more !

[RomanBredehoft]

Hello again @caoganyuan , we believe that we have identified the issue and it seems to have already been fixed in more recent Concrete (public but non-stable) versions 🙂 We are thus planning on publishing a patch release for Concrete ML in the near future, please stay tuned !

Until then, you could try to manually install Concrete's latest nightly version on top through pip install --index-url https://pypi.zama.ai/ concrete-python==2.6.2.dev20240528. However, be aware that this is not a stable release and you could stumble on other issues with this version !

[caoganyuan]

Hello again @caoganyuan , we believe that we have identified the issue and it seems to have already been fixed in more recent Concrete (public but non-stable) versions 🙂 We are thus planning on publishing a patch release for Concrete ML in the near future, please stay tuned !

Until then, you could try to manually install Concrete's latest nightly version on top through pip install --index-url https://pypi.zama.ai/ concrete-python==2.6.2.dev20240528. However, be aware that this is not a stable release and you could stumble on other issues with this version !

Hello again @caoganyuan , we believe that we have identified the issue and it seems to have already been fixed in more recent Concrete (public but non-stable) versions 🙂 We are thus planning on publishing a patch release for Concrete ML in the near future, please stay tuned !

Until then, you could try to manually install Concrete's latest nightly version on top through pip install --index-url https://pypi.zama.ai/ concrete-python==2.6.2.dev20240528. However, be aware that this is not a stable release and you could stumble on other issues with this version !

Great! Thank you very much for updating!

[RomanBredehoft]

Hello again @caoganyuan,
You could try again with our latest release, Concrete ML 1.6.0 ! Normally, your issue should now be fixed 😉

[caoganyuan]

Hello again @caoganyuan, You could try again with our latest release, Concrete ML 1.6.0 ! Normally, your issue should now be fixed 😉

Hi @RomanBredehoft,

It works perfectly now!.

Thanks for the follow-up :)