Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Double free or corruption on numba import w/ icc-rt #6471

Closed
ryan-chien opened this issue Nov 12, 2020 · 14 comments
Closed

Double free or corruption on numba import w/ icc-rt #6471

ryan-chien opened this issue Nov 12, 2020 · 14 comments
Labels
no action required No action was needed to resolve.

Comments

@ryan-chien
Copy link

ryan-chien commented Nov 12, 2020 •
edited

Hey Stuart and Siu,

I began having build errors for numba-accelerated libraries yesterday. I think it may be related to icc-rt. For example:

no-icc-rt env (this works fine):

(no-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# conda list
# packages in environment at /opt/conda/envs/no-icc-rt:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
ca-certificates           2020.11.8            ha878542_0    conda-forge
certifi                   2020.11.8        py37h89c1867_0    conda-forge
ld_impl_linux-64          2.35                 h769bd43_9    conda-forge
libblas                   3.9.0                2_openblas    conda-forge
libcblas                  3.9.0                2_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.3.0               h5dbcf3e_17    conda-forge
libgfortran-ng            9.3.0               he4bcb1c_17    conda-forge
libgfortran5              9.3.0               he4bcb1c_17    conda-forge
libgomp                   9.3.0               h5dbcf3e_17    conda-forge
liblapack                 3.9.0                2_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libstdcxx-ng              9.3.0               h2ae2ef3_17    conda-forge
llvmlite                  0.34.0           py37h5202443_2    conda-forge
ncurses                   6.2                  h58526e2_3    conda-forge
numba                     0.51.2           py37h9fdb41a_0    conda-forge
numpy                     1.19.4           py37h7e9df27_1    conda-forge
openssl                   1.1.1h               h516909a_0    conda-forge
pip                       20.2.4                     py_0    conda-forge
python                    3.7.8           h6f2ec95_1_cpython    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
setuptools                49.6.0           py37he5f6b98_2    conda-forge
sqlite                    3.33.0               h4cf870e_1    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
(no-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# python
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba as nb
>>> exit()

yes-icc-rt env (abort due to corruption):

(yes-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# conda list
# packages in environment at /opt/conda/envs/yes-icc-rt:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
ca-certificates           2020.11.8            ha878542_0    conda-forge
certifi                   2020.11.8        py37h89c1867_0    conda-forge
icc_rt                    2020.2                intel_254    numba
ld_impl_linux-64          2.35                 h769bd43_9    conda-forge
libblas                   3.9.0                2_openblas    conda-forge
libcblas                  3.9.0                2_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.3.0               h5dbcf3e_17    conda-forge
libgfortran-ng            9.3.0               he4bcb1c_17    conda-forge
libgfortran5              9.3.0               he4bcb1c_17    conda-forge
libgomp                   9.3.0               h5dbcf3e_17    conda-forge
liblapack                 3.9.0                2_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libstdcxx-ng              9.3.0               h2ae2ef3_17    conda-forge
llvmlite                  0.34.0           py37h5202443_2    conda-forge
ncurses                   6.2                  h58526e2_3    conda-forge
numba                     0.51.2           py37h9fdb41a_0    conda-forge
numpy                     1.19.4           py37h7e9df27_1    conda-forge
openssl                   1.1.1h               h516909a_0    conda-forge
pip                       20.2.4                     py_0    conda-forge
python                    3.7.8           h6f2ec95_1_cpython    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
setuptools                49.6.0           py37he5f6b98_2    conda-forge
sqlite                    3.33.0               h4cf870e_1    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
(yes-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats# python
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba as nb
double free or corruption (top)
Aborted
(yes-icc-rt) root@b9fa89e79c6b:/workspaces/numba-stats#

Any thoughts would be helpful. Thanks!

This is on an Ubuntu 18.04 on Docker v19.03.13 using Hyper-V.

BR,
Ryan
@stuartarchibald
Copy link
Contributor

Thanks for the report @ryan-chien,

I've tried to reproduce but unfortunately cannot:

$ conda list
# packages in environment at <path>/envs/_tmp_6471_1:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
ca-certificates           2020.11.8            ha878542_0    conda-forge
certifi                   2020.11.8        py37h89c1867_0    conda-forge
icc_rt                    2020.2                intel_254    numba
ld_impl_linux-64          2.35                 h769bd43_9    conda-forge
libblas                   3.9.0                2_openblas    conda-forge
libcblas                  3.9.0                2_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.3.0               h5dbcf3e_17    conda-forge
libgfortran-ng            9.3.0               he4bcb1c_17    conda-forge
libgfortran5              9.3.0               he4bcb1c_17    conda-forge
libgomp                   9.3.0               h5dbcf3e_17    conda-forge
liblapack                 3.9.0                2_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libopenblas               0.3.12          pthreads_h4812303_1    conda-forge
libstdcxx-ng              9.3.0               h2ae2ef3_17    conda-forge
llvmlite                  0.34.0           py37h5202443_2    conda-forge
ncurses                   6.2                  h58526e2_3    conda-forge
numba                     0.51.2           py37h9fdb41a_0    conda-forge
numpy                     1.19.4           py37h7e9df27_1    conda-forge
openssl                   1.1.1h               h516909a_0    conda-forge
pip                       20.2.4                     py_0    conda-forge
python                    3.7.8           h6f2ec95_1_cpython    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
readline                  8.0                  he28a2e2_2    conda-forge
setuptools                49.6.0           py37he5f6b98_2    conda-forge
sqlite                    3.33.0               h4cf870e_1    conda-forge
tk                        8.6.10               hed695b0_1    conda-forge
wheel                     0.35.1             pyh9f0ad1d_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge

$ python
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numba as nb
>>>

$

could you run python under gdb and get a back trace to paste here please? If you aren't sure how to do this then I can provide instructions. Many thanks.

@ryan-chien
Copy link
Author

Yes - can you please tell me how to run python under gdb?

@ryan-chien
Copy link
Author

ryan-chien commented Nov 12, 2020 •
edited

Hey Stuart - one of my coworkers suggested that I try removing Ubuntu packages to see if one was the issue. The answer is yes - numba with icc_rt imports fine if I do not install libudunits2-dev, libgdal-dev, and libhdf2-dev. I'm happy to help do additional testing if you want.

@stuartarchibald
Copy link
Contributor

Hi @ryan-chien,

Yes - can you please tell me how to run python under gdb?

Sure. First off, this demonstrates gdb catching a SIGSEGV and backtracing it.

$ gdb python -ex 'set confirm off' -ex "r -c 'import ctypes;ctypes.CDLL(None).free(0x1)'" -ex "bt" -ex 'q'

the above binds to the free(3) symbol in the Python process and then tries to free address 0x1 which leads to a SIGSEGV (it's like writing free((void*)0x1ul); in C. To confirm that it is indeed working as expected, you should see something like this as output:

$ gdb python -ex 'set confirm off' -ex "r -c 'import ctypes;ctypes.CDLL(None).free(0x1)'" -ex "bt" -ex 'q'
<snip>
Reading symbols from <path>/bin/python3.7...done.
Starting program: <path>/bin/python -c 'import ctypes;ctypes.CDLL(None).free(0x1)'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
__GI___libc_free (mem=0x1) at malloc.c:2941
2941      if (chunk_is_mmapped(p))                       /* release mmapped memory. */
#0  __GI___libc_free (mem=0x1) at malloc.c:2941
#1  0x00007ffff7fee630 in ffi_call_unix64 ()
   from <path>/lib/python3.7/lib-dynload/../../libffi.so.6
#2  0x00007ffff7fedfed in ffi_call ()
   from <path>/lib/python3.7/lib-dynload/../../libffi.so.6
#3  0x00007ffff7e126d7 in _call_function_pointer (argcount=1, resmem=0x7fffffffc610, restype=<optimized out>, 
    atypes=0x7fffffffc5d0, avalues=0x7fffffffc5f0, pProc=0x7ffff7876ad0 <__GI___libc_free>, flags=4353)
    at /usr/local/src/conda/python-3.7.8/Modules/_ctypes/callproc.c:829
#4  _ctypes_callproc () at /usr/local/src/conda/python-3.7.8/Modules/_ctypes/callproc.c:1201
#5  0x00007ffff7e13144 in PyCFuncPtr_call () at /usr/local/src/conda/python-3.7.8/Modules/_ctypes/_ctypes.c:4025
#6  0x00005555556d2afc in _PyObject_FastCallKeywords ()
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:199
#7  0x00005555556d34e1 in call_function ()
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4619
#8  0x000055555571aef2 in _PyEval_EvalFrameDefault ()
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3093
#9  0x000055555566cdb2 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff7edb450)
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#10 _PyEval_EvalCodeWithName () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3930
#11 0x000055555566e1e3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, 
    args=0x0, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3959
#12 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:524
#13 0x000055555577b1d2 in run_mod ()
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:1035
#14 0x000055555578617a in PyRun_StringFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:959
#15 0x00005555557861dc in PyRun_SimpleStringFlags ()
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:455
#16 0x00005555557862d9 in pymain_run_command (cf=0x7fffffffcb60, command=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:414
#17 pymain_run_python (pymain=0x7fffffffcc70)
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:2901
#18 pymain_main () at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:3068
#19 0x000055555578667c in _Py_UnixMain ()
    at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:3103
#20 0x00007ffff7813555 in __libc_start_main (main=0x555555645240 <main>, argc=3, argv=0x7fffffffcdc8, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffcdb8) at ../csu/libc-start.c:266
#21 0x000055555572b101 in _start ()

Assuming the above test works fine, the following line should catch the problem with icc_rt:

$ gdb python -ex 'set confirm off' -ex "r -c 'import numba as nb; exit();'" -ex "bt" -ex 'q'

if you could please share the output of the backtrace from that it'd be appreciated, feel free to redact paths as desired.

Many thanks!

@stuartarchibald
Copy link
Contributor

Hey Stuart - one of my coworkers suggested that I try removing Ubuntu packages to see if one was the issue. The answer is yes - numba with icc_rt imports fine if I do not install libudunits2-dev, libgdal-dev, and libhdf2-dev. I'm happy to help do additional testing if you want.

Thanks for testing this, this could hint at library/symbol version problems, the gdb trace should point to where.

@ryan-chien
Copy link
Author

OK! Thanks for the example I don't think I would have figured out how to use gdb. Here is the output of the backtrace:

(numba-stats) root@ed0d35318e24:/workspaces/numba-stats# gdb python -ex 'set confirm off' -ex "r -c 'import numba as nb; exit();'" -ex "bt" -ex 'q'
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-conda-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
Starting program: /opt/conda/envs/numba-stats/bin/python -c 'import numba as nb; exit();'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after fork from child process 3347]
[New Thread 0x7f8426a76700 (LWP 3349)]
[New Thread 0x7f841e275700 (LWP 3350)]
[New Thread 0x7f8415a74700 (LWP 3351)]
[New Thread 0x7f840d273700 (LWP 3352)]
[New Thread 0x7f8404a72700 (LWP 3353)]
[New Thread 0x7f8404271700 (LWP 3354)]
[New Thread 0x7f83f3a70700 (LWP 3355)]
[New Thread 0x7f83eb26f700 (LWP 3356)]
[New Thread 0x7f83e2a6e700 (LWP 3357)]
[New Thread 0x7f83da26d700 (LWP 3358)]
[New Thread 0x7f83d1a6c700 (LWP 3359)]
double free or corruption (top)

Thread 1 "python" received signal SIGABRT, Aborted.
0x00007f842de87fb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#0  0x00007f842de87fb7 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f842de89921 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f842ded2967 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f842ded99da in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f842dee121c in free () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f842e45f08f in ?? () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f842e45f3d8 in ?? () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f842e462839 in ?? () from /lib64/ld-linux-x86-64.so.2
--Type <RET> for more, q to quit, c to continue without paging--c
#8  0x00007f842e46e0a4 in ?? () from /lib64/ld-linux-x86-64.so.2
#9  0x00007f842dfb01ef in _dl_catch_exception () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x00007f842e46d98a in ?? () from /lib64/ld-linux-x86-64.so.2
#11 0x00007f842dc45f96 in ?? () from /lib/x86_64-linux-gnu/libdl.so.2
#12 0x00007f842dfb01ef in _dl_catch_exception () from /lib/x86_64-linux-gnu/libc.so.6
#13 0x00007f842dfb027f in _dl_catch_error () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x00007f842dc46745 in ?? () from /lib/x86_64-linux-gnu/libdl.so.2
#15 0x00007f842dc46051 in dlopen () from /lib/x86_64-linux-gnu/libdl.so.2
#16 0x00007f842b0d7684 in llvm::sys::DynamicLibrary::HandleSet::DLOpen(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /opt/conda/envs/numba-stats/lib/python3.7/site-packages/llvmlite/binding/libllvmlite.so
#17 0x00007f842b0d791a in llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /opt/conda/envs/numba-stats/lib/python3.7/site-packages/llvmlite/binding/libllvmlite.so
#18 0x00007f842955b97d in llvm::sys::DynamicLibrary::LoadLibraryPermanently () at /usr/local/src/conda-prefix/include/llvm/Support/DynamicLibrary.h:87
#19 LLVMPY_LoadLibraryPermanently (filename=<optimized out>, OutError=0x7f83c800e490) at dylib.cpp:25
#20 0x00007f842d0a0630 in ffi_call_unix64 () from /opt/conda/envs/numba-stats/lib/python3.7/lib-dynload/../../libffi.so.6
#21 0x00007f842d09ffed in ffi_call () from /opt/conda/envs/numba-stats/lib/python3.7/lib-dynload/../../libffi.so.6
#22 0x00007f842d0b66d7 in _call_function_pointer (argcount=2, resmem=0x7ffdd1c86ad0, restype=<optimized out>, atypes=0x7ffdd1c86a90, avalues=0x7ffdd1c86ab0, pProc=0x7f842955b940 <LLVMPY_LoadLibraryPermanently(char const*, char const**)>, flags=4353) at /usr/local/src/conda/python-3.7.8/Modules/_ctypes/callproc.c:829
#23 _ctypes_callproc (pProc=0x7f842955b940 <LLVMPY_LoadLibraryPermanently(char const*, char const**)>, argtuple=<optimized out>, flags=4353, argtypes=<optimized out>, restype=<_ctypes.PyCSimpleType at remote 0x55b1dbadaab0>, checker=0x0) at /usr/local/src/conda/python-3.7.8/Modules/_ctypes/callproc.c:1201
#24 0x00007f842d0b7144 in PyCFuncPtr_call (self=<optimized out>, inargs=<optimized out>, kwds={}) at /usr/local/src/conda/python-3.7.8/Modules/_ctypes/_ctypes.c:4025
#25 0x000055b1da1ffab6 in PyObject_Call (callable=<_FuncPtr(__name__='LLVMPY_LoadLibraryPermanently') at remote 0x7f842c8851f0>, args=<optimized out>, kwargs=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:245
#26 0x000055b1da2a93a3 in do_call_core (kwdict={}, callargs=(b'libsvml.so', <OutputString(_ptr=<c_char_p at remote 0x7f83c800e440>, _as_parameter_=<CArgObject at remote 0x7f83c8087f70>, _owned=True) at remote 0x7f83c8087fd0>), func=<_FuncPtr(__name__='LLVMPY_LoadLibraryPermanently') at remote 0x7f842c8851f0>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4645
#27 _PyEval_EvalFrameDefault (f=Frame 0x7f842d1ee810, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/llvmlite/binding/ffi.py, line 113, in __call__ (self=<_lib_fn_wrapper at remote 0x7f842d127bd0>, args=(b'libsvml.so', <OutputString(_ptr=<c_char_p at remote 0x7f83c800e440>, _as_parameter_=<CArgObject at remote 0x7f83c8087f70>, _owned=True) at remote 0x7f83c8087fd0>), kwargs={}), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3191
#28 0x000055b1da1fddb2 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f842d1ee810, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/llvmlite/binding/ffi.py, line 113, in __call__ (self=<_lib_fn_wrapper at remote 0x7f842d127bd0>, args=(b'libsvml.so', <OutputString(_ptr=<c_char_p at remote 0x7f83c800e440>, _as_parameter_=<CArgObject at remote 0x7f83c8087f70>, _owned=True) at remote 0x7f83c8087fd0>), kwargs={})) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#29 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name='__call__', qualname='_lib_fn_wrapper.__call__') at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3930
#30 0x000055b1da1ff502 in _PyFunction_FastCallDict (kwargs=<optimized out>, nargs=3, args=0x7ffdd1c86fe0, func=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:376
#31 _PyObject_FastCallDict (callable=<optimized out>, args=0x7ffdd1c86fe0, nargs=3, kwargs=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:98
#32 0x000055b1da26b52c in _PyObject_Call_Prepend (kwargs=0x0, args=(b'libsvml.so', <OutputString(_ptr=<c_char_p at remote 0x7f83c800e440>, _as_parameter_=<CArgObject at remote 0x7f83c8087f70>, _owned=True) at remote 0x7f83c8087fd0>), obj=<optimized out>, callable=<function at remote 0x7f842d06a680>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:906
#33 slot_tp_call (self=<optimized out>, args=(b'libsvml.so', <OutputString(_ptr=<c_char_p at remote 0x7f83c800e440>, _as_parameter_=<CArgObject at remote 0x7f83c8087f70>, _owned=True) at remote 0x7f83c8087fd0>), kwds=0x0) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/typeobject.c:6402
#34 0x000055b1da263afc in _PyObject_FastCallKeywords (callable=<_lib_fn_wrapper at remote 0x7f842d127bd0>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:199
#35 0x000055b1da2644e1 in call_function (pp_stack=0x7ffdd1c87178, oparg=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4619
#36 0x000055b1da2abef2 in _PyEval_EvalFrameDefault (f=Frame 0x55b1dba5a4a0, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/llvmlite/binding/dylib.py, line 29, in load_library_permanently (filename='libsvml.so', outerr=<OutputString(_ptr=<c_char_p at remote 0x7f83c800e440>, _as_parameter_=<CArgObject at remote 0x7f83c8087f70>, _owned=True) at remote 0x7f83c8087fd0>), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3093
#37 0x000055b1da21d767 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x55b1dba5a4a0, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/llvmlite/binding/dylib.py, line 29, in load_library_permanently (filename='libsvml.so', outerr=<OutputString(_ptr=<c_char_p at remote 0x7f83c800e440>, _as_parameter_=<CArgObject at remote 0x7f83c8087f70>, _owned=True) at remote 0x7f83c8087fd0>)) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#38 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:283
#39 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7f83c8103dd0, nargs=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:408
#40 0x000055b1da264335 in call_function (pp_stack=0x7ffdd1c87368, oparg=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4616
#41 0x000055b1da2abef2 in _PyEval_EvalFrameDefault (f=Frame 0x7f83c8103c50, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py, line 150, in _try_enable_svml (), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3093
#42 0x000055b1da21d767 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f83c8103c50, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py, line 150, in _try_enable_svml ()) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#43 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:283
#44 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x55b1dba18ae0, nargs=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:408
#45 0x000055b1da2a7ae5 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4616
#46 _PyEval_EvalFrameDefault (f=Frame 0x55b1dba18970, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py, line 200, in <module> (), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3124
#47 0x000055b1da1fddb2 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x55b1dba18970, for file /opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py, line 200, in <module> ()) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#48 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3930
#49 0x000055b1da2bbc90 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals={'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n...(truncated), globals={'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n...(truncated), _co=<code at remote 0x7f842d2c5540>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3959
#50 PyEval_EvalCode (locals={'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n...(truncated), globals={'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n...(truncated), co=<code at remote 0x7f842d2c5540>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:524
#51 builtin_exec_impl.isra.12 (locals={'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n...(truncated), globals={'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n...(truncated), source=<code at remote 0x7f842d2c5540>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/bltinmodule.c:1079
#52 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/clinic/bltinmodule.c.h:283
#53 0x000055b1da22de92 in _PyMethodDef_RawFastCallDict (method=0x55b1da416a60 <builtin_methods+480>, self=<module at remote 0x7f842e4bdd10>, args=<optimized out>, nargs=2, kwargs={}) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:530
#54 0x000055b1da2acb9c in _PyCFunction_FastCallDict (kwargs={}, nargs=<optimized out>, args=0x7f842d2d57e8, func=<built-in method exec of module object at remote 0x7f842e4bdd10>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:585
#55 PyCFunction_Call (kwargs={}, args=(<code at remote 0x7f842d2c5540>, {'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions...(truncated), func=<built-in method exec of module object at remote 0x7f842e4bdd10>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:789
#56 do_call_core (kwdict={}, callargs=(<code at remote 0x7f842d2c5540>, {'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions...(truncated), func=<built-in method exec of module object at remote 0x7f842e4bdd10>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4641
#57 _PyEval_EvalFrameDefault (f=Frame 0x7f842d39da00, for file <frozen importlib._bootstrap>, line 219, in _call_with_frames_removed (f=<built-in method exec of module object at remote 0x7f842e4bdd10>, args=(<code at remote 0x7f842d2c5540>, {'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__...(truncated), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3191
#58 0x000055b1da1fddb2 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f842d39da00, for file <frozen importlib._bootstrap>, line 219, in _call_with_frames_removed (f=<built-in method exec of module object at remote 0x7f842e4bdd10>, args=(<code at remote 0x7f842d2c5540>, {'__name__': 'numba', '__doc__': '\nExpose top-level symbols that are safe for import *\n', '__package__': 'numba', '__loader__': <SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, '__spec__': <ModuleSpec(name='numba', loader=<...>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, '__path__': [...], '__file__': '/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', '__cached__...(truncated)) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#59 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f842d3dccd8, kwcount=<optimized out>, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name='_call_with_frames_removed', qualname='_call_with_frames_removed') at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3930
#60 0x000055b1da21db63 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7f842d3dccc0, nargs=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:433
#61 0x000055b1da264335 in call_function (pp_stack=0x7ffdd1c87b18, oparg=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4616
#62 0x000055b1da2abef2 in _PyEval_EvalFrameDefault (f=Frame 0x7f842d3dcb30, for file <frozen importlib._bootstrap_external>, line 728, in exec_module (self=<SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, module=<module at remote 0x7f842d36b230>, code=<code at remote 0x7f842d2c5540>), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3093
#63 0x000055b1da21d767 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f842d3dcb30, for file <frozen importlib._bootstrap_external>, line 728, in exec_module (self=<SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, module=<module at remote 0x7f842d36b230>, code=<code at remote 0x7f842d2c5540>)) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#64 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:283
#65 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x55b1dba2c8b8, nargs=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:408
#66 0x000055b1da264335 in call_function (pp_stack=0x7ffdd1c87d08, oparg=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4616
#67 0x000055b1da2a7d01 in _PyEval_EvalFrameDefault (f=Frame 0x55b1dba2c730, for file <frozen importlib._bootstrap>, line 677, in _load_unlocked (spec=<ModuleSpec(name='numba', loader=<SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, module=<module at remote 0x7f842d36b230>), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3110
#68 0x000055b1da21d767 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x55b1dba2c730, for file <frozen importlib._bootstrap>, line 677, in _load_unlocked (spec=<ModuleSpec(name='numba', loader=<SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>, module=<module at remote 0x7f842d36b230>)) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#69 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:283
#70 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x55b1dba3e910, nargs=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:408
#71 0x000055b1da2a7ae5 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4616
#72 _PyEval_EvalFrameDefault (f=Frame 0x55b1dba3e760, for file <frozen importlib._bootstrap>, line 967, in _find_and_load_unlocked (name='numba', import_=<built-in method __import__ of module object at remote 0x7f842e4bdd10>, path=None, parent='', spec=<ModuleSpec(name='numba', loader=<SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3124
#73 0x000055b1da21d767 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x55b1dba3e760, for file <frozen importlib._bootstrap>, line 967, in _find_and_load_unlocked (name='numba', import_=<built-in method __import__ of module object at remote 0x7f842e4bdd10>, path=None, parent='', spec=<ModuleSpec(name='numba', loader=<SourceFileLoader(name='numba', path='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py') at remote 0x7f842d2d6650>, origin='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__init__.py', loader_state=None, submodule_search_locations=['/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba'], _set_fileattr=True, _cached='/opt/conda/envs/numba-stats/lib/python3.7/site-packages/numba/__pycache__/__init__.cpython-37.pyc', _initializing=True) at remote 0x7f842d2d66d0>)) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#74 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:283
#75 _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x55b1dba4a648, nargs=<optimized out>, kwnames=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:408
#76 0x000055b1da2a7ae5 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4616
#77 _PyEval_EvalFrameDefault (f=Frame 0x55b1dba4a4b0, for file <frozen importlib._bootstrap>, line 983, in _find_and_load (name='numba', import_=<built-in method __import__ of module object at remote 0x7f842e4bdd10>, module=<object at remote 0x7f842d475060>), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3124
#78 0x000055b1da1ff3a6 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x55b1dba4a4b0, for file <frozen importlib._bootstrap>, line 983, in _find_and_load (name='numba', import_=<built-in method __import__ of module object at remote 0x7f842e4bdd10>, module=<object at remote 0x7f842d475060>)) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#79 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:283
#80 _PyFunction_FastCallDict (kwargs=0x0, nargs=<optimized out>, args=<optimized out>, func=<function at remote 0x7f842d48da70>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:322
#81 _PyObject_FastCallDict (callable=<function at remote 0x7f842d48da70>, args=<optimized out>, nargs=<optimized out>, kwargs=0x0) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:98
#82 0x000055b1da200272 in object_vacall (callable=<function at remote 0x7f842d48da70>, vargs=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:1200
#83 0x000055b1da227cb3 in _PyObject_CallMethodIdObjArgs (obj=<optimized out>, name=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Objects/call.c:1250
#84 0x000055b1da1f907e in import_find_and_load (abs_name='numba') at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/import.c:1652
#85 PyImport_ImportModuleLevelObject (name='numba', globals=<optimized out>, locals=<optimized out>, fromlist=None, level=0) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/import.c:1764
#86 0x000055b1da2a9cb7 in import_name (level=0, fromlist=None, name='numba', f=Frame 0x7f842d3e9450, for file <string>, line 1, in <module> ()) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:4770
#87 _PyEval_EvalFrameDefault (f=Frame 0x7f842d3e9450, for file <string>, line 1, in <module> (), throwflag=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:2600
#88 0x000055b1da1fddb2 in PyEval_EvalFrameEx (throwflag=0, f=Frame 0x7f842d3e9450, for file <string>, line 1, in <module> ()) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:547
#89 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3930
#90 0x000055b1da1ff1e3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:3959
#91 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/ceval.c:524
#92 0x000055b1da30c1d2 in run_mod (mod=<optimized out>, filename=<optimized out>, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55b1db9ce8a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f842e4bdd10>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55b1db9ce8a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f842e4bdd10>}, flags=<optimized out>, arena=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:1035
#93 0x000055b1da31717a in PyRun_StringFlags (str=0x7f842d40cad0 "import numba as nb; exit();\n", start=257, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55b1db9ce8a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f842e4bdd10>}, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x55b1db9ce8a0>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7f842e4bdd10>}, flags=0x7ffdd1c88660) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:959
#94 0x000055b1da3171dc in PyRun_SimpleStringFlags (command=0x7f842d40cad0 "import numba as nb; exit();\n", flags=0x7ffdd1c88660) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Python/pythonrun.c:455
#95 0x000055b1da3172d9 in pymain_run_command (cf=0x7ffdd1c88660, command=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:414
#96 pymain_run_python (pymain=0x7ffdd1c88770) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:2901
#97 pymain_main (pymain=0x7ffdd1c88770) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:3068
#98 0x000055b1da31767c in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1596159872474/work/Modules/main.c:3103
#99 0x00007f842de6abf7 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#100 0x000055b1da2bc101 in _start ()

@ryan-chien
Copy link
Author

Also if you don't mind - hoping to learn something about memory pointers. If ctypes.CDLL(None) points to the following memory address:

>>> ctypes.CDLL(None)
<CDLL 'None', handle 7f22d9b84170 at 0x7f22d87d6f50>

Then why does trying to free 0x1 cause a segfault?

@stuartarchibald
Copy link
Contributor

Also if you don't mind - hoping to learn something about memory pointers. If ctypes.CDLL(None) points to the following memory address:

>>> ctypes.CDLL(None)
<CDLL 'None', handle 7f22d9b84170 at 0x7f22d87d6f50>

Then why does trying to free 0x1 cause a segfault?

Ref: https://docs.python.org/3/library/ctypes.html#loading-shared-libraries

All these classes can be instantiated by calling them with at least one argument, the pathname of the shared library. If you have an existing handle to an already loaded shared library, it can be passed as the handle named parameter, otherwise the underlying platforms dlopen or LoadLibrary function is used to load the library into the process, and to get a handle to it.

As a result CDLL(None) is asking to open the shared library None, None is synonymous with NULL in this case, which means the call looks like the C code dlopen(NULL);. See dlopen(3) for details, the important bit being:

If filename is NULL, then the returned handle is for the main program.

As a result, lib = ctypes.CDLL(None) puts a handle to the main program into lib, and because Python is linked against libc the symbol free(3) is available, and so free(0x1) is just calling the libc free function with argument 0x1, which is segmentation fault territory...

@stuartarchibald
Copy link
Contributor

OK! Thanks for the example I don't think I would have figured out how to use gdb. Here is the output of the backtrace:

Thanks for this, this is a bit strange, it looks like it's following the right path to open the libsvml.so library but then gets in to trouble. I don't suppose those system packages that when removed magically "fixed it" are causing libc up/down-grades?

@ryan-chien
Copy link
Author

ryan-chien commented Nov 13, 2020 •
edited

Maybe so. Looks like libgdal-dev installs libc-dev-bin, linux-libc-dev, and updates libc6. Are these related to libsvml.so?

(numba-stats) root@a1b1511da97c:/workspaces/numba-stats# apt-get install libgdal-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  autotools-dev binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-7 default-libmysqlclient-dev dpkg-dev fakeroot g++ g++-7 gcc gcc-7 gcc-7-base gdal-data
  gir1.2-harfbuzz-0.0 hdf5-helpers icu-devtools libaec-dev libaec0 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libarmadillo-dev libarmadillo8 libarpack2
  libarpack2-dev libasan4 libatomic1 libbinutils libblas-dev libblas3 libboost-dev libboost1.65-dev libc-dev-bin libc6 libc6-dev libcc1-0 libcharls-dev libcharls1 libcilkrts5
  libcurl4-gnutls-dev libdap-dev libdap25 libdapclient6v5 libdapserver7v5 libdpkg-perl libepsilon-dev libepsilon1 libexpat1-dev libfakeroot libfile-fcntllock-perl libfreexl-dev libfreexl1
  libfyba-dev libfyba0 libgcc-7-dev libgdal20 libgeotiff-dev libgeotiff2 libgfortran4 libgif-dev libglib2.0-bin libglib2.0-dev libglib2.0-dev-bin libgomp1 libgraphite2-3 libgraphite2-dev
  libharfbuzz-dev libharfbuzz-gobject0 libharfbuzz-icu0 libharfbuzz0b libhdf4-0-alt libhdf4-alt-dev libhdf5-100 libhdf5-cpp-100 libhdf5-dev libicu-dev libicu-le-hb-dev libicu-le-hb0 libiculx60
  libisl19 libitm1 libjbig-dev libjbig0 libjpeg-dev libjpeg-turbo8-dev libjpeg8-dev libjson-c-dev libjson-c3 libkml-dev libkmlbase1 libkmlconvenience1 libkmldom1 libkmlengine1 libkmlregionator1
  libkmlxsd1 liblapack-dev liblapack3 liblsan0 libltdl-dev libltdl7 liblzma-dev libminizip-dev libminizip1 libmpc3 libmpfr6 libmpx2 libmysqlclient-dev libmysqlclient20 libnetcdf-dev libnetcdf13
  libodbc1 libogdi-dev libogdi4.1 libopenjp2-7 libopenjp2-7-dev libpcre16-3 libpcre3-dev libpcre32-3 libpcrecpp0v5 libpng-dev libpng-tools libpoppler-dev libpoppler-private-dev libpoppler73
  libpq-dev libpq5 libqhull-dev libqhull-r7 libqhull7 libquadmath0 libspatialite-dev libspatialite7 libsqlite3-dev libssl-dev libstdc++-7-dev libsuperlu-dev libsuperlu5 libsz2 libtiff-dev
  libtiff5 libtiff5-dev libtiffxx5 libtool libtsan0 libubsan0 liburiparser-dev liburiparser1 libwebp-dev libwebp6 libwebpdemux2 libwebpmux3 libxerces-c-dev libxerces-c3.2 libxml2-dev
  libzstd-dev linux-libc-dev make manpages manpages-dev mysql-common odbcinst odbcinst1debian2 pkg-config poppler-data proj-bin python3-distutils python3-lib2to3 unixodbc-dev uuid-dev
  zlib1g-dev
Suggested packages:
  binutils-doc cpp-doc gcc-7-locales debian-keyring g++-multilib g++-7-multilib gcc-7-doc libstdc++6-7-dbg gcc-multilib autoconf automake flex bison gdb gcc-doc gcc-7-multilib libgcc1-dbg
  libgomp1-dbg libitm1-dbg libatomic1-dbg libasan4-dbg liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libmpx2-dbg libquadmath0-dbg libitpp-dev liblapack-doc libboost-doc
  libboost1.65-doc libboost-atomic1.65-dev libboost-chrono1.65-dev libboost-container1.65-dev libboost-context1.65-dev libboost-coroutine1.65-dev libboost-date-time1.65-dev
  libboost-exception1.65-dev libboost-fiber1.65-dev libboost-filesystem1.65-dev libboost-graph1.65-dev libboost-graph-parallel1.65-dev libboost-iostreams1.65-dev libboost-locale1.65-dev
  libboost-log1.65-dev libboost-math1.65-dev libboost-mpi1.65-dev libboost-mpi-python1.65-dev libboost-numpy1.65-dev libboost-program-options1.65-dev libboost-python1.65-dev
  libboost-random1.65-dev libboost-regex1.65-dev libboost-serialization1.65-dev libboost-signals1.65-dev libboost-stacktrace1.65-dev libboost-system1.65-dev libboost-test1.65-dev
  libboost-thread1.65-dev libboost-timer1.65-dev libboost-type-erasure1.65-dev libboost-wave1.65-dev libboost1.65-tools-dev libmpfrc++-dev libntl-dev glibc-doc libcurl4-doc libgnutls28-dev
  libidn11-dev libkrb5-dev libldap2-dev librtmp-dev libssh2-1-dev bzr libgdal-doc libgeotiff-epsg geotiff-bin gdal-bin libglib2.0-doc libgraphite2-utils libhdf4-doc hdf4-tools libhdf5-doc
  icu-doc libtool-doc liblzma-doc netcdf-bin netcdf-doc libmyodbc odbc-postgresql tdsodbc unixodbc-bin ogdi-bin libfreetype6-dev postgresql-doc-10 sqlite3-doc libssl-doc libstdc++-7-doc
  libsuperlu-doc automaken gfortran | fortran95-compiler gcj-jdk libxerces-c-doc make-doc man-browser poppler-utils ghostscript fonts-japanese-mincho | fonts-ipafont-mincho
  fonts-japanese-gothic | fonts-ipafont-gothic fonts-arphic-ukai fonts-arphic-uming fonts-nanum
The following NEW packages will be installed:
  autotools-dev binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-7 default-libmysqlclient-dev dpkg-dev fakeroot g++ g++-7 gcc gcc-7 gcc-7-base gdal-data
  gir1.2-harfbuzz-0.0 hdf5-helpers icu-devtools libaec-dev libaec0 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libarmadillo-dev libarmadillo8 libarpack2
  libarpack2-dev libasan4 libatomic1 libbinutils libblas-dev libblas3 libboost-dev libboost1.65-dev libc-dev-bin libc6-dev libcc1-0 libcharls-dev libcharls1 libcilkrts5 libcurl4-gnutls-dev
  libdap-dev libdap25 libdapclient6v5 libdapserver7v5 libdpkg-perl libepsilon-dev libepsilon1 libexpat1-dev libfakeroot libfile-fcntllock-perl libfreexl-dev libfreexl1 libfyba-dev libfyba0
  libgcc-7-dev libgdal-dev libgdal20 libgeotiff-dev libgeotiff2 libgfortran4 libgif-dev libglib2.0-bin libglib2.0-dev libglib2.0-dev-bin libgomp1 libgraphite2-3 libgraphite2-dev libharfbuzz-dev
  libharfbuzz-gobject0 libharfbuzz-icu0 libharfbuzz0b libhdf4-0-alt libhdf4-alt-dev libhdf5-100 libhdf5-cpp-100 libhdf5-dev libicu-dev libicu-le-hb-dev libicu-le-hb0 libiculx60 libisl19 libitm1
  libjbig-dev libjbig0 libjpeg-dev libjpeg-turbo8-dev libjpeg8-dev libjson-c-dev libjson-c3 libkml-dev libkmlbase1 libkmlconvenience1 libkmldom1 libkmlengine1 libkmlregionator1 libkmlxsd1
  liblapack-dev liblapack3 liblsan0 libltdl-dev libltdl7 liblzma-dev libminizip-dev libminizip1 libmpc3 libmpfr6 libmpx2 libmysqlclient-dev libmysqlclient20 libnetcdf-dev libnetcdf13 libodbc1
  libogdi-dev libogdi4.1 libopenjp2-7 libopenjp2-7-dev libpcre16-3 libpcre3-dev libpcre32-3 libpcrecpp0v5 libpng-dev libpng-tools libpoppler-dev libpoppler-private-dev libpoppler73 libpq-dev
  libpq5 libqhull-dev libqhull-r7 libqhull7 libquadmath0 libspatialite-dev libspatialite7 libsqlite3-dev libssl-dev libstdc++-7-dev libsuperlu-dev libsuperlu5 libsz2 libtiff-dev libtiff5
  libtiff5-dev libtiffxx5 libtool libtsan0 libubsan0 liburiparser-dev liburiparser1 libwebp-dev libwebp6 libwebpdemux2 libwebpmux3 libxerces-c-dev libxerces-c3.2 libxml2-dev libzstd-dev
  linux-libc-dev make manpages manpages-dev mysql-common odbcinst odbcinst1debian2 pkg-config poppler-data proj-bin python3-distutils python3-lib2to3 unixodbc-dev uuid-dev zlib1g-dev
The following packages will be upgraded:
  libc6
1 upgraded, 173 newly installed, 0 to remove and 1 not upgraded.
Need to get 113 MB of archives.
After this operation, 590 MB of additional disk space will be used.
Do you want to continue? [Y/n] n

@stuartarchibald
Copy link
Contributor

Maybe so. Looks like libgdal-dev installs libc-dev-bin, linux-libc-dev, and updates libc6. Are these related to libsvml.so?

Well, I think they are related in that libsvml.so also links against libc. It also looks like there's a load of toolchain updating going on too, without being able to reproduce, working out precisely what the issue is could well be challenging. libc itself should be backwards compatible as the symbols are versioned, but I suspect something else is going on too. I'm also now not convinced this is a Numba specific issue. With libgdal-dev installed, if you do this:

$ python -c "import ctypes; print(ctypes.CDLL('libsvml.so'))"

what happens?

@ryan-chien
Copy link
Author 

(numba-stats) root@4b3b3e43f04e:/workspaces/numba-stats# python -c "import ctypes; print(ctypes.CDLL('libsvml.so'))"
Segmentation fault

Thanks for all of the effort looking into this. I think we can close the issue if you cannot reproduce. From my side I don't need libgdal-dev - it is a vestigial part of the Dockerfile.

@stuartarchibald
Copy link
Contributor 

(numba-stats) root@4b3b3e43f04e:/workspaces/numba-stats# python -c "import ctypes; print(ctypes.CDLL('libsvml.so'))"
Segmentation fault

Thanks for all of the effort looking into this. I think we can close the issue if you cannot reproduce. From my side I don't need libgdal-dev - it is a vestigial part of the Dockerfile.

No problem, I think that confirms it's unlikely a Numba problem. Suggest reporting to ubuntu upstream. Will close as suggested. Thanks for your efforts in debugging this!

@stuartarchibald stuartarchibald added no action required No action was needed to resolve. and removed needtriage labels Nov 13, 2020
@mfoglio
Copy link

mfoglio commented Jun 10, 2021

Did you find a solution for this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
no action required No action was needed to resolve.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@stuartarchibald @mfoglio @ryan-chien