scalene breaks when importing pyproj

[aymondebroglie]

Describe the bug
I am trying to profile code where we import the pyproj library, and scalene fails with Scalene error: received signal SIGSEGV

To Reproduce

• Install the pyproj library
• create a file with the following line from pyproj import Proj
• run scalene my_file.py

Scalene will fail with the error above

Expected behaviour
Scalene does not fail and I can profile like for any library !

[emeryberger]

I'm unable to reproduce this with the version in the repo, though that version has not been released yet. Can you give that a try?

[aymondebroglie]

Indeed, I built the current version in local and scalene does not fail, however it seems the behaviour is still not correct.

I tried profiling the following file

from pyproj import Proj
import time

time.sleep(1)
time.sleep(0.1)

and scalene ends with Scalene: Program did not run for long enough to profile.
However if I remove the pyrpoj import line everything works correctly

[emeryberger]

Works for me:

 % python3 -m scalene testpyproj.py 
                              Memory usage: ▅▅▆▅▆▆▆▇▇▇▇█▇▇████████ (max:  26.57MB, growth rate:  27%)                             
                                        testpyproj.py: % of time = 100.00% out of   1.15s.                                        
         ╷       ╷       ╷     ╷       ╷      ╷              ╷       ╷                                                            
    Line │Time % │Time % │Sys  │Mem %  │Net   │Memory usage  │Copy   │                                                            
         │Python │native │%    │Python │(MB)  │over time / % │(MB/s) │testpyproj.py                                               
 ╺━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━┿━━━━━━━┿━━━━━━┿━━━━━━━━━━━━━━┿━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸
       1 │    4% │    0% │     │   93% │   24 │▇███100%      │     2 │from pyproj import Proj                                     
       2 │       │       │     │       │      │              │       │import time                                                 
       3 │       │       │     │       │      │              │       │                                                            
       4 │   88% │       │ 88% │       │      │              │       │time.sleep(1)                                               
       5 │    7% │       │  7% │       │      │              │       │time.sleep(0.1)                                             
         ╵       ╵       ╵     ╵       ╵      ╵              ╵       ╵                                                            
 Top net memory consumption, by line:
 (1)     1:    24 MB

[aymondebroglie]

hmm this is weird , I'm on te latest version of pyproj as well so I don't see why it does not work

[emeryberger]

Can you give me more info about your system? Are you using the latest version of Scalene? (python3 -m pip install -U scalene)

[aymondebroglie]

Scalene version

I cloned the scalene repo in local , i am on commit a85e8b1bc49e58eb7a0ed0151718afe028914c77
I ran python setup.py develop

Pyproj version

3.0.1

Python version

Python 3.8.8

I am on macOSX version 10.15.7 (19H524)

To note that on this commit of scalene I now have Scalene error: received signal SIGSEGV error back

[emeryberger]

Runs cleanly on Mac OS X with the same commit version, same Pyproj version, and same Python version. The only difference is that I am running on an M1 with a much more recent version of Mac OS X (11.2.3, Big Sur). You might try updating Heap-Layers to see if it helps (long shot) -- cd Heap-Layers && git pull -- and then rebuild.

[aymondebroglie]

I tried but it didn't work. For reference I reproduced the problem on a minimal environment with just scalene and pyproj installed

# Name                    Version                   Build  Channel
c-ares                    1.17.1               h0d85af4_1    conda-forge
ca-certificates           2020.12.5            h033912b_0    conda-forge
certifi                   2020.12.5        py38h50d1736_1    conda-forge
cloudpickle               1.6.0                    pypi_0    pypi
colorama                  0.4.4                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
jpeg                      9d                   hbcb3906_0    conda-forge
krb5                      1.17.2               h60d9502_0    conda-forge
libcurl                   7.75.0               h8ef9fac_0    conda-forge
libcxx                    11.1.0               habf9029_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libev                     4.33                 haf1e3a3_1    conda-forge
libffi                    3.3                  h046ec9c_2    conda-forge
libnghttp2                1.43.0               h07e645a_0    conda-forge
libssh2                   1.9.0                h52ee1ee_6    conda-forge
libtiff                   4.2.0                h355d032_0    conda-forge
libwebp-base              1.2.0                h0d85af4_2    conda-forge
lz4-c                     1.9.3                h046ec9c_0    conda-forge
ncurses                   6.2                  h2e338ed_4    conda-forge
numpy                     1.20.1                   pypi_0    pypi
openssl                   1.1.1j               hbcf498f_0    conda-forge
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
proj                      8.0.0                h1512c50_0    conda-forge
pygments                  2.8.1                    pypi_0    pypi
pyproj                    3.0.1            py38h87d968f_1    conda-forge
python                    3.8.8           h4e93d89_0_cpython    conda-forge
python_abi                3.8                      1_cp38    conda-forge
readline                  8.0                  h0678c8f_2    conda-forge
rich                      9.13.0                   pypi_0    pypi
scalene                   1.3.0                     dev_0    <develop>
setuptools                49.6.0           py38h50d1736_3    conda-forge
sqlite                    3.34.0               h17101e1_0    conda-forge
tk                        8.6.10               h0419947_1    conda-forge
typing-extensions         3.7.4.3                  pypi_0    pypi
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge
zstd                      1.4.9                h582d3a0_0    conda-forge

[sternj]

Successfully replicated on x86 mac, I'll get to the bottom of it!

[sternj]

This seems to be a more fundamental issue with how Scalene interacts with sqlite3, the way it messes with function pointers at runtime makes it difficult to debug! It also seems to be unique to the macos build, which narrows it down further, so I'm working with the homebrew formula.

[emeryberger]

FWIW @sternj I just tried this on Mac OS X x86 with the current repo version and Python 3.7, and it works fine (Ubuntu 16). @aymondebroglie can you try installing it from the repo?

python3 -m pip install git+https://github.com/plasma-umass/scalene

[aymondebroglie]

@emeryberger I tried and I still have the same issue Scalene error: received signal SIGSEGV

[jaltmayerpizzorno]

Could you post the backtrace you obtain with the Console application (within Utilities), looking for crash reports?
(Sorry, don’t have access to a mac just now to ask more exactly).

[sternj]

We have it tracked down to an issue with malloc interposition on Mac interacting with sqlite3

[sternj]

(I can send you some more details-- it is acting like an issue with HeapLayers rather than with Scalene itself)

[emeryberger]

I believe this may actually be solved (!). @aymondebroglie if you can reach out to us on Discord, that would be great! https://discord.com/invite/6Dtaf6HW3m

[emeryberger]

Confirmed fixed by @aymondebroglie - thanks!