Takes 0.8 seconds to import tree_sitter

[MrPrezident]

I'm seeing exceptionally long load time on importing the tree_sitter module. This is more than doubling the load time of my app. Is this expected?

import time

start_time = time.time()
import tree_sitter
end_time = time.time()

tree_sitter_elapsed_time = end_time - start_time
print(f"It took {tree_sitter_elapsed_time} seconds to import tree_sitter")

output:

It took 0.8178912149742246 seconds to import tree_sitter

[amaanq]

Something's off in your system, same code for me:

It took 0.0004069805145263672 seconds to import tree_sitter

Are you on Linux? Can you use perf to profile it and upolad the data here?

[narpfel]

There’s also python -Ximporttime to profile import times (or importtime-waterfall to get a graphical output), for me importing tree_sitter takes 140 ms, 130ms of which are spent in importing distutils:

$ python -Ximporttime -c 'import tree_sitter'
import time: self [us] | cumulative | imported package
[...]
import time:       613 |        613 |         json.encoder
import time:       347 |       2441 |       json
import time:      2406 |     129771 |     distutils
import time:       513 |     130284 |   distutils.ccompiler
import time:       444 |        444 |     shlex
import time:       176 |        176 |       distutils.py39compat
import time:       168 |        168 |       distutils._functools
import time:       927 |       1269 |     distutils.sysconfig
import time:       168 |        168 |     distutils._macos_compat
import time:       361 |       2240 |   distutils.unixccompiler
import time:       653 |        653 |   tree_sitter.binding
import time:       871 |     140812 | tree_sitter

Unrelated to this issue, I think it’s probably worthwhile to move the distutils import into Language.build_library to only pay for the import when it’s actually needed.

[MrPrezident]

Nice! I didn't know about importtime. I'm running this on linux (SLES12). I did some further debugging. It turns out that I had my PYTHONPATH set to point directly to the site packages (which are on NFS), however the tree_sitter module was only installed in my venv (also on NFS). That was the only module that was in my local venv and not in site packages which is why I did not see a similar time penalty for any of the other modules I'm using. Python was still able to find tree_sitter, but I guess using PYTHONPATH caused a huge time penalty.

Here is the import time with the bad PYTHONPATH:

[...]
import time:       419 |        419 |         json.encoder
import time:       296 |       1698 |       json
import time:      2723 |     827802 |     distutils
import time:      1107 |     828909 |   distutils.ccompiler
import time:       314 |        314 |     shlex
import time:       808 |        808 |       distutils.py39compat
import time:       665 |        665 |       distutils._functools
import time:      1197 |       2670 |     distutils.sysconfig
import time:       730 |        730 |     distutils._macos_compat
import time:      1078 |       4792 |   distutils.unixccompiler
import time:       932 |        932 |   tree_sitter.binding
import time:      1825 |     840382 | tree_sitter`

Here it is without PYTHONPATH being set:

[...]
import time:       373 |        373 |         json.encoder
import time:       258 |       1510 |       json
import time:      4514 |     157710 |     distutils
import time:      1198 |     158908 |   distutils.ccompiler
import time:       315 |        315 |     shlex
import time:       931 |        931 |       distutils.py39compat
import time:       804 |        804 |       distutils._functools
import time:      1400 |       3133 |     distutils.sysconfig
import time:       827 |        827 |     distutils._macos_compat
import time:      1528 |       5801 |   distutils.unixccompiler
import time:      1314 |       1314 |   tree_sitter.binding
import time:      4009 |     173965 | tree_sitter