pypesto import is slow

[dweindl]

python -c "import pypesto" takes 7s on my computer.
python -c "import scipy" takes 0.5s on my computer.

I think we should think more carefully about what is imported in pypesto/__init__.py

[dweindl]

import pypesto currently imports all of those packages:

amici
asttokens
attr
backcall
cffi
cloudpickle
cma
colorama
cyipopt
dateutil
defusedxml
_distutils_hack
dlib
emcee
executing
fides
h5py
idna
IPython
jedi
joblib
jsonschema
kiwisolver
libsbml
llvmlite
matplotlib
mpl_toolkits
mpmath
networkx
nlopt
numba
numpy
packaging
pandas
parso
petab
pexpect
PIL
pkg_resources
prompt_toolkit
psutil
ptyprocess
pure_eval
pycparser
pygments
pynndescent
pyparsing
pyrsistent
pysb
pyswarm
pyswarms
pytz
scipy
seaborn
setuptools
sklearn
stack_data
sympy
tqdm
traitlets
umap
wcwidth
yaml

[yannikschaelte]

that's ofc bad. thought we had fixed that already, but how do all the optimizers and samplers come in again?

[yannikschaelte]

a big chunk comes from pypesto.visualize

[yannikschaelte]

ok .visualize internally refers to at least .sample and .profile (which in turn refers to .optimize) for utility methods. Thus, we need to either

• remove .visualize from standard import (then need to manually import it each time it's used; but this keeps the long import time then);
• or disentangle those utility functions from the heavy submodule imports;
• or lazily import each heavy import only when it's really used (not sure how to tho)

[dweindl]

.visualize internally refers to at least .sample and .profile (which in turn refers to .optimize) for utility methods.

Thanks for checking.

or disentangle those utility functions from the heavy submodule imports;

👍

I didn't check in detail, but I guess a big part can be resolved by moving some imports under if TYPE_CHECKING and importing the expensive things only inside individual functions.

[yannikschaelte]

I guess actually importing pymc, julia, emcee, nlopt etc only at the beginning of the respective classes' __init__methods should be both fine and canonical (i.e. only entry point), so that sounds like a feasible solution which would also allow us to keep default imports without requiring to manually import everything.

[yannikschaelte]

Reduced in #946 from 19s to 3.5s on my system. Further improvements could be gained from not importing .visualize by default (2s), and not auto-importing amici+petab, which are main culprits according to

pynndescent 3,923
umap 3,831
seaborn 1,372
petab 0,851
amici 0,835
cma 0,610
pysb 0,505
sklearn 0,451
emcee 0,443
pandas 0,399
numba 0,354
fides 0,302
cyipopt 0,287
pyswarms 0,279
sympy 0,266
IPython 0,258
matplotlib 0,230
joblib 0,180
nlopt 0,155
h5py 0,153
setuptools 0,148
pkg_resources 0,146
scipy 0,140
libsbml 0,138
numpy 0,136
pyswarm 0,130
prompt_toolkit 0,110
networkx 0,105
jedi 0,094
jsonschema 0,089
pyparsing 0,059
parso 0,048
psutil 0,042
mpmath 0,042
stack_data 0,040
attr 0,040
pycparser 0,039
cloudpickle 0,035
pyrsistent 0,034
yaml 0,033
tqdm 0,032
pure_eval 0,032
dlib 0,032
cffi 0,032
_distutils_hack 0,031
pexpect 0,030
packaging 0,030
ptyprocess 0,027
llvmlite 0,027
kiwisolver 0,027
executing 0,027
backcall 0,027
wcwidth 0,026
traitlets 0,026
PIL 0,026
pytz 0,025
defusedxml 0,025
asttokens 0,025
idna 0,023
colorama 0,023
pygments 0,022
mpl_toolkits 0,022
dateutil 0,020

[dweindl]

Reduced in #946 from 19s to 3.5s on my system.

Thanks. Sounds much better already.

Further improvements could be gained from not importing .visualize by default (2s), and not auto-importing amici+petab, which are main culprits according to

In favor of not importing those by default.

[yannikschaelte]

So you would also not import amici+petab by default? Then there'd be two options: Either not auto-import e.g. pypesto.objective.amici when importing from pypesto.objective (and pypesto.petab, and some tidy-ups in pypesto.ensemble may be needed), or moving each import to lazy level in-class/function. (1) would require some user code adaptations, (2) would look a bit nasty. Preferences?

[dweindl]

So you would also not import amici+petab by default?

Yes, I think it's fair to require from pypesto.petab import PetabImporter, as pypesto's focus is not necessarily PEtab.

[yannikschaelte]

For the petab submodule that's already the case, it would rather affect from pypesto.objective.amici import AmiciObjective

[dweindl]

For the petab submodule that's already the case, it would rather affect from pypesto.objective.amici import AmiciObjective

amici should not be imported by default, only when importing the PetabImporter.

With the current state if #946, import takes about 3s for me. Much better, but removing amici (1.5s import time) would be preferable.

[dweindl]

v0.2.14