Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PERF: pandas takes 8-9 seconds to import in a new environment #53488

Open
2 of 3 tasks
pyrito opened this issue Jun 1, 2023 · 9 comments
Open
2 of 3 tasks

PERF: pandas takes 8-9 seconds to import in a new environment #53488

pyrito opened this issue Jun 1, 2023 · 9 comments
Labels
Build Library building on various platforms Internals Related to non-user accessible pandas implementation Performance Memory or execution speed performance

Comments

@pyrito
Copy link
Contributor

pyrito commented Jun 1, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this issue exists on the latest version of pandas.

  • I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

I did the following steps:

  1. conda create --name py39-repro python=3.9
  2. conda activate py39-repro
  3. pip install ipython; pip install pandas
  4. ipython
In [1]: %time import pandas
CPU times: user 277 ms, sys: 146 ms, total: 423 ms
Wall time: 8.67 s

Installed Versions

INSTALLED VERSIONS

commit : 965ceca
python : 3.9.16.final.0
python-bits : 64
OS : Darwin
OS-release : 21.3.0
Version : Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.2
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.8.0
pip : 23.0.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.13.2
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : 2023.4.0
fsspec : 2023.5.0
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

Prior Performance

No response
@pyrito pyrito added Needs Triage Issue that has not been reviewed by a pandas team member Performance Memory or execution speed performance labels Jun 1, 2023
@mvashishtha mvashishtha mentioned this issue Jun 1, 2023
Open
@lithomas1
Copy link
Member

Can you paste output of python -X importtime -c "import pandas as pd"?

@lithomas1 lithomas1 added Build Library building on various platforms Needs Info Clarification about behavior needed to assess issue and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 1, 2023
@pyrito
Copy link
Contributor Author

pyrito commented Jun 1, 2023

@lithomas1 sure thing!

import time: self [us] | cumulative | imported package
import time:       473 |        473 |   _io
import time:        43 |         43 |   marshal
import time:       576 |        576 |   posix
import time:       827 |       1917 | _frozen_importlib_external
import time:       689 |        689 |   time
import time:       325 |       1014 | zipimport
import time:        60 |         60 |     _codecs
import time:       517 |        577 |   codecs
import time:       809 |        809 |   encodings.aliases
import time:      1070 |       2455 | encodings
import time:       293 |        293 | encodings.utf_8
import time:       109 |        109 | _signal
import time:       396 |        396 | encodings.latin_1
import time:        45 |         45 |     _abc
import time:       286 |        331 |   abc
import time:       320 |        650 | io
import time:        54 |         54 |       _stat
import time:       197 |        250 |     stat
import time:      1136 |       1136 |     _collections_abc
import time:       187 |        187 |       genericpath
import time:       255 |        441 |     posixpath
import time:       576 |       2402 |   os
import time:       200 |        200 |   _sitebuiltins
import time:        64 |         64 |     _locale
import time:       185 |        248 |   _bootlocale
import time:       593 |        593 |   _distutils_hack
import time:        81 |         81 |   sitecustomize
import time:        63 |         63 |   usercustomize
import time:      1260 |       4844 | site
import time:       375 |        375 |   __future__
import time:       266 |        266 |     warnings
import time:       332 |        332 |         types
import time:       789 |       1120 |       enum
import time:       327 |       1447 |     numpy._globals
import time:       137 |        137 |     numpy.__config__
import time:        86 |         86 |     numpy._distributor_init
import time:        57 |         57 |                   _sre
import time:       318 |        318 |                     sre_constants
import time:       831 |       1149 |                   sre_parse
import time:       315 |       1520 |                 sre_compile
import time:       907 |        907 |                       _heapq
import time:       197 |       1104 |                     heapq
import time:        99 |         99 |                     itertools
import time:       231 |        231 |                     keyword
import time:        60 |         60 |                       _operator
import time:       313 |        372 |                     operator
import time:       188 |        188 |                     reprlib
import time:        51 |         51 |                     _collections
import time:       727 |       2770 |                   collections
import time:        40 |         40 |                   _functools
import time:       814 |       3623 |                 functools
import time:       311 |        311 |                 copyreg
import time:       572 |       6024 |               re
import time:       960 |        960 |                 _json
import time:       528 |       1488 |               json.scanner
import time:       396 |       7907 |             json.decoder
import time:       538 |        538 |             json.encoder
import time:       211 |       8655 |           json
import time:        86 |       8741 |         numpy._version
import time:       184 |       8924 |       numpy.version
import time:       730 |        730 |               math
import time:       661 |        661 |               _datetime
import time:      1299 |       2689 |             datetime
import time:   5012430 |    5015119 |           numpy.core._multiarray_umath
import time:       364 |        364 |               numpy.compat._inspect
import time:       404 |        404 |                   fnmatch
import time:        86 |         86 |                     nt
import time:        63 |         63 |                     nt
import time:        57 |         57 |                     nt
import time:        57 |         57 |                     nt
import time:       578 |        840 |                   ntpath
import time:        56 |         56 |                   errno
import time:       303 |        303 |                     urllib
import time:      1155 |       1457 |                   urllib.parse
import time:      1275 |       4029 |                 pathlib
import time:        78 |         78 |                 pickle5
import time:       680 |        680 |                     _struct
import time:       297 |        976 |                   struct
import time:       440 |        440 |                   _compat_pickle
import time:      1041 |       1041 |                   _pickle
import time:        81 |         81 |                       org
import time:        16 |         96 |                     org.python
import time:        13 |        109 |                   org.python.core
import time:       925 |       3488 |                 pickle
import time:       224 |       7817 |               numpy.compat.py3k
import time:       413 |       8593 |             numpy.compat
import time:        18 |       8611 |           numpy.compat._inspect
import time:       278 |    5024007 |         numpy.core.overrides
import time:       529 |    5024535 |       numpy.core.multiarray
import time:       418 |        418 |       numpy.core.umath
import time:       577 |        577 |         numbers
import time:       268 |        268 |         numpy.core._string_helpers
import time:       196 |        196 |           numpy.core._dtype
import time:       604 |        800 |         numpy.core._type_aliases
import time:       514 |       2158 |       numpy.core.numerictypes
import time:       583 |        583 |               contextlib
import time:       287 |        287 |               numpy.core._exceptions
import time:       297 |        297 |                 collections.abc
import time:       700 |        700 |                   _contextvars
import time:       276 |        976 |                 contextvars
import time:       309 |       1581 |               numpy.core._ufunc_config
import time:       278 |       2728 |             numpy.core._methods
import time:       866 |       3594 |           numpy.core.fromnumeric
import time:       401 |       3994 |         numpy.core.shape_base
import time:       745 |        745 |         numpy.core.arrayprint
import time:       205 |        205 |         numpy.core._asarray
import time:       926 |       5868 |       numpy.core.numeric
import time:       998 |        998 |       numpy.core.defchararray
import time:       477 |        477 |       numpy.core.records
import time:       304 |        304 |       numpy.core.memmap
import time:       324 |        324 |       numpy.core.function_base
import time:       173 |        173 |       numpy.core._machar
import time:       268 |        268 |       numpy.core.getlimits
import time:       334 |        334 |       numpy.core.einsumfunc
import time:     36731 |      36731 |         numpy.core._multiarray_tests
import time:      1252 |      37983 |       numpy.core._add_newdocs
import time:       491 |        491 |       numpy.core._add_newdocs_scalars
import time:       172 |        172 |       numpy.core._dtype_ctypes
import time:       660 |        660 |           _ast
import time:      1044 |       1703 |         ast
import time:      2176 |       2176 |           _ctypes
import time:       469 |        469 |           ctypes._endian
import time:      1124 |       3768 |         ctypes
import time:       850 |       6320 |       numpy.core._internal
import time:       366 |        366 |       numpy._pytesttester
import time:       496 |    5090603 |     numpy.core
import time:       401 |        401 |       numpy.lib.mixins
import time:       319 |        319 |           numpy.lib.ufunclike
import time:      1090 |       1408 |         numpy.lib.type_check
import time:       627 |       2035 |       numpy.lib.scimath
import time:       394 |        394 |                   numpy.lib.stride_tricks
import time:       362 |        756 |                 numpy.lib.twodim_base
import time:     50178 |      50178 |                 numpy.linalg._umath_linalg
import time:       868 |      51802 |               numpy.linalg.linalg
import time:       234 |      52035 |             numpy.linalg
import time:       423 |      52457 |           numpy.matrixlib.defmatrix
import time:       222 |      52678 |         numpy.matrixlib
import time:       509 |        509 |           numpy.lib.histograms
import time:      1174 |       1683 |         numpy.lib.function_base
import time:       475 |      54836 |       numpy.lib.index_tricks
import time:       601 |        601 |       numpy.lib.nanfunctions
import time:       459 |        459 |       numpy.lib.shape_base
import time:       635 |        635 |       numpy.lib.polynomial
import time:      1022 |       1022 |         textwrap
import time:       671 |       1693 |       numpy.lib.utils
import time:       400 |        400 |       numpy.lib.arraysetops
import time:       344 |        344 |           _weakrefset
import time:       589 |        933 |         weakref
import time:       421 |        421 |         numpy.lib.format
import time:       341 |        341 |         numpy.lib._datasource
import time:       468 |        468 |         numpy.lib._iotools
import time:       593 |       2754 |       numpy.lib.npyio
import time:       298 |        298 |       numpy.lib.arrayterator
import time:       331 |        331 |       numpy.lib.arraypad
import time:       196 |        196 |       numpy.lib._version
import time:       569 |      65201 |     numpy.lib
import time:     72762 |      72762 |         numpy.fft._pocketfft_internal
import time:       519 |      73280 |       numpy.fft._pocketfft
import time:       461 |        461 |       numpy.fft.helper
import time:       335 |      74075 |     numpy.fft
import time:       387 |        387 |         numpy.polynomial.polyutils
import time:       438 |        438 |         numpy.polynomial._polybase
import time:       822 |       1647 |       numpy.polynomial.polynomial
import time:       520 |        520 |       numpy.polynomial.chebyshev
import time:       426 |        426 |       numpy.polynomial.legendre
import time:       352 |        352 |       numpy.polynomial.hermite
import time:       452 |        452 |       numpy.polynomial.hermite_e
import time:       565 |        565 |       numpy.polynomial.laguerre
import time:       426 |       4385 |     numpy.polynomial
import time:       128 |        128 |               backports_abc
import time:     51751 |      51878 |             numpy.random._common
import time:      1050 |       1050 |                 binascii
import time:       429 |       1479 |               base64
import time:      1596 |       1596 |                 _hashlib
import time:       460 |        460 |                   _blake2
import time:       446 |        906 |                 hashlib
import time:       413 |       2914 |               hmac
import time:       380 |        380 |                   _bisect
import time:       309 |        689 |                 bisect
import time:       391 |        391 |                 _random
import time:       385 |        385 |                 _sha512
import time:       695 |       2158 |               random
import time:       406 |       6955 |             secrets
import time:      1207 |       1207 |             threading
import time:     42577 |     102617 |           numpy.random.bit_generator
import time:     59999 |      59999 |           numpy.random._bounded_integers
import time:     32288 |      32288 |           numpy.random._mt19937
import time:     68548 |     263450 |         numpy.random.mtrand
import time:     31926 |      31926 |         numpy.random._philox
import time:     99334 |      99334 |         numpy.random._pcg64
import time:     31689 |      31689 |         numpy.random._sfc64
import time:     77153 |      77153 |         numpy.random._generator
import time:       348 |     503897 |       numpy.random._pickle
import time:       388 |     504285 |     numpy.random
import time:       689 |        689 |     numpy.ctypeslib
import time:       558 |        558 |               _opcode
import time:       389 |        947 |             opcode
import time:       650 |       1596 |           dis
import time:       368 |        368 |             importlib
import time:       318 |        685 |           importlib.machinery
import time:       509 |        509 |               token
import time:       901 |       1409 |             tokenize
import time:       289 |       1698 |           linecache
import time:      1408 |       5385 |         inspect
import time:      2491 |       7876 |       numpy.ma.core
import time:      1091 |       1091 |       numpy.ma.extras
import time:       385 |       9351 |     numpy.ma
import time:      3840 |    5754361 |   numpy
import time:       505 |        505 |     pytz.exceptions
import time:       305 |        305 |     pytz.lazy
import time:       775 |        775 |     pytz.tzinfo
import time:       197 |        197 |     pytz.tzfile
import time:       909 |       2690 |   pytz
import time:       292 |        292 |     dateutil._version
import time:       370 |        662 |   dateutil
import time:       878 |        878 |         signal
import time:        48 |         48 |         pwd
import time:       786 |        786 |         grp
import time:        83 |         83 |         msvcrt
import time:       598 |        598 |         _posixsubprocess
import time:       576 |        576 |         select
import time:       689 |        689 |         selectors
import time:       671 |       4326 |       subprocess
import time:      1694 |       6020 |     platform
import time:      1497 |       1497 |       typing
import time:      2195 |       3692 |     pandas._typing
import time:       372 |        372 |     pandas.compat._constants
import time:       314 |        314 |         _compression
import time:       811 |        811 |         _bz2
import time:       389 |       1512 |       bz2
import time:      1472 |       1472 |         _lzma
import time:      1133 |       2604 |       lzma
import time:       342 |       4458 |     pandas.compat.compressors
import time:     52707 |      52707 |                             pandas._libs.tslibs.np_datetime
import time:     56804 |     109511 |                           pandas._libs.tslibs.dtypes
import time:     48111 |      48111 |                             pandas._libs.tslibs.base
import time:     62200 |      62200 |                                 pandas._libs.tslibs.nattype
import time:       279 |        279 |                                       pandas.util._exceptions
import time:      2246 |       2246 |                                       pandas.util.version
import time:       373 |       2897 |                                     pandas.compat._optional
import time:       508 |        508 |                                         sysconfig
import time:       616 |        616 |                                         _sysconfigdata__darwin_darwin
import time:       557 |        557 |                                         _osx_support
import time:      1269 |       2949 |                                       zoneinfo._tzpath
import time:       337 |        337 |                                       zoneinfo._common
import time:       913 |        913 |                                       _zoneinfo
import time:       411 |       4609 |                                     zoneinfo
import time:       541 |        541 |                                             importlib.abc
import time:       359 |        900 |                                           importlib.util
import time:       998 |       1898 |                                         six
import time:        30 |         30 |                                         six.moves
import time:       258 |        258 |                                         dateutil.tz._common
import time:       246 |        246 |                                         dateutil.tz._factories
import time:        19 |         19 |                                           six.moves.winreg
import time:       299 |        317 |                                         dateutil.tz.win
import time:       750 |       3496 |                                       dateutil.tz.tz
import time:       304 |       3800 |                                     dateutil.tz
import time:     61336 |      72641 |                                   pandas._libs.tslibs.timezones
import time:     49927 |      49927 |                                     pandas._libs.tslibs.ccalendar
import time:      1116 |       1116 |                                     locale
import time:       635 |        635 |                                       calendar
import time:       898 |       1533 |                                     _strptime
import time:       962 |        962 |                                         pandas._config.config
import time:       394 |        394 |                                         pandas._config.dates
import time:       160 |        160 |                                         pandas._config.display
import time:       222 |       1737 |                                       pandas._config
import time:       161 |       1897 |                                     pandas._config.localization
import time:     68703 |     123174 |                                   pandas._libs.tslibs.fields
import time:     77791 |     273605 |                                 pandas._libs.tslibs.timedeltas
import time:     63053 |      63053 |                                 pandas._libs.tslibs.tzconversion
import time:       179 |        179 |                                 backports_abc
import time:     77810 |     476844 |                               pandas._libs.tslibs.timestamps
import time:        79 |         79 |                               backports_abc
import time:       358 |        358 |                               dateutil.easter
import time:       304 |        304 |                                 dateutil._common
import time:       431 |        734 |                               dateutil.relativedelta
import time:     54950 |      54950 |                               pandas._libs.properties
import time:    107338 |     640302 |                             pandas._libs.tslibs.offsets
import time:       138 |        138 |                               backports_abc
import time:      1585 |       1585 |                                 _decimal
import time:       384 |       1969 |                               decimal
import time:        39 |         39 |                                     _string
import time:       782 |        821 |                                   string
import time:       775 |       1595 |                                 dateutil.parser._parser
import time:       476 |        476 |                                 dateutil.parser.isoparser
import time:       316 |       2385 |                               dateutil.parser
import time:     63363 |      63363 |                               pandas._libs.tslibs.strptime
import time:     74877 |     142731 |                             pandas._libs.tslibs.parsing
import time:     60596 |     891738 |                           pandas._libs.tslibs.conversion
import time:     67615 |      67615 |                           pandas._libs.tslibs.period
import time:     58137 |      58137 |                           pandas._libs.tslibs.vectorized
import time:       482 |    1127481 |                         pandas._libs.tslibs
import time:        35 |    1127515 |                       pandas._libs.tslibs.nattype
import time:     52422 |      52422 |                       pandas._libs.ops_dispatch
import time:     60445 |    1240381 |                     pandas._libs.missing
import time:    141722 |    1382102 |                   pandas._libs.hashtable
import time:    175403 |     175403 |                   pandas._libs.algos
import time:    113002 |    1670507 |                 pandas._libs.interval
import time:       211 |    1670717 |               pandas._libs
import time:        22 |    1670738 |             pandas._libs.properties
import time:       406 |    1671144 |           pandas.util._decorators
import time:       261 |        261 |               pandas.core
import time:       239 |        499 |             pandas.core.util
import time:     76527 |      76527 |             pandas._libs.lib
import time:     58404 |      58404 |             pandas._libs.hashing
import time:       221 |        221 |               pandas.core.dtypes
import time:       556 |        556 |                 pandas.errors
import time:       322 |        322 |                 pandas.core.dtypes.generic
import time:       389 |       1267 |               pandas.core.dtypes.base
import time:       187 |        187 |                 pandas.core.dtypes.inference
import time:      1047 |       1234 |               pandas.core.dtypes.dtypes
import time:       452 |       3172 |             pandas.core.dtypes.common
import time:       259 |     138858 |           pandas.core.util.hashing
import time:       252 |    1810253 |         pandas.util
import time:        22 |    1810274 |       pandas.util.version
import time:       428 |    1810702 |     pandas.compat.numpy
import time:        97 |         97 |       pyarrow
import time:       156 |        252 |     pandas.compat.pyarrow
import time:       359 |    1825851 |   pandas.compat
import time:     61192 |      61192 |   pandas._libs.tslib
import time:       793 |        793 |   pandas.core.config_init
import time:       211 |        211 |     pandas.core.dtypes.missing
import time:       387 |        387 |       pandas.core.dtypes.cast
import time:       141 |        141 |         pandas.core.dtypes.astype
import time:       164 |        304 |       pandas.core.dtypes.concat
import time:       116 |        116 |         pandas.core.array_algos
import time:       238 |        238 |           pandas.core.common
import time:       160 |        397 |         pandas.core.construction
import time:       266 |        779 |       pandas.core.array_algos.take
import time:       169 |        169 |         pandas.core.indexers.utils
import time:       115 |        284 |       pandas.core.indexers
import time:       712 |       2463 |     pandas.core.algorithms
import time:        69 |         69 |                 org
import time:        12 |         81 |               org.python
import time:        10 |         90 |             org.python.core
import time:       414 |        504 |           copy
import time:       596 |        596 |           unicodedata
import time:       350 |        350 |           pandas.util._validators
import time:       113 |        113 |           pandas.core.roperator
import time:    179370 |     179370 |                   pandas._libs.ops
import time:       271 |        271 |                   pandas.core.computation
import time:       257 |        257 |                     pandas.core.computation.check
import time:       352 |        608 |                   pandas.core.computation.expressions
import time:       150 |        150 |                   pandas.core.ops.missing
import time:       113 |        113 |                   pandas.core.ops.dispatch
import time:       114 |        114 |                   pandas.core.ops.invalid
import time:       360 |     180984 |                 pandas.core.ops.array_ops
import time:       296 |        296 |                 pandas.core.ops.common
import time:       216 |        216 |                 pandas.core.ops.docstrings
import time:       308 |        308 |                 pandas.core.ops.mask_ops
import time:       168 |        168 |                 pandas.core.ops.methods
import time:       363 |     182333 |               pandas.core.ops
import time:        17 |     182350 |             pandas.core.ops.common
import time:       366 |     182715 |           pandas.core.arraylike
import time:       346 |        346 |             pandas.compat.numpy.function
import time:       277 |        277 |             pandas.core.missing
import time:       834 |        834 |             pandas.core.array_algos.quantile
import time:       247 |        247 |             pandas.core.sorting
import time:       679 |       2381 |           pandas.core.arrays.base
import time:       162 |        162 |             pandas.core.strings
import time:       318 |        479 |           pandas.core.strings.base
import time:       384 |        384 |             pandas.tseries
import time:       763 |       1146 |           pandas.tseries.frequencies
import time:       974 |     189253 |         pandas.core.arrays.arrow.array
import time:       216 |        216 |         pandas.core.arrays.arrow.dtype
import time:       163 |     189631 |       pandas.core.arrays.arrow
import time:       129 |        129 |         pandas.core.array_algos.masked_accumulations
import time:       846 |        846 |           pandas.core.nanops
import time:       151 |        151 |           pandas.core.array_algos.masked_reductions
import time:       806 |       1803 |         pandas.core.arrays.masked
import time:       252 |       2183 |       pandas.core.arrays.boolean
import time:       833 |        833 |           _csv
import time:       559 |       1391 |         csv
import time:       778 |        778 |           zlib
import time:      1150 |       1928 |         shutil
import time:     55151 |      55151 |         pandas._libs.arrays
import time:       525 |        525 |         pandas.core.accessor
import time:       124 |        124 |           pandas.core.array_algos.transforms
import time:       599 |        723 |         pandas.core.arrays._mixins
import time:       762 |        762 |         pandas.core.base
import time:       338 |        338 |         pandas.core.strings.object_array
import time:       155 |        155 |           pandas.io
import time:       210 |        365 |         pandas.io.formats
import time:       147 |        147 |         pandas.io.formats.console
import time:       779 |      62105 |       pandas.core.arrays.categorical
import time:       126 |        126 |           pandas.core.array_algos.datetimelike_accumulations
import time:       214 |        214 |             pandas.core.arrays.numeric
import time:       231 |        444 |           pandas.core.arrays.integer
import time:      1238 |       1807 |         pandas.core.arrays.datetimelike
import time:       160 |        160 |         pandas.core.arrays._ranges
import time:       275 |        275 |         pandas.tseries.offsets
import time:       519 |       2760 |       pandas.core.arrays.datetimes
import time:       189 |        189 |       pandas.core.arrays.floating
import time:       371 |        371 |         pandas.core.arrays.timedeltas
import time:      1431 |       1802 |       pandas.core.arrays.interval
import time:       396 |        396 |       pandas.core.arrays.numpy_
import time:       463 |        463 |       pandas.core.arrays.period
import time:    101223 |     101223 |             pandas._libs.sparse
import time:       474 |        474 |             pandas.core.arrays.sparse.dtype
import time:       405 |        405 |             pandas.io.formats.printing
import time:       560 |     102659 |           pandas.core.arrays.sparse.array
import time:       266 |     102925 |         pandas.core.arrays.sparse.accessor
import time:       147 |     103071 |       pandas.core.arrays.sparse
import time:       416 |        416 |       pandas.core.arrays.string_
import time:       230 |        230 |       pandas.core.arrays.string_arrow
import time:       189 |     363430 |     pandas.core.arrays
import time:       109 |        109 |     pandas.core.flags
import time:     54558 |      54558 |         pandas._libs.reduction
import time:       655 |        655 |         pandas.core.apply
import time:        49 |         49 |             gc
import time:     48634 |      48634 |               pandas._libs.indexing
import time:       222 |        222 |                 pandas.core.indexes
import time:     92640 |      92640 |                   pandas._libs.index
import time:       157 |        157 |                     backports_abc
import time:     61677 |      61834 |                   pandas._libs.internals
import time:    190091 |     190091 |                   pandas._libs.join
import time:       264 |        264 |                   pandas.core.array_algos.putmask
import time:       161 |        161 |                   pandas.core.indexes.frozen
import time:      2468 |       2468 |                   pandas.core.strings.accessor
import time:      1799 |     349255 |                 pandas.core.indexes.base
import time:       207 |        207 |                   pandas.core.indexes.extension
import time:       305 |        512 |                 pandas.core.indexes.category
import time:       377 |        377 |                     pandas.core.indexes.range
import time:       122 |        122 |                       pandas.core.tools
import time:       187 |        309 |                     pandas.core.tools.timedeltas
import time:       690 |       1375 |                   pandas.core.indexes.datetimelike
import time:       123 |        123 |                   pandas.core.tools.times
import time:       995 |       2492 |                 pandas.core.indexes.datetimes
import time:      1718 |       1718 |                   pandas.core.indexes.multi
import time:       327 |        327 |                   pandas.core.indexes.timedeltas
import time:       929 |       2974 |                 pandas.core.indexes.interval
import time:       463 |        463 |                 pandas.core.indexes.period
import time:       337 |     356251 |               pandas.core.indexes.api
import time:      1368 |     406252 |             pandas.core.indexing
import time:       161 |        161 |             pandas.core.sample
import time:       132 |        132 |             pandas.core.array_algos.replace
import time:     61246 |      61246 |                   pandas._libs.writers
import time:       637 |      61883 |                 pandas.core.internals.blocks
import time:       168 |      62051 |               pandas.core.internals.api
import time:       287 |        287 |                 pandas.core.internals.base
import time:       559 |        846 |               pandas.core.internals.array_manager
import time:       347 |        347 |                   pandas.core.internals.ops
import time:       822 |       1168 |                 pandas.core.internals.managers
import time:       249 |       1417 |               pandas.core.internals.concat
import time:       163 |      64475 |             pandas.core.internals
import time:       253 |        253 |             pandas.core.internals.construction
import time:       125 |        125 |               pandas.core.methods
import time:        97 |         97 |                 pandas.core.reshape
import time:       431 |        528 |               pandas.core.reshape.concat
import time:       779 |        779 |                   dataclasses
import time:       585 |        585 |                   gzip
import time:       733 |        733 |                   mmap
import time:       953 |        953 |                   tarfile
import time:      1278 |       1278 |                   zipfile
import time:       140 |        140 |                   pandas.core.shared_docs
import time:      1551 |       6016 |                 pandas.io.common
import time:       612 |       6628 |               pandas.io.formats.format
import time:       352 |       7631 |             pandas.core.methods.describe
import time:        99 |         99 |                   pandas._libs.window
import time:     74247 |      74345 |                 pandas._libs.window.aggregations
import time:     55648 |      55648 |                   pandas._libs.window.indexers
import time:       585 |      56232 |                 pandas.core.indexers.objects
import time:       254 |        254 |                 pandas.core.util.numba_
import time:       146 |        146 |                 pandas.core.window.common
import time:       233 |        233 |                 pandas.core.window.doc
import time:       151 |        151 |                 pandas.core.window.numba_
import time:       105 |        105 |                 pandas.core.window.online
import time:       129 |        129 |                   pandas.core._numba
import time:       155 |        155 |                   pandas.core._numba.executor
import time:      2204 |       2487 |                 pandas.core.window.rolling
import time:       607 |     134557 |               pandas.core.window.ewm
import time:       961 |        961 |               pandas.core.window.expanding
import time:       162 |     135679 |             pandas.core.window
import time:      5717 |     620346 |           pandas.core.generic
import time:       341 |        341 |           pandas.core.methods.selectn
import time:       119 |        119 |             pandas.core.reshape.util
import time:       154 |        154 |             pandas.core.tools.numeric
import time:       376 |        649 |           pandas.core.reshape.melt
import time:     62686 |      62686 |             pandas._libs.reshape
import time:       675 |        675 |             pandas.core.indexes.accessors
import time:       143 |        143 |               pandas.arrays
import time:       607 |        749 |             pandas.core.tools.datetimes
import time:       894 |        894 |             pandas.io.formats.info
import time:      1482 |       1482 |               pandas.plotting._core
import time:       299 |        299 |               pandas.plotting._misc
import time:       145 |       1925 |             pandas.plotting
import time:      3674 |      70601 |           pandas.core.series
import time:      7345 |     699280 |         pandas.core.frame
import time:      1054 |       1054 |         pandas.core.groupby.base
import time:       154 |        154 |             backports_abc
import time:    159211 |     159365 |           pandas._libs.groupby
import time:       224 |        224 |           pandas.core.groupby.numba_
import time:       103 |        103 |               pandas.core.groupby.categorical
import time:       318 |        421 |             pandas.core.groupby.grouper
import time:       602 |       1023 |           pandas.core.groupby.ops
import time:       319 |        319 |           pandas.core.groupby.indexing
import time:      2276 |     163205 |         pandas.core.groupby.groupby
import time:      2318 |     921067 |       pandas.core.groupby.generic
import time:       133 |     921199 |     pandas.core.groupby
import time:       219 |    1287629 |   pandas.core.api
import time:       118 |        118 |   pandas.tseries.api
import time:        78 |         78 |           pandas.core.computation.common
import time:       131 |        209 |         pandas.core.computation.align
import time:       583 |        583 |             pprint
import time:       194 |        777 |           pandas.core.computation.scope
import time:       282 |       1058 |         pandas.core.computation.ops
import time:       158 |       1425 |       pandas.core.computation.engines
import time:       105 |        105 |         pandas.core.computation.parsing
import time:       761 |        866 |       pandas.core.computation.expr
import time:       147 |       2437 |     pandas.core.computation.eval
import time:        89 |       2525 |   pandas.core.computation.api
import time:       157 |        157 |     pandas.core.reshape.encoding
import time:       653 |        653 |         _uuid
import time:       660 |       1312 |       uuid
import time:      1388 |       2699 |     pandas.core.reshape.merge
import time:      7373 |       7373 |     pandas.core.reshape.pivot
import time:       314 |        314 |     pandas.core.reshape.tile
import time:       146 |      10688 |   pandas.core.reshape.api
import time:       164 |        164 |     pandas.api.extensions
import time:       107 |        107 |     pandas.api.indexers
import time:        88 |         88 |         pandas.core.interchange
import time:       650 |        737 |       pandas.core.interchange.dataframe_protocol
import time:       140 |        140 |             pandas.core.dtypes.api
import time:       138 |        277 |           pandas.api.types
import time:       132 |        409 |         pandas.core.interchange.utils
import time:       167 |        575 |       pandas.core.interchange.from_dataframe
import time:       108 |       1418 |     pandas.api.interchange
import time:       161 |       1849 |   pandas.api
import time:      1076 |       1076 |           _socket
import time:       906 |        906 |           array
import time:      1399 |       3381 |         socket
import time:       203 |        203 |         pandas._testing._random
import time:       508 |        508 |           tempfile
import time:       188 |        696 |         pandas._testing.contexts
import time:       226 |       4504 |       pandas._testing._io
import time:       129 |        129 |       pandas._testing._warnings
import time:      1120 |       1120 |           cmath
import time:     51368 |      52487 |         pandas._libs.testing
import time:       285 |      52772 |       pandas._testing.asserters
import time:       161 |        161 |       pandas._testing.compat
import time:       851 |      58415 |     pandas._testing
import time:       207 |      58621 |   pandas.testing
import time:       163 |        163 |   pandas.util._print_versions
import time:       127 |        127 |     pandas.io.clipboards
import time:       139 |        139 |           backports_abc
import time:     75160 |      75299 |         pandas._libs.parsers
import time:       217 |        217 |         pandas.io.excel._util
import time:        89 |         89 |               pandas.io._util
import time:       418 |        418 |               pandas.io.parsers.base_parser
import time:       175 |        681 |             pandas.io.parsers.arrow_parser_wrapper
import time:       166 |        166 |             pandas.io.parsers.c_parser_wrapper
import time:       335 |        335 |             pandas.io.parsers.python_parser
import time:      2026 |       3206 |           pandas.io.parsers.readers
import time:       148 |       3354 |         pandas.io.parsers
import time:       307 |        307 |         pandas.io.excel._odfreader
import time:       488 |        488 |         pandas.io.excel._openpyxl
import time:       179 |        179 |         pandas.io.excel._pyxlsb
import time:       169 |        169 |         pandas.io.excel._xlrd
import time:      1718 |      81726 |       pandas.io.excel._base
import time:     33728 |      33728 |         pandas._libs.json
import time:       346 |      34073 |       pandas.io.excel._odswriter
import time:       318 |        318 |       pandas.io.excel._xlsxwriter
import time:       144 |     116260 |     pandas.io.excel
import time:       264 |        264 |     pandas.io.feather_format
import time:       108 |        108 |     pandas.io.gbq
import time:       476 |        476 |     pandas.io.html
import time:       209 |        209 |         pandas.io.json._normalize
import time:       140 |        140 |         pandas.io.json._table_schema
import time:       945 |       1294 |       pandas.io.json._json
import time:       140 |       1434 |     pandas.io.json
import time:       122 |        122 |     pandas.io.orc
import time:       412 |        412 |     pandas.io.parquet
import time:       171 |        171 |       pandas.compat.pickle_compat
import time:       317 |        488 |     pandas.io.pickle
import time:       354 |        354 |       pandas.core.computation.pytables
import time:      1185 |       1538 |     pandas.io.pytables
import time:       268 |        268 |       pandas.io.sas.sasreader
import time:       120 |        388 |     pandas.io.sas
import time:        86 |         86 |     pandas.io.spss
import time:       515 |        515 |     pandas.io.sql
import time:      1555 |       1555 |     pandas.io.stata
import time:       752 |        752 |     pandas.io.xml
import time:       200 |     124717 |   pandas.io.api
import time:        96 |         96 |   pandas.util._tester
import time:        81 |         81 |   pandas._version
import time:     14154 |    9146555 | pandas

@pyrito
Copy link
Contributor Author

pyrito commented Jun 1, 2023

@lithomas1 for further context, it seems like this is a one time cost that we pay in a new environment. I see that a fair amount of time is spent importing numpy, so I'm not sure if there's anything we can do here. Future imports of pandas is much faster, which I presume is some caching done by conda/python.

@lithomas1
Copy link
Member

Yeah, the numpy import is taking 5 seconds for some reason.

Ideally pandas should load faster, but the main reason for the slowness is that we load all of our extension modules at import time (which is slow, since some are quite large).

@lithomas1 lithomas1 added Internals Related to non-user accessible pandas implementation and removed Needs Info Clarification about behavior needed to assess issue labels Jun 2, 2023
@rhshadrach
Copy link
Member

Something indeed seems off; your walltime is significantly higher than your CPU time. This indicates to me that something is blocked (e.g. IO). Is your pandas installation on a networked drive (e.g. an NFS)?

@pyrito
Copy link
Contributor Author

pyrito commented Jun 6, 2023

@rhshadrach no this is all on my local M1 macbook pro.

@mroeschke mroeschke mentioned this issue May 24, 2024
Closed
3 tasks
@carbon-starlight
Copy link

carbon-starlight commented May 28, 2024 •
edited

@lithomas1 for further context, it seems like this is a one time cost that we pay in a new environment. I see that a fair amount of time is spent importing numpy, so I'm not sure if there's anything we can do here. Future imports of pandas is much faster, which I presume is some caching done by conda/python.

I face the same problem (more severe—importing pandas may take up to 114 seconds on my old Debian 12 IA-32 computer, no GUI running), but is takes long every time I run a script with importing it in the same virtual environment. I don't use conda though.

My import times 
(.venv) user@debian:~/Projects/WikiSchedule$ python3 -X importtime -c "import pandas as pd"
import time: self [us] | cumulative | imported package
import time:      1577 |       1577 |   _io
import time:       660 |        660 |   marshal
import time:      3310 |       3310 |   posix
import time:      3527 |       9073 | _frozen_importlib_external
import time:      1082 |       1082 |   time
import time:      1439 |       2520 | zipimport
import time:       532 |        532 |     _codecs
import time:      2349 |       2880 |   codecs
import time:      3530 |       3530 |   encodings.aliases
import time:      5398 |      11807 | encodings
import time:      1827 |       1827 | encodings.utf_8
import time:       779 |        779 | _signal
import time:       275 |        275 |     _abc
import time:      1010 |       1284 |   abc
import time:      1568 |       2851 | io
import time:       297 |        297 |       _stat
import time:       506 |        802 |     stat
import time:      5039 |       5039 |     _collections_abc
import time:       357 |        357 |       genericpath
import time:       719 |       1075 |     posixpath
import time:      2526 |       9440 |   os
import time:       600 |        600 |   _sitebuiltins
import time:      3685 |       3685 |   _distutils_hack
import time:       559 |        559 |     apport_python_hook
import time:      1159 |       1718 |   sitecustomize
import time:      5526 |      20968 | site
import time:      1278 |       1278 |   __future__
import time:      2351 |       2351 |   warnings
import time:      2379 |       2379 |         types
import time:       569 |        569 |           _operator
import time:      2450 |       3019 |         operator
import time:       662 |        662 |             itertools
import time:      1415 |       1415 |             keyword
import time:      1723 |       1723 |             reprlib
import time:       593 |        593 |             _collections
import time:      6484 |      10875 |           collections
import time:       534 |        534 |           _functools
import time:      3875 |      15284 |         functools
import time:      9638 |      30319 |       enum
import time:      1295 |       1295 |         numpy._utils._convertions
import time:      1449 |       2744 |       numpy._utils
import time:      3009 |      36070 |     numpy._globals
import time:      1886 |       1886 |     numpy.exceptions
import time:      1043 |       1043 |     numpy.version
import time:       329 |        329 |       numpy._distributor_init_local
import time:      1189 |       1518 |     numpy._distributor_init
import time:      1164 |       1164 |               numpy._utils._inspect
import time:       917 |        917 |                   math
import time:       964 |        964 |                   _datetime
import time:      5872 |       7752 |                 datetime
import time:      2026 |       2026 |                 numpy.core._exceptions
import time:       932 |        932 |                 numpy.dtypes
import time:     18367 |      29075 |               numpy.core._multiarray_umath
import time:      2252 |      32490 |             numpy.core.overrides
import time:      4234 |      36724 |           numpy.core.multiarray
import time:      1607 |       1607 |           numpy.core.umath
import time:      2309 |       2309 |             numbers
import time:      1335 |       1335 |             numpy.core._string_helpers
import time:       463 |        463 |                           _sre
import time:      2365 |       2365 |                             re._constants
import time:      2753 |       5117 |                           re._parser
import time:      1122 |       1122 |                           re._casefix
import time:      2315 |       9015 |                         re._compiler
import time:      1314 |       1314 |                         copyreg
import time:      4142 |      14470 |                       re
import time:      1324 |      15794 |                     fnmatch
import time:       637 |        637 |                     errno
import time:      1256 |       1256 |                       urllib
import time:      7189 |       8444 |                     urllib.parse
import time:       707 |        707 |                       _winapi
import time:       804 |        804 |                       nt
import time:       769 |        769 |                       nt
import time:       905 |        905 |                       nt
import time:       795 |        795 |                       nt
import time:       762 |        762 |                       nt
import time:      1328 |       6066 |                     ntpath
import time:      6268 |      37207 |                   pathlib
import time:       993 |        993 |                   pickle5
import time:       497 |        497 |                       _struct
import time:      1300 |       1796 |                     struct
import time:      2048 |       2048 |                     _compat_pickle
import time:       903 |        903 |                     _pickle
import time:       770 |        770 |                         org
import time:       196 |        966 |                       org.python
import time:       195 |       1161 |                     org.python.core
import time:     10052 |      15957 |                   pickle
import time:      2316 |      56471 |                 numpy.compat.py3k
import time:      1511 |      57982 |               numpy.compat
import time:      1446 |       1446 |               numpy.core._dtype
import time:      3725 |      63152 |             numpy.core._type_aliases
import time:      3081 |      69875 |           numpy.core.numerictypes
import time:      4207 |       4207 |                   contextlib
import time:      1226 |       1226 |                     collections.abc
import time:      1152 |       1152 |                       _contextvars
import time:      1432 |       2584 |                     contextvars
import time:      2036 |       5845 |                   numpy.core._ufunc_config
import time:      2120 |      12171 |                 numpy.core._methods
import time:      8317 |      20488 |               numpy.core.fromnumeric
import time:      2870 |      23358 |             numpy.core.shape_base
import time:      3892 |       3892 |             numpy.core.arrayprint
import time:      1310 |       1310 |             numpy.core._asarray
import time:      7273 |      35832 |           numpy.core.numeric
import time:      7647 |       7647 |           numpy.core.defchararray
import time:      2325 |       2325 |           numpy.core.records
import time:      1499 |       1499 |           numpy.core.memmap
import time:      2339 |       2339 |           numpy.core.function_base
import time:      1394 |       1394 |           numpy.core._machar
import time:      2029 |       2029 |           numpy.core.getlimits
import time:      2154 |       2154 |           numpy.core.einsumfunc
import time:      1526 |       1526 |             numpy.core._multiarray_tests
import time:     13246 |      14771 |           numpy.core._add_newdocs
import time:      4127 |       4127 |           numpy.core._add_newdocs_scalars
import time:      1247 |       1247 |           numpy.core._dtype_ctypes
import time:      8175 |       8175 |               _ast
import time:      6927 |      15102 |             ast
import time:      2517 |       2517 |               _ctypes
import time:      2265 |       2265 |               ctypes._endian
import time:      6526 |      11306 |             ctypes
import time:      6280 |      32687 |           numpy.core._internal
import time:      1416 |       1416 |           numpy._pytesttester
import time:     10000 |     227664 |         numpy.core
import time:       254 |     227917 |       numpy.core._multiarray_umath
import time:      3043 |     230960 |     numpy.__config__
import time:      2039 |       2039 |       numpy.lib.mixins
import time:      1278 |       1278 |           numpy.lib.ufunclike
import time:      2839 |       4116 |         numpy.lib.type_check
import time:      2556 |       6672 |       numpy.lib.scimath
import time:      1136 |       1136 |                   _typing
import time:     15208 |      16344 |                 typing
import time:      1674 |       1674 |                   numpy.lib.stride_tricks
import time:      3128 |       4801 |                 numpy.lib.twodim_base
import time:      6303 |       6303 |                 numpy.linalg._umath_linalg
import time:      1930 |       1930 |                   numpy._typing._nested_sequence
import time:      1140 |       1140 |                   numpy._typing._nbit
import time:      6993 |       6993 |                   numpy._typing._char_codes
import time:      2949 |       2949 |                   numpy._typing._scalars
import time:      1509 |       1509 |                   numpy._typing._shape
import time:     12421 |      12421 |                   numpy._typing._dtype_like
import time:     18100 |      18100 |                   numpy._typing._array_like
import time:      3534 |      48575 |                 numpy._typing
import time:     10052 |      86073 |               numpy.linalg.linalg
import time:      1579 |      87651 |             numpy.linalg
import time:      2962 |      90613 |           numpy.matrixlib.defmatrix
import time:      1452 |      92064 |         numpy.matrixlib
import time:      2518 |       2518 |           numpy.lib.histograms
import time:      7946 |      10463 |         numpy.lib.function_base
import time:      3692 |     106218 |       numpy.lib.index_tricks
import time:      3289 |       3289 |       numpy.lib.nanfunctions
import time:      3211 |       3211 |       numpy.lib.shape_base
import time:      4403 |       4403 |       numpy.lib.polynomial
import time:      6776 |       6776 |         textwrap
import time:     13075 |      13075 |         platform
import time:      5047 |      24897 |       numpy.lib.utils
import time:      2555 |       2555 |       numpy.lib.arraysetops
import time:      2270 |       2270 |           _weakrefset
import time:      4568 |       6837 |         weakref
import time:      2418 |       2418 |         numpy.lib.format
import time:      3367 |       3367 |         numpy.lib._datasource
import time:      3468 |       3468 |         numpy.lib._iotools
import time:      5432 |      21520 |       numpy.lib.npyio
import time:      1820 |       1820 |       numpy.lib.arrayterator
import time:      2880 |       2880 |       numpy.lib.arraypad
import time:      1504 |       1504 |       numpy.lib._version
import time:      5701 |     186701 |     numpy.lib
import time:      1546 |       1546 |         numpy.fft._pocketfft_internal
import time:      4901 |       6446 |       numpy.fft._pocketfft
import time:      1957 |       1957 |       numpy.fft.helper
import time:      2149 |      10552 |     numpy.fft
import time:      2703 |       2703 |         numpy.polynomial.polyutils
import time:      4440 |       4440 |         numpy.polynomial._polybase
import time:      4115 |      11257 |       numpy.polynomial.polynomial
import time:      3273 |       3273 |       numpy.polynomial.chebyshev
import time:      2490 |       2490 |       numpy.polynomial.legendre
import time:      2490 |       2490 |       numpy.polynomial.hermite
import time:      2817 |       2817 |       numpy.polynomial.hermite_e
import time:      2528 |       2528 |       numpy.polynomial.laguerre
import time:      2251 |      27102 |     numpy.polynomial
import time:       768 |        768 |               backports_abc
import time:      3701 |       4469 |             numpy.random._common
import time:       460 |        460 |                 binascii
import time:      2461 |       2921 |               base64
import time:     15275 |      15275 |                 _hashlib
import time:       463 |        463 |                   _blake2
import time:      2676 |       3139 |                 hashlib
import time:      2124 |      20537 |               hmac
import time:       266 |        266 |                   _bisect
import time:      1242 |       1507 |                 bisect
import time:       419 |        419 |                 _random
import time:       539 |        539 |                 _sha512
import time:      3323 |       5786 |               random
import time:      1444 |      30686 |             secrets
import time:      4737 |       4737 |             threading
import time:      3590 |      43480 |           numpy.random.bit_generator
import time:      1990 |       1990 |           numpy.random._bounded_integers
import time:      1707 |       1707 |           numpy.random._mt19937
import time:      6682 |      53857 |         numpy.random.mtrand
import time:      1992 |       1992 |         numpy.random._philox
import time:      1985 |       1985 |         numpy.random._pcg64
import time:      1973 |       1973 |         numpy.random._sfc64
import time:      8760 |       8760 |         numpy.random._generator
import time:      1945 |      70509 |       numpy.random._pickle
import time:      2002 |      72510 |     numpy.random
import time:      4935 |       4935 |     numpy.ctypeslib
import time:       293 |        293 |               _opcode
import time:      3002 |       3295 |             opcode
import time:      4982 |       8276 |           dis
import time:      1477 |       1477 |             importlib
import time:       971 |       2447 |           importlib.machinery
import time:      1592 |       1592 |               token
import time:      6203 |       7794 |             tokenize
import time:      1517 |       9311 |           linecache
import time:     11586 |      31619 |         inspect
import time:     18885 |      50503 |       numpy.ma.core
import time:      7705 |       7705 |       numpy.ma.extras
import time:      2566 |      60774 |     numpy.ma
import time:     19216 |     653261 |   numpy
import time:      1762 |       1762 |     pytz.exceptions
import time:      2024 |       2024 |     pytz.lazy
import time:      1956 |       1956 |     pytz.tzinfo
import time:      1398 |       1398 |     pytz.tzfile
import time:      5213 |      12351 |   pytz
import time:      1208 |       1208 |     dateutil._version
import time:      1903 |       3110 |   dateutil
import time:      2926 |       2926 |       sysconfig
import time:      4412 |       4412 |       _sysconfigdata__i386-linux-gnu
import time:      4258 |      11595 |     pandas.compat._constants
import time:      1849 |       1849 |         _compression
import time:      2313 |       2313 |         _bz2
import time:      2338 |       6499 |       bz2
import time:      2514 |       2514 |         _lzma
import time:      3317 |       5831 |       lzma
import time:      2225 |      14553 |     pandas.compat.compressors
import time:      1316 |       1316 |         pandas.util
import time:     16489 |      17804 |       pandas.util.version
import time:      2592 |      20395 |     pandas.compat.numpy
import time:       628 |        628 |       pyarrow
import time:      1287 |       1915 |     pandas.compat.pyarrow
import time:      2272 |      50728 |   pandas.compat
import time:     25084 |      25084 |       pandas._typing
import time:      1409 |       1409 |       pandas.util._exceptions
import time:      5769 |      32261 |     pandas._config.config
import time:      2012 |       2012 |     pandas._config.dates
import time:       559 |        559 |         _locale
import time:      5221 |       5779 |       locale
import time:      1790 |       7568 |     pandas._config.display
import time:      2095 |      43936 |   pandas._config
import time:      1148 |       1148 |     pandas.core
import time:      9735 |      10883 |   pandas.core.config_init
import time:      1611 |       1611 |       pandas._libs.pandas_parser
import time:      1038 |       1038 |           numpy._core
import time:      1970 |       3008 |         numpy._core._multiarray_umath
import time:      1702 |       4709 |       pandas._libs.pandas_datetime
import time:      1437 |       1437 |                   pandas._libs.tslibs.ccalendar
import time:      2397 |       2397 |                   pandas._libs.tslibs.np_datetime
import time:      8525 |      12358 |                 pandas._libs.tslibs.dtypes
import time:      1338 |       1338 |                   pandas._libs.tslibs.base
import time:      2578 |       2578 |                       pandas._libs.tslibs.nattype
import time:      1258 |       1258 |                           pandas.compat._optional
import time:      1764 |       1764 |                             zoneinfo._tzpath
import time:      1576 |       1576 |                             zoneinfo._common
import time:      1811 |       1811 |                             _zoneinfo
import time:      1885 |       7034 |                           zoneinfo
import time:      1512 |       1512 |                                   importlib._abc
import time:      1195 |       2707 |                                 importlib.util
import time:      6251 |       8957 |                               six
import time:       434 |        434 |                               six.moves
import time:      1813 |       1813 |                               dateutil.tz._common
import time:      1515 |       1515 |                               dateutil.tz._factories
import time:       343 |        343 |                                 six.moves.winreg
import time:      1873 |       2216 |                               dateutil.tz.win
import time:      6103 |      21037 |                             dateutil.tz.tz
import time:      1705 |      22741 |                           dateutil.tz
import time:      3778 |      34811 |                         pandas._libs.tslibs.timezones
import time:      3139 |       3139 |                             calendar
import time:      5788 |       8927 |                           _strptime
import time:      4769 |       4769 |                               signal
import time:       524 |        524 |                               fcntl
import time:       916 |        916 |                               msvcrt
import time:       478 |        478 |                               _posixsubprocess
import time:       593 |        593 |                               select
import time:      3976 |       3976 |                               selectors
import time:      4147 |      15399 |                             subprocess
import time:      1435 |      16834 |                           pandas._config.localization
import time:      3174 |      28933 |                         pandas._libs.tslibs.fields
import time:      6139 |      69881 |                       pandas._libs.tslibs.timedeltas
import time:      2797 |       2797 |                       pandas._libs.tslibs.tzconversion
import time:       678 |        678 |                       backports_abc
import time:      6462 |      82394 |                     pandas._libs.tslibs.timestamps
import time:       691 |        691 |                     backports_abc
import time:      1460 |       1460 |                     pandas._libs.properties
import time:     10123 |      94667 |                   pandas._libs.tslibs.offsets
import time:       670 |        670 |                     backports_abc
import time:      3897 |       3897 |                       _decimal
import time:      1300 |       5197 |                     decimal
import time:       302 |        302 |                           _string
import time:      4291 |       4593 |                         string
import time:      1044 |       1044 |                         dateutil._common
import time:      7131 |      12766 |                       dateutil.parser._parser
import time:      2356 |       2356 |                       dateutil.parser.isoparser
import time:      2186 |      17308 |                     dateutil.parser
import time:      6757 |       6757 |                     pandas._libs.tslibs.strptime
import time:      5500 |      35430 |                   pandas._libs.tslibs.parsing
import time:      4434 |     135868 |                 pandas._libs.tslibs.conversion
import time:      4422 |       4422 |                 pandas._libs.tslibs.period
import time:      3015 |       3015 |                 pandas._libs.tslibs.vectorized
import time:      2064 |     157726 |               pandas._libs.tslibs
import time:       269 |     157994 |             pandas._libs.tslibs.nattype
import time:      1363 |       1363 |             pandas._libs.ops_dispatch
import time:      2898 |     162254 |           pandas._libs.missing
import time:     10632 |     172885 |         pandas._libs.hashtable
import time:      3624 |       3624 |         pandas._libs.algos
import time:      9071 |     185580 |       pandas._libs.interval
import time:      1463 |     193362 |     pandas._libs
import time:      1145 |       1145 |       pandas.core.dtypes
import time:       695 |        695 |         pyarrow
import time:      6272 |       6967 |       pandas._libs.lib
import time:      4580 |       4580 |       pandas.errors
import time:      3562 |       3562 |         pandas.core.dtypes.generic
import time:      2640 |       6201 |       pandas.core.dtypes.base
import time:      1368 |       1368 |       pandas.core.dtypes.inference
import time:      9919 |      30179 |     pandas.core.dtypes.dtypes
import time:      2265 |       2265 |       pandas.core.dtypes.common
import time:      2478 |       4743 |     pandas.core.dtypes.missing
import time:      1741 |       1741 |       pandas.util._decorators
import time:      1053 |       1053 |           pandas.io
import time:      2018 |       3071 |         pandas.io._util
import time:      4326 |       7396 |       pandas.core.dtypes.cast
import time:      1428 |       1428 |         pandas.core.dtypes.astype
import time:      1701 |       3128 |       pandas.core.dtypes.concat
import time:      1100 |       1100 |         pandas.core.array_algos
import time:      2358 |       2358 |           pandas.core.common
import time:      2075 |       4433 |         pandas.core.construction
import time:      2770 |       8301 |       pandas.core.array_algos.take
import time:      1599 |       1599 |         pandas.core.indexers.utils
import time:      1354 |       2953 |       pandas.core.indexers
import time:      4936 |      28453 |     pandas.core.algorithms
import time:      1931 |       1931 |         pandas.core.arrays.arrow.accessors
import time:       594 |        594 |           unicodedata
import time:      1492 |       1492 |           pandas.util._validators
import time:      3323 |       3323 |           pandas.core.missing
import time:      2112 |       2112 |               pandas._libs.ops
import time:      1187 |       1187 |               pandas.core.roperator
import time:      1066 |       1066 |               pandas.core.computation
import time:      1901 |       1901 |                 pandas.core.computation.check
import time:      1987 |       3887 |               pandas.core.computation.expressions
import time:      1060 |       1060 |               pandas.core.ops.missing
import time:      1254 |       1254 |               pandas.core.ops.dispatch
import time:      1089 |       1089 |               pandas.core.ops.invalid
import time:      3118 |      14769 |             pandas.core.ops.array_ops
import time:      1397 |       1397 |             pandas.core.ops.common
import time:      1740 |       1740 |             pandas.core.ops.docstrings
import time:      1307 |       1307 |             pandas.core.ops.mask_ops
import time:      1698 |      20908 |           pandas.core.ops
import time:      3236 |       3236 |           pandas.core.arraylike
import time:      1392 |       1392 |           pandas.core.arrays._arrow_string_mixins
import time:      1468 |       1468 |           pandas.core.arrays._utils
import time:      2453 |       2453 |             pandas.compat.numpy.function
import time:      1245 |       1245 |             pandas.core.array_algos.quantile
import time:      1796 |       1796 |             pandas.core.sorting
import time:      5184 |      10676 |           pandas.core.arrays.base
import time:      7425 |       7425 |             pandas.core.nanops
import time:      1340 |       1340 |             pandas.core.array_algos.masked_accumulations
import time:      1402 |       1402 |             pandas.core.array_algos.masked_reductions
import time:      1082 |       1082 |               pandas.core.util
import time:      2454 |       2454 |               pandas._libs.hashing
import time:      2249 |       5783 |             pandas.core.util.hashing
import time:      6977 |      22925 |           pandas.core.arrays.masked
import time:      1704 |       1704 |             pandas._libs.arrays
import time:      1674 |       1674 |               pandas.core.arrays.numeric
import time:      2220 |       3893 |             pandas.core.arrays.floating
import time:      2807 |       2807 |             pandas.core.arrays.integer
import time:       975 |        975 |                 pandas.core.array_algos.transforms
import time:      4017 |       4992 |               pandas.core.arrays._mixins
import time:      1036 |       1036 |                 pandas.core.strings
import time:      2046 |       2046 |                 pandas.core.strings.base
import time:      3473 |       6554 |               pandas.core.strings.object_array
import time:      6581 |      18126 |             pandas.core.arrays.numpy_
import time:      3785 |      30312 |           pandas.core.arrays.string_
import time:      1034 |       1034 |             pandas.tseries
import time:      3144 |       4177 |           pandas.tseries.frequencies
import time:     10428 |     110926 |         pandas.core.arrays.arrow.array
import time:      1701 |     114558 |       pandas.core.arrays.arrow
import time:      2102 |       2102 |       pandas.core.arrays.boolean
import time:       439 |        439 |           _csv
import time:      2613 |       3052 |         csv
import time:      1733 |       1733 |           zlib
import time:      5287 |       7019 |         shutil
import time:      3576 |       3576 |         pandas.core.accessor
import time:      6350 |       6350 |         pandas.core.base
import time:      1410 |       1410 |         pandas.io.formats
import time:      1927 |       1927 |         pandas.io.formats.console
import time:      7597 |      30927 |       pandas.core.arrays.categorical
import time:      4839 |       4839 |         pandas._libs.tslib
import time:      1048 |       1048 |           pandas.core.array_algos.datetimelike_accumulations
import time:     10546 |      11593 |         pandas.core.arrays.datetimelike
import time:      1221 |       1221 |         pandas.core.arrays._ranges
import time:      1278 |       1278 |         pandas.tseries.offsets
import time:      6593 |      25522 |       pandas.core.arrays.datetimes
import time:      4086 |       4086 |         pandas.core.arrays.timedeltas
import time:      9693 |      13779 |       pandas.core.arrays.interval
import time:      3834 |       3834 |       pandas.core.arrays.period
import time:      2622 |       2622 |             pandas._libs.sparse
import time:      2726 |       2726 |             pandas.io.formats.printing
import time:      5422 |      10769 |           pandas.core.arrays.sparse.array
import time:      2473 |      13242 |         pandas.core.arrays.sparse.accessor
import time:      1387 |      14629 |       pandas.core.arrays.sparse
import time:      3376 |       3376 |       pandas.core.arrays.string_arrow
import time:      2362 |     211084 |     pandas.core.arrays
import time:      1414 |       1414 |     pandas.core.flags
import time:       669 |        669 |             backports_abc
import time:      4040 |       4709 |           pandas._libs.internals
import time:      1044 |       1044 |             pandas.core._numba
import time:      2067 |       3111 |           pandas.core._numba.executor
import time:      4780 |      12598 |         pandas.core.apply
import time:       769 |        769 |                   org
import time:       178 |        947 |                 org.python
import time:       191 |       1137 |               org.python.core
import time:      1558 |       2695 |             copy
import time:       525 |        525 |             gc
import time:      1301 |       1301 |                   _json
import time:      3299 |       4599 |                 json.scanner
import time:      3228 |       7827 |               json.decoder
import time:      3051 |       3051 |               json.encoder
import time:      1824 |      12701 |             json
import time:      1313 |       1313 |               pandas._libs.indexing
import time:       952 |        952 |                 pandas.core.indexes
import time:       809 |        809 |                     backports_abc
import time:      5734 |       6543 |                   pandas._libs.index
import time:      2409 |       2409 |                   pandas._libs.writers
import time:      2613 |       2613 |                   pandas._libs.join
import time:      1257 |       1257 |                   pandas.core.array_algos.putmask
import time:      1507 |       1507 |                   pandas.core.indexes.frozen
import time:     17134 |      17134 |                   pandas.core.strings.accessor
import time:     14753 |      46214 |                 pandas.core.indexes.base
import time:      1807 |       1807 |                   pandas.core.indexes.extension
import time:      2898 |       4705 |                 pandas.core.indexes.category
import time:      4195 |       4195 |                     pandas.core.indexes.range
import time:      1195 |       1195 |                       pandas.core.tools
import time:      2043 |       3237 |                     pandas.core.tools.timedeltas
import time:      6095 |      13526 |                   pandas.core.indexes.datetimelike
import time:      1405 |       1405 |                   pandas.core.tools.times
import time:      7406 |      22336 |                 pandas.core.indexes.datetimes
import time:      9446 |       9446 |                   pandas.core.indexes.multi
import time:      6020 |       6020 |                   pandas.core.indexes.timedeltas
import time:      6388 |      21853 |                 pandas.core.indexes.interval
import time:      3843 |       3843 |                 pandas.core.indexes.period
import time:      2811 |     102711 |               pandas.core.indexes.api
import time:     10017 |     114040 |             pandas.core.indexing
import time:      1316 |       1316 |             pandas.core.sample
import time:      1554 |       1554 |             pandas.core.array_algos.replace
import time:      5158 |       5158 |                 pandas.core.internals.blocks
import time:      2168 |       7326 |               pandas.core.internals.api
import time:      1987 |       1987 |                 pandas.core.internals.base
import time:      2646 |       2646 |                   pandas.core.internals.ops
import time:      5589 |       8235 |                 pandas.core.internals.managers
import time:      4763 |      14984 |               pandas.core.internals.array_manager
import time:      2614 |       2614 |               pandas.core.internals.concat
import time:      1619 |      26541 |             pandas.core.internals
import time:      2541 |       2541 |             pandas.core.internals.construction
import time:      1088 |       1088 |               pandas.core.methods
import time:      1056 |       1056 |                 pandas.core.reshape
import time:      2812 |       3867 |               pandas.core.reshape.concat
import time:      4119 |       4119 |                   dataclasses
import time:      3921 |       3921 |                   gzip
import time:      2041 |       2041 |                   mmap
import time:       410 |        410 |                     pwd
import time:       723 |        723 |                     grp
import time:      7535 |       8667 |                   tarfile
import time:      7020 |       7020 |                   zipfile
import time:      1447 |       1447 |                   pandas.core.shared_docs
import time:     11730 |      38942 |                 pandas.io.common
import time:      5780 |      44721 |               pandas.io.formats.format
import time:      3041 |      52716 |             pandas.core.methods.describe
import time:      1073 |       1073 |                   pandas._libs.window
import time:      7783 |       8855 |                 pandas._libs.window.aggregations
import time:      2002 |       2002 |                   pandas._libs.window.indexers
import time:      3766 |       5768 |                 pandas.core.indexers.objects
import time:      1382 |       1382 |                 pandas.core.util.numba_
import time:      1579 |       1579 |                 pandas.core.window.common
import time:      1842 |       1842 |                 pandas.core.window.doc
import time:      1711 |       1711 |                 pandas.core.window.numba_
import time:      1270 |       1270 |                 pandas.core.window.online
import time:     14734 |      14734 |                 pandas.core.window.rolling
import time:      6794 |      43931 |               pandas.core.window.ewm
import time:      8831 |       8831 |               pandas.core.window.expanding
import time:      1554 |      54316 |             pandas.core.window
import time:     50048 |     318988 |           pandas.core.generic
import time:      6091 |       6091 |           pandas.core.methods.selectn
import time:      1229 |       1229 |             pandas.core.reshape.util
import time:      1501 |       1501 |             pandas.core.tools.numeric
import time:      2573 |       5302 |           pandas.core.reshape.melt
import time:      2368 |       2368 |             pandas._libs.reshape
import time:      5280 |       5280 |             pandas.core.indexes.accessors
import time:      1215 |       1215 |               pandas.arrays
import time:      5879 |       7093 |             pandas.core.tools.datetimes
import time:      6415 |       6415 |             pandas.io.formats.info
import time:      7014 |       7014 |               pandas.plotting._core
import time:      1754 |       1754 |               pandas.plotting._misc
import time:      1620 |      10387 |             pandas.plotting
import time:     26806 |      58346 |           pandas.core.series
import time:     56043 |     444768 |         pandas.core.frame
import time:      7623 |       7623 |         pandas.core.groupby.base
import time:       796 |        796 |             backports_abc
import time:      4543 |       5339 |           pandas._libs.groupby
import time:      1087 |       1087 |             pandas.core.groupby.categorical
import time:      3269 |       4356 |           pandas.core.groupby.grouper
import time:      4735 |      14429 |         pandas.core.groupby.ops
import time:      1364 |       1364 |           pandas.core.groupby.numba_
import time:      2375 |       2375 |           pandas.core.groupby.indexing
import time:     15050 |      18788 |         pandas.core.groupby.groupby
import time:     14694 |     512897 |       pandas.core.groupby.generic
import time:      1381 |     514278 |     pandas.core.groupby
import time:      2502 |     986011 |   pandas.core.api
import time:      1105 |       1105 |   pandas.tseries.api
import time:      1267 |       1267 |           pandas.core.computation.common
import time:      1592 |       2858 |         pandas.core.computation.align
import time:      2152 |       2152 |             pprint
import time:      2804 |       4955 |           pandas.core.computation.scope
import time:      3164 |       8118 |         pandas.core.computation.ops
import time:      1976 |      12951 |       pandas.core.computation.engines
import time:      1188 |       1188 |         pandas.core.computation.parsing
import time:      7429 |       8616 |       pandas.core.computation.expr
import time:      1630 |      23196 |     pandas.core.computation.eval
import time:      1127 |      24322 |   pandas.core.computation.api
import time:      1548 |       1548 |     pandas.core.reshape.encoding
import time:      1900 |       1900 |         _uuid
import time:      2879 |       4778 |       uuid
import time:      6286 |      11064 |     pandas.core.reshape.merge
import time:      5244 |       5244 |     pandas.core.reshape.pivot
import time:      1894 |       1894 |     pandas.core.reshape.tile
import time:      1786 |      21534 |   pandas.core.reshape.api
import time:      1393 |       1393 |     pandas.api.extensions
import time:      1210 |       1210 |     pandas.api.indexers
import time:      1072 |       1072 |         pandas.core.interchange
import time:     12274 |      13345 |       pandas.core.interchange.dataframe_protocol
import time:      1396 |       1396 |         pandas.core.interchange.utils
import time:      2052 |       3447 |       pandas.core.interchange.from_dataframe
import time:      1304 |      18095 |     pandas.api.interchange
import time:      1076 |       1076 |       pandas.core.dtypes.api
import time:      1589 |       2665 |     pandas.api.types
import time:     50906 |      50906 |       pandas.core.resample
import time:      1445 |       1445 |             pandas._libs.json
import time:      1505 |       1505 |             pandas.io.json._normalize
import time:      1746 |       1746 |             pandas.io.json._table_schema
import time:       827 |        827 |                     backports_abc
import time:      4204 |       5031 |                   pandas._libs.parsers
import time:      3859 |       3859 |                     pandas.io.parsers.base_parser
import time:      2002 |       5861 |                   pandas.io.parsers.arrow_parser_wrapper
import time:      1897 |       1897 |                   pandas.io.parsers.c_parser_wrapper
import time:      3676 |       3676 |                   pandas.io.parsers.python_parser
import time:     13655 |      30118 |                 pandas.io.parsers.readers
import time:      1360 |      31478 |               pandas.io.parsers
import time:       273 |      31751 |             pandas.io.parsers.readers
import time:      7243 |      43688 |           pandas.io.json._json
import time:      1497 |      45184 |         pandas.io.json
import time:       283 |      45467 |       pandas.io.json._json
import time:      9918 |       9918 |       pandas.io.stata
import time:      1568 |     107857 |     pandas.api.typing
import time:      2302 |     133520 |   pandas.api
import time:      3385 |       3385 |           tempfile
import time:      1946 |       5331 |         pandas._testing.contexts
import time:      1611 |       6941 |       pandas._testing._io
import time:      1524 |       1524 |       pandas._testing._warnings
import time:       465 |        465 |           cmath
import time:      1808 |       2273 |         pandas._libs.testing
import time:      3054 |       5326 |       pandas._testing.asserters
import time:      1324 |       1324 |       pandas._testing.compat
import time:      3210 |      18324 |     pandas._testing
import time:      1377 |      19701 |   pandas.testing
import time:      1465 |       1465 |   pandas.util._print_versions
import time:      1369 |       1369 |     pandas.io.clipboards
import time:      1470 |       1470 |         pandas.io.excel._util
import time:      2144 |       2144 |         pandas.io.excel._calamine
import time:      2485 |       2485 |         pandas.io.excel._odfreader
import time:      2799 |       2799 |         pandas.io.excel._openpyxl
import time:      2462 |       2462 |         pandas.io.excel._pyxlsb
import time:      2913 |       2913 |         pandas.io.excel._xlrd
import time:     10887 |      25158 |       pandas.io.excel._base
import time:      2105 |       2105 |       pandas.io.excel._odswriter
import time:      2449 |       2449 |       pandas.io.excel._xlsxwriter
import time:      1844 |      31554 |     pandas.io.excel
import time:      2763 |       2763 |     pandas.io.feather_format
import time:      1579 |       1579 |     pandas.io.gbq
import time:      7067 |       7067 |     pandas.io.html
import time:      1934 |       1934 |     pandas.io.orc
import time:      6841 |       6841 |     pandas.io.parquet
import time:      2089 |       2089 |       pandas.compat.pickle_compat
import time:      3761 |       5850 |     pandas.io.pickle
import time:      5196 |       5196 |       pandas.core.computation.pytables
import time:     14149 |      19345 |     pandas.io.pytables
import time:      2209 |       2209 |       pandas.io.sas.sasreader
import time:      1355 |       3563 |     pandas.io.sas
import time:      1304 |       1304 |     pandas.io.spss
import time:      5817 |       5817 |     pandas.io.sql
import time:      5588 |       5588 |     pandas.io.xml
import time:      2715 |      97281 |   pandas.io.api
import time:      1346 |       1346 |   pandas.util._tester
import time:      1266 |       1266 |   pandas._version_meson
import time:      4712 |    2070152 | pandas

@rhshadrach
Copy link
Member

@carbon-starlight - the import times you posted says pandas is importing in 2 seconds.

@carbon-starlight
Copy link

That's strange: importing pandas from command line seems to be fast, but running it from inside script is slow

Code:

print('Importing pandas...')
timestamp1 = time.time()
import pandas as pd
print(f"Importing took {time.time() - timestamp1}{' f#cking' if time.time() - timestamp1 > 10 else ''} seconds")

Terminal:

Importing pandas...
Importing took 113.06395554542542 f#cking seconds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Build Library building on various platforms Internals Related to non-user accessible pandas implementation Performance Memory or execution speed performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pyrito @rhshadrach @lithomas1 @carbon-starlight