Kedro initialization is slow

[astrojuanlu]

(Related: #2879)

I've noticed that Kedro takes some time to initialize itself, which adds to the time needed to run the actual command.

For example, kedro info on a spaceflights project takes a bit more than 6 seconds on my machine (M1 MacBook Pro from 2022):

$ time kedro info
 _            _
| | _____  __| |_ __ ___
| |/ / _ \/ _` | '__/ _ \
|   <  __/ (_| | | | (_) |
|_|\_\___|\__,_|_|  \___/
v0.18.13

Kedro is a Python framework for
creating reproducible, maintainable
and modular data science code.

Installed plugins:
kedro_telemetry: 0.2.5 (entry points:cli_hooks,hooks)
kedro_viz: 6.5.0 (entry points:global,hooks,line_magic)

________________________________________________________
Executed in    6.27 secs    fish           external
   usr time    1.15 secs    0.26 millis    1.15 secs
   sys time    0.67 secs    2.65 millis    0.67 secs

After warmup (i.e. executing the command again immediately after) this can go down to ~1 second.

Ahead of this I did some profiling using https://github.com/joerick/pyinstrument and https://www.speedscope.app/, and this is what I found:

For kedro info with instrumentation on and no warmup, 7.5 seconds were spent importing the various kedro.framework.cli.* subpackages, and 1.7 seconds actually executing the function.

This can become a problem in situations when low latency is desired.

[noklam]

#1476
Reference, noted 3 is solved now because we remove pkg_resource, 1 is partly solved with some optimisation in kedro-viz

[noklam]

Try python - X importtime to isolate the module import time.

The warmup time is interesting, could it be the first time you run the project and it spends time on compiling the bytecode? Can you try deleting the pyc files?

[astrojuanlu]

Sorry I forgot about #1476, I'll copy my comment there tomorrow.

[datajoely]

Do you think we could build this into kedro info so users could ship us a report we could work with?

[astrojuanlu]

Would be rad to have this and probably more things under a kedro info --full, kedro profile, kedro debug, or something like that, indeed. Or at least tell people how to profile their kedro commands.