Nikola help is really slow

[ralsina]

It takes about 1.5 seconds on my machine... that should be much faster.

Actions taken:

• Don't load compiler plugins that are not needed.
• Don't load compiler extension plugins not needed.
• Don't load DISABLED_PLUGINS
• New PluginCategory metadata is used to not load compilers that are not in COMPILERS
• Only load Command plugins when using "nikola help"

This cut down (in my admittedly fast machine) startup from 0.7 seconds to <0.3

Should be enough for this shot.

[ralsina]

Current hypothesis:

Loading a bunch of plugins manully via the imp module like Yapsy does is slower than loading it via normal imports. A default installation of Nikola has ~60 modules, which means this is noticeable.

[Kwpolska]

Better hypothesis: loading 10k SLOC is really slow.

[schettino72]

If plugins or even loading too much code is the problem... doit plugins are loaded on demand http://pydoit.org/extending.html#plugins

[ralsina]

Sadly migrating now off Yapsy and to another plugin mechanism feels like a bad idea.

Maybe we should special-case help so it only loads the command plugins.

[Kwpolska]

Crazy and (probably) impossible optimization idea: don’t load any plugins unless we need something from that plugin. Instead, (for help) just read the .plugin files, which should hold the one-line description that normally appears.

And if we do nikola help bootswatch_theme? Load only that command. Same if we actually want to execute the command.

[Kwpolska]

@tibonihoo Do you have any ideas? Is there any way to get rid of .plugin files? Or to speed things up?

[tibonihoo]

You'd need a specific pluginmanader decorator for that but it seems clearly feasible as plugin drtection and loading happens in two separate steps. Considering my current latency on my open projects like yapsy I may not be of a great help though (maybe I'll have a "window" in 1 weeks though).

[ralsina]

Actually looking at things with a profiler shows we can bring this down without touching yapsy:

   - collectPlugins (yapsy.PluginManager:533)                                                                                                                                      64.1%  0.00%      1     18.91s 18.91s         41     41
    - loadPlugins (yapsy.PluginManager:445)                                                                                                                                        61.1%  0.27%      1     18.00s 18.00s     78.949 78.949
     + yapsy_loaded_plugin_ipynb_0                                                                                                                                                 23.3%  0.00%      1      6.86s  6.86s        848    848
     + yapsy_loaded_plugin_rest_post_list_0                                                                                                                                        17.3%  0.00%      1      5.09s  5.09s        485    485
     + yapsy_loaded_plugin_import_feed_0                                                                                                                                           8.90%  0.00%      1      2.62s  2.62s      1.398  1.398
     + yapsy_loaded_plugin_import_wordpress_0                                                                                                                                      2.92%  0.00%      1    859.355 59.355      1.127  1.127
     + yapsy_loaded_plugin_markdown_0                                                                                                                                              2.33%  0.00%      1    685.777 85.777      1.265  1.265
     + yapsy_loaded_plugin_create_bundles_0                                                                                                                                        1.99%  0.00%      1    585.432 85.432        326    326
     + yapsy_loaded_plugin_rest_chart_0                                                                                                                                            1.61%  0.00%      1    473.519 73.519        385    385
     + yapsy_loaded_plugin_rest_0                                                                                                                                                  0.85%  0.00%      1    251.162 51.162      1.407  1.407
     + yapsy_loaded_plugin_jinja_0                                                                                                                                                 0.79%  0.00%      1    233.339 33.339        428    428
     + yapsy_loaded_plugin_auto_0                                                                                                                                                  0.15%  0.01%      1     44.850 44.850      2.422  2.422
     + yapsy_loaded_plugin_rest_listing_0                                                                                                                                          0.14%  0.00%      1     41.772 41.772        557    557
     + yapsy_loaded_plugin_serve_0                                                                                                                                                 0.13%  0.00%      1     37.307 37.307        823    823
     + yapsy_loaded_plugin_render_listings_0                                                                                                                                       0.11%  0.00%      1     33.476 33.476        371    371
     + yapsy_loaded_plugin_rest_media_0                                                                                                                                            0.10%  0.00%      1     28.464 28.464        357    357
     + yapsy_loaded_plugin_scale_images_0                                                                                                                                          0.09%  0.00%      1     25.700 25.700        163    163
     + yapsy_loaded_plugin_plugin_0                                                                                                                                                0.04%  0.00%      1     11.475 11.475        174    174

So, ipynb and post_list plugins account for about 40% of startup time.

[ralsina]

We are activating compiler plugins that we should not. We are looking at COMPILERS.keys() but compiler plugins that match nothing in POSTS/PAGES should not be enabled.

Merged: don't load a bunch of plugins.

[ralsina]

Next idea: only load the needed kind of plugins:

1. For nikola help -> load command plugins
2. For nikola help command -> load that one command
3. For nikola build -> don't load commands (not all that important, this one is naturally slower)