[performance] XBMC plugins are compiled each time they are used

[bitplane]

XBMC plugins are compiled every time they are executed (no pyo file is created upon execution), while pyo files distributed with plugins are ignored. This means that large plugins on slower/embedded systems have a compilation overhead on directory navigation, in the order of several seconds.

To prove this, create a file with a compile-time warning and notice how the warning is logged every time the plugin is executed. For example:

def log_compile_warning():
    from time import *

If this is in your plugin's main file then you'll see a SyntaxWarning in xbmc.log every time the plugin is executed. If it's imported by your plugin's main file then it will only be logged the first time the plugin is executed.

[sraue]

i investigated 2 years ago this "issue", python ALWAYS compiles py files to pyc or pyo, even if pyo or pyc exist, to avoid this you must remove the py files, this is what we do in our python packages in system to speedup python (at least on rpi this is significant). for addons we dont have control, the user should/must remove py files byself after pyo's are created to benefit here

[MilhouseVH]

I'm seeing the repeated compilation problem with default.py, but correct single compilation (and pyo generation) with all other modules (testing on R-Pi with latest OE master).

Taking the BBC iPlayer addon as an example, default.py is being repeatedly compiled - the above compile-time warning is logged every time the addon is entered. A pyo is never created, so the default.py is being compiled every time.

However, a lower level module in this addon, eg. utils.py (which is imported by default.py) is being compiled to pyo and the module logs the compile warning only once, when the pyo is first created. Subsequent invocations of the addon don't report another compile time warning, so presumably the compiled utils.pyo is now being used.

Looking at my installed addons, it seems that hardly any have a compiled default.pyo, but all of the modules that these addons import are being compiled to pyo.

If I force the compilation of default.py to pyo with python -m default.py, a default.pyo is generated, but deleting default.py then running the addon results in an error:

15:09:27 1473.125488 T:2780279888   ERROR: Unable to run plugin iPlayer
15:09:27 1473.126221 T:3059396608   ERROR: GetDirectory - Error getting plugin://plugin.video.iplayer/?content_type=video
15:09:27 1473.133423 T:3059396608   ERROR: CGUIMediaWindow::GetDirectory(plugin://plugin.video.iplayer/?content_type=video) failed

so it looks like the Python interpreter on OpenELEC (or maybe it's an XBMC issue? Is this the same on non-OE builds?) is not configured to create and use a pre-compiled default.py.

[popcornmix]

I tried iplayer with xbmc running on raspbian. I deleted all pyo files first. These are iplayer py files:

.xbmc/addons/plugin.video.iplayer/lib/stations.py
.xbmc/addons/plugin.video.iplayer/lib/httplib2/__init__.py
.xbmc/addons/plugin.video.iplayer/lib/httplib2/socks.py
.xbmc/addons/plugin.video.iplayer/lib/httplib2/iri2uri.py
.xbmc/addons/plugin.video.iplayer/lib/iplayer2.py
.xbmc/addons/plugin.video.iplayer/lib/listparser.py
.xbmc/addons/plugin.video.iplayer/lib/iplayer_search.py
.xbmc/addons/plugin.video.iplayer/utils.py
.xbmc/addons/plugin.video.iplayer/default.py

This is what I see after using iplayer:

.xbmc/addons/plugin.video.iplayer/lib/listparser.pyo
.xbmc/addons/plugin.video.iplayer/lib/httplib2/socks.pyo
.xbmc/addons/plugin.video.iplayer/lib/httplib2/__init__.pyo
.xbmc/addons/plugin.video.iplayer/lib/httplib2/iri2uri.pyo
.xbmc/addons/plugin.video.iplayer/lib/iplayer2.pyo
.xbmc/addons/plugin.video.iplayer/lib/stations.pyo
.xbmc/addons/plugin.video.iplayer/lib/iplayer_search.pyo
.xbmc/addons/plugin.video.iplayer/utils.pyo

Looks like I see the same thing of default.py not being compiled.

[MilhouseVH]

This would suggest an XBMC rather than OpenELEC issue?

[sraue]

i think its a general python problem:
if you have a python script which you run, it cant be a pyc/o file, you only can start a py file, but all files which are imported from this one script can be a pyc/o file

[popcornmix]

@MilhouseVH
If you compile default.py to default.pyo then rename that to python.py does it work?

I believe python can launch both py files and pyo files. Obviously this is ugly, but if it works then xbmc could be made to try launching default.pyo first, and falling back to python.py.

Also how long does it take to compile the largest default.py file?

The low-tech solution is to encourage add-on authors to move all code from default.py into a submodule that just gets imported).

[MilhouseVH]

If I rename default.pyo to python.py (and remove default.py) then I get the same error as before when starting the addon: Unable to run plugin iPlayer.

I agree that creating a "minimalist" default.py would seem to be the obvious solution, but can't see that flying amongst addon developers. not least because it doesn't entirely eliminate the problem - you're still wasting time (re-)compiling default.py no matter how trivial it might be.

[stefansaraev]

it's not a bug. default.py is special. I'll just close this ;)