Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
peformance problem with pluginprocess #2209
Steps to Reproduce the Problem
kdb gmount python script=/path/to/elektra/src/plugins/python/python/python_filter.py kdb get /something
Should be about same speed.
It is about 10x slower, from 2sec -> 20sec for my keys.
@mpranj Your global mmap would be the perfect fix because in this case, where I have this problem, basically exclusively
Nevertheless, I am interested if exchanging the dump plugin in ~e/src/libs/pluginprocess/pluginprocess.c with mmapstorage also improves the situation. 20 -> 10 seconds would already be great
I only did a short benchmark and found out that the python plugin causes the overhead and that the python code itself does not make a difference (e.g. if I immediately return some value). So I suspected the serialization. I am, however, not sure about that. So please do not invest too much time into it. If you find some way to finish mmap global plugin in September you can simply forget this task
If this improves the situation I have a big motivation to quickly merge your #1990 PR
#1990 is mmapstorage and not yet tested as a global plugin. It is, however, nearly finished and working well as a default storage plugin. I feel like it can almost be merged, I am just testing a few last things.
I would like to merge #1990 as a storage plugin. After that, the plan is to make minor modifications to the global-plugins and mmapstorage. Mmapstorage only needs a mode where, as a global plugin, it uses a predefined cache file to store the complete KeySet. If everything works as planned, there will be no difference between mmapstorage and the global mmap cache (except the file path).
It will definitely be finished in September.
For what I propose here, it would be used as storage plugin as serialization of KeySet(s) for inter-process communication, not as global plugin.
should be the only necessary code modification. You can have this little commit within #1990.
If it does not work or is not faster we might learn something nevertheless. (Maybe it is unsuitable for streaming.)
Yes, of course you can use the mmapstorage plugin within your mmapcache plugin. This makes a lot of sense
I just looked into this. Unfortunately, it is not as easy because pluginprocess uses pipes and mmap can not work with pipes. (at least until this thesis from 2000 becomes popular)
The simple solution:
The complex solution: