Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

peformance problem with pluginprocess #2209

Closed
markus2330 opened this Issue Aug 24, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@markus2330
Copy link
Contributor

markus2330 commented Aug 24, 2018

Steps to Reproduce the Problem

  1. have a huge KeySet
  2. globally mount some plugin using gmount and then interact with KDB:
kdb gmount python script=/path/to/elektra/src/plugins/python/python/python_filter.py
kdb get /something

Expected Result

Should be about same speed.

Actual Result

It is about 10x slower, from 2sec -> 20sec for my keys.

System Information

  • Elektra Version: master

Further Input

@mpranj Your global mmap would be the perfect fix because in this case, where I have this problem, basically exclusively kdb get is done. (kdb set happens only at night in cron jobs.) With global mmap no pluginprocess or python would be involved, and not even the ini plugin, so we should be even below 2 sec.

Nevertheless, I am interested if exchanging the dump plugin in ~e/src/libs/pluginprocess/pluginprocess.c with mmapstorage also improves the situation. 20 -> 10 seconds would already be great 👍

I only did a short benchmark and found out that the python plugin causes the overhead and that the python code itself does not make a difference (e.g. if I immediately return some value). So I suspected the serialization. I am, however, not sure about that. So please do not invest too much time into it. If you find some way to finish mmap global plugin in September you can simply forget this task 😄

If this improves the situation I have a big motivation to quickly merge your #1990 PR 😉

@markus2330 markus2330 changed the title peformanceproblem with pluginprocess peformance problem with pluginprocess Aug 24, 2018

@mpranj

This comment has been minimized.

Copy link
Member

mpranj commented Aug 24, 2018

#1990 is mmapstorage and not yet tested as a global plugin. It is, however, nearly finished and working well as a default storage plugin. I feel like it can almost be merged, I am just testing a few last things.

I would like to merge #1990 as a storage plugin. After that, the plan is to make minor modifications to the global-plugins and mmapstorage. Mmapstorage only needs a mode where, as a global plugin, it uses a predefined cache file to store the complete KeySet. If everything works as planned, there will be no difference between mmapstorage and the global mmap cache (except the file path).

It will definitely be finished in September.

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Aug 24, 2018

For what I propose here, it would be used as storage plugin as serialization of KeySet(s) for inter-process communication, not as global plugin.

sed -i 's/dump/mmap/' src/libs/pluginprocess/pluginprocess.c

should be the only necessary code modification. You can have this little commit within #1990.

If it does not work or is not faster we might learn something nevertheless. (Maybe it is unsuitable for streaming.)

If everything works as planned, there will be no difference between mmapstorage and the global mmap cache (except the file path).

Yes, of course you can use the mmapstorage plugin within your mmapcache plugin. This makes a lot of sense 😄

@mpranj

This comment has been minimized.

Copy link
Member

mpranj commented Aug 24, 2018

I just looked into this. Unfortunately, it is not as easy because pluginprocess uses pipes and mmap can not work with pipes. (at least until this thesis from 2000 becomes popular)

The simple solution:
Use the global cache when it is done and ignore any underlying problems.

The complex solution:
Write a new plugin reusing large parts of the mmapstorage, but replacing mmap() with a buffer and using fread/fwrite to work with the pipe.

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Aug 25, 2018

Thank you for looking into it! Then I hope for the simple solution in September 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.