Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to check if the cache plugin works? #2610

Open
markus2330 opened this Issue Apr 9, 2019 · 7 comments

Comments

Projects
None yet
2 participants
@markus2330
Copy link
Contributor

markus2330 commented Apr 9, 2019

Steps to Reproduce the Problem

kdb info cache says: "The cache plugin is compiled and enabled on compatible systems by default. No actions are needed to enable it."

kdb mount
#>
kdb mount /large/file.json system/json/test json 

Expected Result

That kdb ls / will be faster the second time. And that kdb gmount cache fails (like it is the case for spec).

Actual Result

time kdb ls / > /dev/null
#> kdb ls / > /dev/null  0,73s user 0,02s system 94% cpu 0,795 total
time kdb ls / > /dev/null
#> kdb ls / > /dev/null  0,82s user 0,01s system 98% cpu 0,840 total
kdb gmount
#> 
kdb gmount cache
#> 
kdb gmount
#> spec
#> cache

System Information

  • Elektra Version: master

@markus2330 markus2330 added the question label Apr 9, 2019

@mpranj mpranj added this to the 0.8.27 milestone Apr 9, 2019

@mpranj

This comment has been minimized.

Copy link
Member

mpranj commented Apr 9, 2019

Currently, there is no easy way to detect cache hits. Maybe we could add a way of checking for cache hit (adding a meta key to the parent, or enforcing cache by a custom command kdb cache-get).

Another problem here is that the cache is not mounted in the usual fashion (using list). The idea here was to avoid overhead. We can either mount it with list as usual (make it gmount compatible), or add a custom command (kdb cache enable, kdb cache disable).

Another handy thing would be kdb cache clear to remove all cached files.

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Apr 9, 2019

Thank you for the answer! Do you have any idea why the parsing of JSON did not speed up in the example above? Can you give an example where the cache benefit is clearly shown?

@mpranj

This comment has been minimized.

Copy link
Member

mpranj commented Apr 9, 2019

Testing it with / can be problematic for many reasons. A cache miss could easily be caused by non-file based storage plugins (#2454), mounted but non-existent config files (resolver says NO_UPDATE, regardless whether it is up to date or the file does not exist) ...

It might just work if you try getting system/json/test.

Another way to test it out is the benchmark I added (./bin/benchmark_kdb).

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Apr 9, 2019

It might just work if you try getting system/json/test.

I tried getting several mountpoints (with kdb ls) which are big enough that the cache could pay off. Unfortunately I did not see any improvement with any of them. It also seems like that ~/.cache/elektra is not written to anymore. What exactly does kdb gmount needs to show so that it is present?

The benchmark works but it does not give me any comparison:

./bin/benchmark_kdb
plugin;operation;microseconds
core;kdbOpen;212
core;kdbGet;86
core;kdbSet;44
core;kdbOpen;93
core;kdbGet;57
core;kdbOpen;74
core;kdbGet;26
core;kdbOpen;71
core;kdbGet;25
core;kdbOpen;40
core;kdbGet;55
core;kdbOpen;39
core;kdbGet;24
core;kdbOpen;39
core;kdbGet;24
core;kdbOpen;40
core;kdbGet;24
@mpranj

This comment has been minimized.

Copy link
Member

mpranj commented Apr 9, 2019

Something is definitely off here. The benchmark inserts 40k keys, the first kdbGet is slow and then we should have only cache hits. This is how it looks on my machine:

./bin/benchmark_kdb 
plugin;operation;microseconds
core;kdbOpen;6707
core;kdbGet;758
core;kdbSet;171587
core;kdbOpen;449
core;kdbGet;134177
core;kdbOpen;406
core;kdbGet;12351
core;kdbOpen;356
core;kdbGet;11514
core;kdbOpen;379
core;kdbGet;12184
core;kdbOpen;361
core;kdbGet;11556
core;kdbOpen;377
core;kdbGet;12263
core;kdbOpen;367
core;kdbGet;11444

Note that I have only the default mountpoint. It seems that in your case even the set has failed, or maybe even kdbOpen.

I don't know what's going on here. 😕

@markus2330

This comment has been minimized.

Copy link
Contributor Author

markus2330 commented Apr 9, 2019

I think we need some debugging mechanisms for the cache. Maybe like you suggested before: that the cache adds some meta data to the parent key and the kdb command prints it on request.

I think for now you can simply add a warning if the cache does not work (together with a reason why it did not.) Later we might introduce logging/info messages (even less severity of warning, only displayed with -v).

@markus2330 markus2330 referenced this issue Apr 9, 2019

Open

Error Concept Improvements Metaissue #2572

1 of 13 tasks complete
@mpranj

This comment has been minimized.

Copy link
Member

mpranj commented Apr 9, 2019

That makes sense to me too. Up to now I painfully tracked/debugged it using the logger.

Warnings on a cache miss are definitely not usable for production, but info severity would be great for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.