Speed up client schema cache #488
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
self-assigned this
|
@tiran Currently the file is first copied into BytesIO and then all reading is done from it. Your modification IMO supersedes the need for the BytesIO copy because everything is read into memory at once. Could you remove it? |
It's inefficient to open a zip file over and over again. By loading all
members of the schema cache file at once, the ipa CLI script starts
about 25 to 30% faster for simple cases like help and ping.
Before:
$ time for i in {1..20}; do ./ipa ping >/dev/null; done
real 0m13.608s
user 0m10.316s
sys 0m1.121s
After:
$ time for i in {1..20}; do ./ipa ping >/dev/null; done
real 0m9.330s
user 0m7.635s
sys 0m1.146s
https://fedorahosted.org/freeipa/ticket/6690
Signed-off-by: Christian Heimes <cheimes@redhat.com>
The schema cache used a BytesIO buffer to read/write schema cache before it got flushed to disk. Since the schema cache is now loaded in one go, the temporary buffer is no longer needed. File locking has been replaced with a temporary file and atomic rename. Signed-off-by: Christian Heimes <cheimes@redhat.com>
tiran
force-pushed
the
cli_fast_schema_cache
branch
from
26d3b96
to
98f5960
Compare
|
@dkupka Makes sense, I dropped the temporary buffer and replaced the file locking logic with tempfile + os.rename. |
|
The speedup I see is smaller (10-15%) [1] than what you're reporting but that might be caused by the fact that I store the cache on really slow file system (NFS mount). Anyway the changes makes sense, thanks. |
ghost
mentioned this pull request
|
It looks like your IPA server is about half as fast (26sec / 13sec for 20 pings). In absolute numbers, it's still ~2.5 sec faster. In your case, performance probably dominated by server latency. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It's inefficient to open a zip file over and over again. By loading all
members of the schema cache file at once, the ipa CLI script starts
about 25 to 30% faster for simple cases like help and ping.
Before:
After:
https://fedorahosted.org/freeipa/ticket/6690
Signed-off-by: Christian Heimes cheimes@redhat.com