Faster JSON encoder/decoder #459
Conversation
|
Is there a way(I did not read changes thoroughly) to enable sorting and indentation, e.g. for testing purposes? |
|
Why would you want to sort or indent the raw output? The extra verbose output of |
|
Right, as long as ipa CLI is capable to print formatted debug output, that's enough. |
|
It's usually quicker to read raw response in browser than the folded "preview" because everything is visible and no clicking is required. Same for curl testing. But for curl I can imagine piping it to some tool. |
|
|
|
As mention on meeting, if rpcserver prettyprints into output in debug mode then it is fine. |
|
@pvoborni I have modified the PR and added a pretty_print option. JSON is now pretty printed for verbose level 2 and higher. The old implementation converted all list to tuples. With |
| """ | ||
| __slots__ = ('version', '_cap_datetime', '_cap_dnsname') | ||
|
|
||
| _identity = object() |
There was a problem hiding this comment.
Could you please add a comment explaining that _identity = object() with if func is _identity then arg else func(arg) is faster than _identity = lambda x: x with func(arg) so that no one comes in the future and replaces the former with the latter because it's duplicate code?
| :see: _ipa_obj_hook, _JSONPrimer | ||
| """ | ||
| if isinstance(val, bytes): | ||
| val = val.decode('utf-8') |
There was a problem hiding this comment.
IMO decoding the value should be the caller's responsibility.
There was a problem hiding this comment.
It should be handled by the JSON library if possible. The approach allows us to replace the JSON library with a more efficient one without breaking API.
|
LGTM |
| :note: pretty printing triggers a slow path in Python's JSON module. Only | ||
| use pretty_print in debug mode. | ||
| """ | ||
| result = _JSONPrimer(version).convert(val) |
There was a problem hiding this comment.
Wouldn't it be a little bit faster if we kept the type - encoder mapping in a global object initialized once rather than reconstructing it in every json_encode_binary() call?
There was a problem hiding this comment.
Yes, it might be a tiny bit more efficient, but also more complicated. I would have to find another way to handle client versions difference efficiently. It's a constant startup cost any way. __hash__ and __eq__ are efficient.
MartinBasti
changed the title
Improve performance of FreeIPA's JSON serializer and deserializer. * Don't indent and sort keys. Both options trigger a slow path in Python's json package. Without indention and sorting, encoding mostly happens in optimized C code. * Replace O(n) type checks with O(1) type lookup and eliminate the use of isinstance(). * Check each client capability only once for every conversion. * Use decoder's obj_hook feature to traverse the object tree once and to eliminate calls to isinstance(). Closes: https://fedorahosted.org/freeipa/ticket/6655 Signed-off-by: Christian Heimes <cheimes@redhat.com>
Some tests assume that JSON deserializier returns tuples instead of lists. I don't think it is necessary but let's pass the tests for now. Signed-off-by: Christian Heimes <cheimes@redhat.com>
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Signed-off-by: Christian Heimes <cheimes@redhat.com>
Improve performance of FreeIPA's JSON serializer and deserializer.
Python's json package. Without indention and sorting, encoding
mostly happens in optimized C code.
the use of isinstance().
to eliminate calls to isinstance().
Closes: https://fedorahosted.org/freeipa/ticket/6655
Signed-off-by: Christian Heimes cheimes@redhat.com