Reduce memory overhead for TransportableObject #1883
+258
−181
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1883 +/- ##
===========================================
+ Coverage 84.60% 85.15% +0.54%
===========================================
Files 295 181 -114
Lines 14499 11275 -3224
Branches 195 0 -195
===========================================
- Hits 12267 9601 -2666
+ Misses 2098 1674 -424
+ Partials 134 0 -134
|
cjao
force-pushed
the
improve-transportable-overhead
branch
3 times, most recently
from
9e9caf9
to
3ebee90
Compare
cjao
force-pushed
the
improve-transportable-overhead
branch
2 times, most recently
from
3b05bad
to
e890bc5
Compare
|
Tests pass but codecov fails. |
cjao
force-pushed
the
improve-transportable-overhead
branch
3 times, most recently
from
1f56a35
to
e32764f
Compare
cjao
force-pushed
the
improve-transportable-overhead
branch
2 times, most recently
from
0bca7df
to
defd1eb
Compare
* Always represent TransportableObject internally as a single array of bytes. Various properties, such as `header`, or `object_string`, decode various segments of the byte array. * Store the serialized object as raw picklebytes without base64-encoding. As a result, `get_deserialized()` no longer needs to create a temporary copy of the raw picklebytes. The data segment is directly unpickled. Base64-encoding is applied to the data segment or the entire internal buffer whenever a print friendly representation of the `TransportableObject` is desired. * Since the properties of `TransportableObject` are simply views into the underlying buffer, `TransportableObject` may itself be serialized efficiently by simply writing out the byte array. ` Add backward compatibility layer for deserialization
cjao
force-pushed
the
improve-transportable-overhead
branch
from
4b4a905
to
90b869b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Always represent TransportableObject internally as a single array of bytes. Various properties, such as
header, orobject_string, decode various segments of the byte array. This allowsTransportableObjectto be serialized efficiently by simply writing out the byte array.Store the serialized object as raw picklebytes without base64-encoding. As a result,
get_deserialized()no longer needs to create a temporary copy of the raw picklebytes. The data segment is directly unpickled. Base64-encoding is applied to the data segment or the entire internal buffer whenever a print friendly representation of theTransportableObjectis desired. A backward compatibility layer is introduced to automatically handleTransportableObjects serialized in the earlier format.