Protobuf packed encoding/decoding #1830
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pdvrieze
force-pushed
the
packedProtobuf
branch
from
e6f2692
to
72dedf3
Compare
pdvrieze
force-pushed
the
packedProtobuf
branch
3 times, most recently
from
fdcd95a
to
a0b9502
Compare
sandwwraith
requested changes
Jan 28, 2022
formats/protobuf/commonMain/src/kotlinx/serialization/protobuf/ProtoTypes.kt
Outdated
Show resolved
Hide resolved
formats/protobuf/commonMain/src/kotlinx/serialization/protobuf/internal/Helpers.kt
Outdated
Show resolved
Hide resolved
formats/protobuf/commonMain/src/kotlinx/serialization/protobuf/internal/PackedArrayDecoder.kt
Outdated
Show resolved
Hide resolved
formats/protobuf/commonMain/src/kotlinx/serialization/protobuf/internal/PackedArrayDecoder.kt
Outdated
Show resolved
Hide resolved
formats/protobuf/commonMain/src/kotlinx/serialization/protobuf/internal/ProtobufDecoding.kt
Outdated
Show resolved
Hide resolved
formats/protobuf/commonMain/src/kotlinx/serialization/protobuf/internal/ProtobufDecoding.kt
Show resolved
Hide resolved
formats/protobuf/commonTest/src/kotlinx/serialization/protobuf/PackedArraySerializerTest.kt
Show resolved
Hide resolved
formats/protobuf/commonTest/src/kotlinx/serialization/protobuf/PackedArraySerializerTest.kt
Outdated
Show resolved
Hide resolved
… 1, 2 or 5) to allow decoders decode either format independently of the proto specification. Make sure to test this flexibility in the parser.
…t to be used to effectively peek the type without assumptions on state.
…ruct methods to be in line with other list handling. It also ensures consistent reading of run lengths and means that the encoders/decoders don't need to handle the initial structure start (and writing can handle the writing of the subBuffer automatically at the end of the structure when the run length is known).
…ted structures - invalid in packing).
…/ProtoTypes.kt Add link to packed Co-authored-by: Leonid Startsev <sandwwraith@users.noreply.github.com>
…that outputs of the serializer should never hit this exception.
…ports inputs in either format, independent of the annotation. The annotation only affects writing.
… of strings and "packed" toplevel arrays. The checking for eof works as bytesize is always >= array length.
pdvrieze
force-pushed
the
packedProtobuf
branch
from
f2950fd
to
e3a5107
Compare
sandwwraith
approved these changes
Feb 2, 2022
|
@sandwwraith is this going to be included in the next release? |
|
@achojoao Yes, it is merged to dev already. |
pdvrieze
added a commit
to pdvrieze/kotlinx.serialization
that referenced
this pull request
Apr 29, 2022
* Create an annotation to request packing of collections. * The specification only allows packing for primitive types (wire types 1, 2 or 5) to allow decoders decode either format independently of the proto specification. * Make pushback work in respect to currentType/currentId. This allows it to be used to effectively peek the type without assumptions on state. * Clarify in the documentation that reading will (per the standard) supports inputs in either format, independent of the annotation. The annotation only affects writing. * Support decoding "packed" arrays as toplevels. Add tests for handling of strings and "packed" toplevel arrays. The checking for eof works as bytesize is always >= array length.
|
@sandwwraith Any news on when the new version of the library with this included is going to be released? |
|
I think we need a fresh release, so I'd try to publish it in the following days |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To address issue #1829 I've implemented support for protobuf packed fields. This is enabled using a new annotation.
Per the protocol it only supports primitive integers.
While there are some small updates in the libraries I've not attempted to move existing code paths over (the
RepeatedDecoderand
NestedRepeatedEncoderclasses etc. come to mind.