-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fully implement the proto3 JSON mapping spec #72
Comments
* Rewrote JSON support to better implement the proto3 JSON spec. This includes support for the `json_name` protobuf option. * Added `JsonConfig` class that can be used for configuring JSON marshalling/unmarshalling at runtime. Currently supported configuration includes `ignoreUnknownFieldsInInput`, `outputDefaultValues`, and `outputProtoFieldNames`, which match the options described at https://developers.google.com/protocol-buffers/docs/proto3#json_options. * Got rid of the need for projects using `pbandk` to use the `kotlinx-serialization` gradle plugin and add a dependency on the `kotlinx-serialization` library. The library is now an internal implementation detail of `pbandk`. * Moved all of the binary marshalling functionality and most of the binary unmarshalling functionality from the generated code to the runtime library. * Added `protoMarshal(OutputStream)`, `protoUnmarshal(InputStream)`, and `protoUnmarshal(ByteBuffer)` overloads on the JVM. * Fixed some bugs with handling of `packed` fields, marshalling of enums in Javascript, and base64 encoding/decoding in Javascript. Breaking API changes: * Changed `Message` so that it no longer needs to extend from itself. * Added `@PbandkInternal` and `@PublicForGeneratedCode` annotations on relevant portions of the public API that are only public for pbandk's internal use. * The below methods are now defined as extension methods rather than being part of the `Message` interface. Code that calls these methods will now need to import them first. * `Message.jsonMarshal()` * `Message.Companion.jsonUnmarshal(String)` * `Message.protoMarshal()` * `Message.Companion.protoUnmarshal(ByteArray)` * These two method overloads: * `Message.jsonMarshal(kotlinx.serialization.json.Json)` * `Message.Companion.jsonUnmarshal(kotlinx.serialization.json.Json, String)` have been replaced with: * `Message.jsonMarshal(pbandk.json.JsonConfig)` * `Message.Companion.jsonUnmarshal(String, pbandk.json.JsonConfig)` * These two method overloads: * `Message.protoMarshal(pbandk.Marshaller)` * `Message.Companion.protoUnmarshal(pbandk.Unmarshaller)` have been replaced with: * `Message.marshal(pbandk.MessageMarshaller)` * `Message.Companion.unmarshal(pbandk.MessageUnmarshaller)` * Replaced `Marshaller` and `Unmarshaller` interfaces with `MessageMarshaller` and `MessageUnmarshaller`, which are much simpler and function differently from the previous interfaces. * Removed `Sizer` and `Util` from the public API. * Removed `UnknownField` constructors and the `UnknownField.size()` method from the public API. * `MessageMap.entries` is now a `Set` instead of a `List`. Fixes #23, fixes #61. Portions of the proto3 JSON spec that still need to be implemented are tracked in #72.
* Rewrote JSON support to better implement the proto3 JSON spec. This includes support for the `json_name` protobuf option. * Added `JsonConfig` class that can be used for configuring JSON marshalling/unmarshalling at runtime. Currently supported configuration includes `ignoreUnknownFieldsInInput`, `outputDefaultValues`, and `outputProtoFieldNames`, which match the options described at https://developers.google.com/protocol-buffers/docs/proto3#json_options. * Got rid of the need for projects using `pbandk` to use the `kotlinx-serialization` gradle plugin and add a dependency on the `kotlinx-serialization` library. The library is now an internal implementation detail of `pbandk`. * Moved all of the binary marshalling functionality and most of the binary unmarshalling functionality from the generated code to the runtime library. * Added `protoMarshal(OutputStream)`, `protoUnmarshal(InputStream)`, and `protoUnmarshal(ByteBuffer)` overloads on the JVM. * Fixed some bugs with handling of `packed` fields, marshalling of enums in Javascript, and base64 encoding/decoding in Javascript. Breaking API changes: * Changed `Message` so that it no longer needs to extend from itself. * Added `@PbandkInternal` and `@PublicForGeneratedCode` annotations on relevant portions of the public API that are only public for pbandk's internal use. * The below methods are now defined as extension methods rather than being part of the `Message` interface. Code that calls these methods will now need to import them first. * `Message.jsonMarshal()` * `Message.Companion.jsonUnmarshal(String)` * `Message.protoMarshal()` * `Message.Companion.protoUnmarshal(ByteArray)` * These two method overloads: * `Message.jsonMarshal(kotlinx.serialization.json.Json)` * `Message.Companion.jsonUnmarshal(kotlinx.serialization.json.Json, String)` have been replaced with: * `Message.jsonMarshal(pbandk.json.JsonConfig)` * `Message.Companion.jsonUnmarshal(String, pbandk.json.JsonConfig)` * These two method overloads: * `Message.protoMarshal(pbandk.Marshaller)` * `Message.Companion.protoUnmarshal(pbandk.Unmarshaller)` have been replaced with: * `Message.marshal(pbandk.MessageMarshaller)` * `Message.Companion.unmarshal(pbandk.MessageUnmarshaller)` * Replaced `Marshaller` and `Unmarshaller` interfaces with `MessageMarshaller` and `MessageUnmarshaller`, which are much simpler and function differently from the previous interfaces. * Removed `Sizer` and `Util` from the public API. * Removed `UnknownField` constructors and the `UnknownField.size()` method from the public API. * `MessageMap.entries` is now a `Set` instead of a `List`. Fixes #23, fixes #61. Portions of the proto3 JSON spec that still need to be implemented are tracked in #72.
We accidentally forgot to re-enable these in `pbandk.conformance.doTestIo()` after we added full JSON support to pbandk. Also enable the "recommended" tests since we're already passing most of the ones that aren't related to JSON. After enabling the tests, fixed a few bugs in the JSON code that the tests exposed. The remaining failing tests were added to the `failing_tests.txt` exception list and also to the checklist in #72.
We accidentally forgot to re-enable these in `pbandk.conformance.doTestIo()` after we added full JSON support to pbandk. Also enable the "recommended" tests since we're already passing most of the ones that aren't related to JSON. After enabling the tests, fixed a few bugs in the JSON code that the tests exposed. The remaining failing tests were added to the `failing_tests.txt` exception list and also to the checklist in #72.
https://developers.google.com/protocol-buffers/docs/proto3#json This is not implemented yet. The behavior is dependent on the This is important so that disregarding of the options of the writer/reader, they stay compatible. |
@kolloch that quote is how the pbandk JSON parser is already implemented. The relevant code is here:
If you've found a case where pbandk is not doing the correct thing, can you please open a new issue with more details about how to reproduce the problem? |
From https://developers.google.com/protocol-buffers/docs/proto3#json.
Well-known types
Any
(see also Add special handling for theAny
protobuf type #63)Duration
Empty
FieldMask
Struct
/Value
/NullValue
/ListValue
Timestamp
:"9999-12-31T23:59:59.999999999Z"
fails on Kotlin/JS't'
and lowercase'z'
should fail on Kotlin/JVM and Kotlin/JS"0001-01-01T00:00:00Z"
to"9999-12-31T23:59:59.999999999Z"
) should fail on Kotlin/JVM and Kotlin/JSConformance test edge cases
more than one value for a
oneof
field in a message should be rejected (currently the last value in the JSON input wins)Failing conformance test:
base64-url support for
bytes
fields on Kotlin/JVM and Kotlin/JSFailing conformance tests:
Invalid Unicode surrogate pairs in
string
fields should be rejected (currently they're accepted and the invalid characters are replaced with?
)Failing conformance tests:
Recently-added conformance tests that still need to be triaged for root cause:
exponent notation and floating-point numbers for unmarshalling numeric types:
1e5
,100000.000
,2.147483647e9
,-2.147483648e9
,4.294967295e9
Failing conformance tests:
uint32
max value (4294967295
)uint64
max value (18446744073709549568
,"18446744073709549568"
)floating-point numbers outside of the min/max range for
float
/double
should be rejected (currently they're being read in as-Inf
/Inf
)Failing conformance tests:
unquoted
NaN
,Infinity
,-Infinity
values forfloat
/double
fields in JSON input should be rejected (currently they're accepted)Failing conformance tests:
The text was updated successfully, but these errors were encountered: