feat(bigquery/storage/managedwriter): improve protobuf support #4589
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
After some ongoing discussions with the Storage API team, this PR improves support for proto2/proto3 syntax in protocol buffer code. * Updates testdata so that we have a proto2 and proto3 form of our SimpleMessage data. * Add reference schemas to testdata. * Updates proto conversion code in the adapt package so it's creating proto2 messages by default. The code paths for doing proto3 conversions are present, but not exported yet as the storage API doesn't properly handle proto3 expectations for conversion. Namely, conversion doesn't properly account for default values and use of wrapper types. * Adds benchmarks for dynamic schema generation and static serialization, to aid in some internal discussions.
product-auto-label
bot
added
the
api: bigquery
google-cla
bot
added
the
cla: yes
shollyman
added
the
do not merge
tswast
reviewed
Aug 12, 2021
| {Name: "four", Value: 1}, | ||
| {Name: "five", Value: 2}, | ||
| var testSimpleData = []*testdata.SimpleMessageProto2{ | ||
| {Name: proto.String("one"), Value: proto.Int64(1)}, |
There was a problem hiding this comment.
Why do we need these proto.String and proto.Int64 wrapper types in proto2?
There was a problem hiding this comment.
The structs take pointers to strings rather than strings directly, the proto.String, proto.Int64 etc funcs are just wrappers to take a value and return a pointer to the value.
shollyman
removed
the
do not merge
codyoss
approved these changes
Aug 13, 2021
bigquery/storage/managedwriter/benchmarks/protobenchmarks_test.go
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After some ongoing discussions with the Storage API team, this PR
improves support for proto2/proto3 syntax in protocol buffer code.
Updates testdata so that we have a proto2 and proto3 form of
our SimpleMessage data.
Add reference schemas to testdata.
Updates proto conversion code in the adapt package so it's creating
proto2 messages by default, but supports generation of proto3 style
descriptors.
Adds benchmarks for dynamic schema generation and static
serialization, to aid in some internal discussions. The GithubArchive schema
has all fields fully nullable, so it should provide a good comparison when having
to represent data using well known wrapper types.
Towards: #4366
Benchmark results thus far: