feat(stdlib): Add proto_encode and parse_proto to encode and parse protocol buffers

[flavioc]

parse_proto accepts a bytes value, a proto descriptor file path and a message type and returns the VRL value as parsed from the proto.

parse_proto!(base64_decode("data"), "person.desc", "proto.Person") => {"id": 42, "name": "John Doe"}

For encode_proto, it does the reverse:

encode_proto!({"id": 42, "name": "John Doe"}, "person.desc", "proto.Person") => bytes data

Note that the path to descriptor file and message type are required to be literals. We do this so that during compilation we load the message descriptor and reuse it for encoding or parsing.

Most of the code was copied over from the proto codec support from vector. However, there's one notable difference. In parse_proto, we don't set fields that are not set. This ensures that encode_proto and parse_proto remain symmetric (decoding the result of parsing will return the original result).

[pront]

Any ideas on reducing code duplication here? Could we pull the shared bit into VRL? We want changes to proto codecs to be in sync with these new functions.

[flavioc]

My first instinct is to provide the following three functions:

• fn get_message_descriptor(descriptor_set_path: &Path, message_type: &str,
• fn encode_proto(descriptor: &MessageDescriptor, value: Value)
• fn parse_proto(descriptor: &MessageDescriptor, value: Value)

as public functions in src/protobuf/ and expose that through a public module as vrl::protobuf. Then we can just use them in the vector crate.

Let me know if that makes sense and I can do the changes. Open to other suggestions since I am not super familiar with the code yet.

[pront]

This plan sounds reasonable to me. This will require some effort:

1. add the protobuf utils to VRL
2. wait for a VRL release
3. bump VRL version in Vector
4. finally replace duplicate Vector code with vrl::protobuf utils

We can start with the first step and if you want, we can maintain a dev branch on Vector for the last step.

[pront]

This is also a good opportunity to cleanup the tests/data a bit.

I am not sure if all of those separate files are required. Worth taking a look if some of them can be merged. Not sure if we could avoid duplication across repos, worth a quick consideration but not a blocker.

[pront]

Thank you @flavioc for refactoring, this is starting to look great!

As a next step, just to be sure that the exposed utils are good to go, can you follow up with a draft PR in Vector where:

• Change the VRL version to point to this commit vrl = { git = "https://github.com/vectordotdev/vrl", rev = "9b25e9954c794f54a6e0e1c8706d79718b72983c" ...
• Delete duplicate code and use the utils you exposed in this PR

[flavioc]

Thanks! So, this is how the vector side will look like: vectordotdev/vector@8538929

[pront]

Is this commit that part of a PR? Interested to see the CI checks passing.

[flavioc]

Yes, I created a draft PR: vectordotdev/vector#20074

[pront]

👍 if those checks I think we can go ahead with this PR

[pront]

There are some failing checks. They all look minor. This needs a changelog entry and some clippy fixes.

[flavioc]

There are some failing checks. They all look minor. This needs a changelog entry and some clippy fixes.

Fixed those (I think) and also the licenses file.

[pront]

There are some test failures. You can repo locally with ./scripts/checks.sh vrl_tests tests.

[pront]

Thanks @flavioc! Let's follow up on the Vector PR after a new VRL version is released.

[pront]

I forgot to mention, currently the documentation lives in the Vector repo. Every time we add new VRL function we also followup with a doc PR e.g. vectordotdev/vector#20048. Can you please create one when have some time? 🙏

[flavioc]

I forgot to mention, currently the documentation lives in the Vector repo. Every time we add new VRL function we also followup with a doc PR e.g. vectordotdev/vector#20048. Can you please create one when have some time? 🙏

For sure, I will follow up with that. Thanks for the review!