feat: add protobuf to json converter

[Yash Shrivastava (alephys26)]

Description

We need the ability to convert protobuf to json to allow handling of protobuf records in caterpillar. Adding the functionality in converter task to perform this action.

Changes

Protobuf format support:

• Added a new protobuf converter implementation in protobuf.go, allowing conversion from binary Protobuf messages to JSON using a descriptor file and configurable options.
• Updated the converter task documentation in README.md to describe Protobuf support, configuration options, and usage examples. [1] [2] [3] [4]
• Registered the new protobuf converter type in the converter core logic.

Dependency management:

• Added a direct dependency on google.golang.org/protobuf in go.mod to support Protobuf operations. [1] [2]

Testing and examples:

• Added test files and a sample pipeline (person.desc, person.bin, protobuf.yaml) to demonstrate and verify Protobuf conversion functionality. [1] [2] [3]

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Tests

Added a test pipeline and it produced correct output record in the expected file:

{"id":42,"name":"Ada Lovelace","email":"ada@example.com","tags":["engineer","pioneer"]}

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation and I have updated the documentation accordingly.
• I have added tests to cover my changes.

[Copilot]

Pull request overview

Adds Protobuf decoding support to Caterpillar’s converter task so pipelines can convert binary Protobuf messages into JSON using a compiled FileDescriptorSet and configurable marshal options.

Changes:

• Implemented a new protobuf converter that loads a descriptor set, dynamically unmarshals messages, and marshals them to JSON.
• Registered protobuf as a supported converter format and documented configuration/usage.
• Added a sample pipeline plus .desc / .bin fixtures and promoted google.golang.org/protobuf to a direct dependency.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
internal/pkg/pipeline/task/converter/protobuf.go	New Protobuf-to-JSON converter implementation using descriptor sets + protojson.
internal/pkg/pipeline/task/converter/converter.go	Registers the new protobuf converter in the supported formats map.
internal/pkg/pipeline/task/converter/README.md	Documents Protobuf converter options and usage (also updates the format list).
README.md	Updates top-level supported task summary to mention Protobuf for converter.
go.mod	Adds google.golang.org/protobuf as a direct dependency.
test/pipelines/converter/protobuf.yaml	Example pipeline wiring file-read → protobuf-convert → file-write.
test/pipelines/converter/person.desc	Descriptor set fixture for the sample message schema.
test/pipelines/converter/person.bin	Binary Protobuf fixture to validate decoding behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[prasadlohakpure]

Yash Shrivastava (@alephys26) can we know the use case? Because if there is need to ship the schema file, then we might need to add support for s3 reads as well, similar to file task type

[wiz-55ccc8b716]

Wiz Scan Summary

Scanner	Findings
Vulnerabilities	-
Sensitive Data	-
Secrets	-
IaC Misconfigurations	-
SAST Findings	1
Software Management Findings	-
Total	1

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension.