Protobuf linting (and more!) using Buf CLI

[jyggen]

One thing about Pants that has really streamlined my team's workflow is the seamless integration between Python and Protocol Buffers. There are however a couple of additional things we currently do with protos in our CI/CD pipeline that would be awesome if Pants could handle as well, namely linting and breaking change detection.

To achieve this we use an open-source tool called Buf CLI. I could write a sales pitch myself, but seems easier to just copy some introductions from their documentation - so here goes...

buf lint:

buf provides lint functionality through buf lint, which runs a set of lint rules across your entire Protobuf schema. By default, buf uses a carefully curated set of lint rules designed to guarantee consistency and maintainability across a Protobuf schema of any size and any purpose, but without being so opinionated as to restrict organizations from making the design decisions they need to make for their individual APIs.

buf breaking:

buf provides a breaking change detector through buf breaking, which runs a set of breaking rules across the current version of your entire Protobuf schema in comparison to a past version of your Protobuf schema. The rules are selectable, and split up into logical categories depending on the nature of breaking changes you care about.

Describe alternatives you've considered
I did plan (and still plan) to try and write a Pants plugin for Buf, but until I do I figured it wouldn't hurt to "plant the seed" here as well in case someone else wants to give it a shot :)

Additional context
As a side note, Buf also provides buf protoc, a "modern, high-performance, drop-in replacement for protoc" that uses buf's internal compiler. According to their docs it gives "a 7.5x real-world improvement on a modern personal computer". If Buf support is added to Pants, maybe it's worth looking into using its codegen capabilities too.

[Eric-Arellano]

That'd be amazing! Thank you for the suggestion!

Would you be interested in contributing this directly to Pants? There's a plugin guide at https://www.pantsbuild.org/v2.8/docs/plugins-lint-goal and Shellcheck will look very similar to what you're proposing: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/shell/lint/shellcheck/rules.py. We're also happy to help along the way :)

[jyggen]

Sure, I could give it a go :)

[Eric-Arellano]

Sweet! We'd probably want this to live in the folder backend/protobuf/lint/buf I think, with its own register.py, so that you have to opt into the linter specifically.

As mentioned in the first step of the guide, a good first step is to get buf downloaded by using TemplatedExternalTool and working with running --version. (The docs mention ExternalTool, but we have to use the subclass TemplatedExternalTool.)