auto_proto_registry_loader

Context

Automatically load up your Protobuf generated Golang types into the Protobuf registry (from the generated code), so that you can lookup by name of Protobuf message type.

Consider you have a Golang module containing the Golang types generated by Protobuf and you wish to look up the type of the Protobuf message using the name of the message type i.e. like have a utility method to do look up in a schema registry.

This project helps you to automatically loads your types into protobuf registry by using the generated golang code instead of the proto definition files.

Think of a loaded schema registry available as library.

Motivation

Enable your Golang module to now serve as a Schema Registry as well. i.e. make the 'protoregistry.GlobalTypes.FindMessageByName(name)' method work without reading proto files.

This project has been created to guide you on creating a schema registry as a library.

Use Case

Consider an ingestion system which is ingesting various types of events. Each event has a schema. Now the ingestion system receives an event payload with the type of event mentioned. It needs to validate the payload with the schema. So it looks up the schema registry using the event type name and gets back the schema.

If the proto defined golang types are available as golang module, the ingestion system imports this module and along with import, automatically registers the various protobuf golang types in the golang proto registry object.

Once registered, the ingestion system just needs to call the 'FindMessageByName()' method to get the type.

Using reflection, it can create an instance of that type & unmarshall the payload into that instance.

If unmarshalling fails, it means that payload does not match the schema.

The alternative solution is to have 'n' switch case blocks in your code - one for each type of event & check the payload. This is not feasiable and a code smell.

How to prepare your golang module ?

Say you have the golang module 'github.com/dc/batman' which you intend to also serve a golang schema registry:

project_dir(github.com/dc/batman)   
    |-- LICENSE
    |-- README.md
    |-- go.mod
    |-- go.sum
    |-- example_go_bindings (contains the golang code generated by protofbuf binary protoc)
    |   |-- foo
    |   |   `-- bar.pb.go
    |-- example_proto (contains the the various example_proto defintion files)
    |   `-- foo
    |       `-- bar
    |           `-- bar.example_proto

1. prepare the registry for loading the types

cd $project_dir
#go run -mod=mod github.com/jaihind213/auto_proto_registry_loader/load/ <full_path_to__go_bindings> <go_module_url> 
go run -mod=mod github.com/jaihind213/auto_proto_registry_loader/load/ $go_workspace_dir/src/github.com/dc/batman/go_bindings github.com/dc/batman
#run go mod tidy, to undo the change go run does to your go.mod file
go mod tidy

After running the auto_proto_registry_loader, you will notice a new file has been generated (pocket_registry/register_proto_defs.go)

project_dir(github.com/dc/batman)    
    |-- LICENSE
    |-- README.md
    |-- go.mod
    |-- go.sum
    |-- example_go_bindings (contains the golang code generated by protofbuf binary protoc)
    |   |-- foo
    |   |   `-- bar.pb.go
    |   `-- pocket_registry
    |       `-- register_proto_defs.go
    |-- example_proto (contains the the various example_proto defintion files)
    |   `-- foo
    |       `-- bar
    |           `-- bar.example_proto

2. Commit this file (pocket_registry/register_proto_defs.go) into the github.com/dc/batman repo !

3. Run your build and package the golang module.

4. In your golang source code which imports this golang module (i.e imports github.com/dc/batman), add an implicit import.

import (
   _ "github.com/dc/batman/go_bindings/pocket_registry"
   //Make sure you add this import in a package you know will definitely be used.(say where main() is defined)
   ...
)

5. Now for the Schema Registry lookup, add the following Code for lookup of type by name as follows:

//this requires an import of "google.golang.org/protobuf/reflect/protoregistry"
msgType, err := protoregistry.GlobalTypes.FindMessageByName(protoreflect.FullName("foo.bar.Keeper"))
//now use golang reflection to create instance of that type.
msgInstance := msgType.New().Interface()
if e := protojson.Unmarshal(json, msgInstance.(proto.Message)); e != nil {
    fmt.Print(e.Error())
}

Protecting the build

As you have seen, once we prepare the auto-generated file, we need to commit it into the repo. Sometimes, a developer might forget to re-run the preparation step after he/she modifies the proto definitions.

You could automate the 2 steps (preparation + commit to repo) as part of your build process

or

The 2 steps remain manual with a check in build process to see if developer has done the 2 steps.

Here is an example of how I did it gitlab-ci.yml

validate_imports:
  stage: check_imports_file
  image: golang:1.15
  script:
    - go run -mod=mod github.com/jaihind213/auto_proto_registry_loader/load/ <full_path_to__go_bindings> <go_module_url>
    - git status
    - exit_code=0
    - git status|grep register_proto_defs.go |grep modified || exit_code=$?
    - if [ ${exit_code} -eq 0 ];then echo "please run 'go run -mod=mod github.com/jaihind213/auto_proto_registry_loader/load/ <full_path_to__go_bindings> <go_module_url>' & commit the generated file 'register_proto_defs.go' for build to succeed"; exit 2; fi

Example Usage in a project

This project itself has some proto defintions & corresponding generated golang code. You will notice that '$Project_home/example_go_bindings/pocket_registry/register_proto_defs.go' has been generated. If this module is imported in your project, you can now lookup the proto registry for the types defined in the proto files. Hence this golang module now serves as schema registry too.

refer $Project_home/lookup_test.go for an example lookup.

Feedback

As always, if you have any feedback , I welcome it.

Stars

If you find the project useful, kindly leave it a github star. 🙏