Schema Store - Protobuf

[yanivbh1]

Summary

Unique, robust, dynamic, and easy-to-use schema store for Protobufs, Avro, and JSON based data formats.
Will be available to manage via the GUI, SDKs, and REST and with dedicated resources embedded in the broker itself without the need to deploy dedicated tools/system and add extra dependencies and management overhead.

Context

A schema defines how a message or event should look and in which format, creating a contract between producer and consumer that the broker will enforce or be aware of.

• We want to redesign how schema management and serialization work compared to other platforms such Schema Registry and AWS GLUE. Memphis will let the user the option to choose if your broker will enforce the schema or will be able to update dynamically with default values.
• Auto schema creator based on message existing schema.
• Ability to import schemas from git repo with CI/CD integrations to update the schema dynamically using DataOps Methodologies.
• Self-service for personas such as product/other devs/BIs/Analysts

Value

1. Enrich Memphis use cases
2. Help users to decouple business logic from consumers and producers
3. Recude code at the producers/consumers level
4. Reduce management overhead compared to other solution
5. Provide self-service for other persona to work with schemas

Persona(s)

• Active users
• Kafka/SQS/Redis/GCP users
• Data Engineers for building complex pipelines

Out of Scope

• Avro
• JSON Schema
  Will be available in Q4

[yanivbh1]

Released on v0.4.0