Schema Transformation

[yanivbh1]

Motivation

Different schemas often occur when working with multiple data sources / different services / different pipelines that produce data on the same topic, while the consumers on the other side expect specific structure.
Usually, that job is distributed across different teams and business units and requires expensive sync in time and effort.
Upstream changes are often impossible.
For example, AWS Lambda cannot ingest JSON-based payload with a boolean value, but rather should be converted into string before.

Summary

Ability to perform deep, inline transformation of events structure at the broker level while in transit.
Functions like: Concat / trim / regex / sum / and more...
Would be great to get comments about expected functions

Value

Reduce logic from producers and consumers, increase data quality, reduce downtime and crashes, and enable complex streaming pipelines for in-app purposes.

Persona(s)

• Data Engineers
• Developers
• Data Analysts

[yanivbh1]

Will be part of Memphis function #944