Customizable prefix for Kafka topics

[hekike]

Currently, we use a hardcoded prefix om_ to prefix Kafka topics. For example:

• om_events
• om_detected_events
• om_meter_m1

Requirements:

• Support prefixes via header as OM-Prefix.
• Default to om_ prefix.
• Add API to create events and detected_events topics: POST /api/ingests
• By default create om_events and om_detected_events topics.
• Optionally configure ingestion topics via Kafka instead of ksql streams (in this case move partition and scheme a config to Kafka create)

[sagikazarmark]

This proved to be a tougher nut to crack than expected, but after some experimentation and bouncing off some walls, here is what I came up with:

• The name "prefix" seems too implementation specific. I'd recommend going back to the "namespace" terminology as it's easier to explain without delving too much into the implementation.
• I did some research and it seems there is nothing against sending a query string in a POST request (ie. the HTTP spec does not specifically deny it and people seem to accept it as a viable solution). Why is a query string better? Easier to represent in the OpenAPI spec.

With those in mind:

• OpenMeter would create a "default" (empty) namespace on launch (In case of Kafka and KSQLDB this means the current behavior)
• Sending a POST /api/namespaces request will create a namespace (TODO: come up with a validation rule for the name) (In case of Kafka and KSQLDB, it means new events and detected_events topic/table)
• Sending a POST /api/ingest request (without namespace) will land the event in the "default" (empty) namespace (ie. no topic name prefix)
• Sending a POST /api/ingest?namespace=asd request will land the event in the "asd" namespace (ie. topic name will be om_asd_events)

It took me some time to figure out a way to even think about the prefix concept in a consistent way and I realized that it's not a good term we can easily explain or reason about, so I decided to change it.

Let me know what you think.

[sagikazarmark]

This is done