Custom topic naming strategy implementation based on custom event (Record) field with JSONPath expression

[embano1]

What is your idea?

Implement a custom topic naming strategy based on a specified event (Record) field.

For example, given the following Kafka Record value (JSON-encoded):

{
  "order_id": "123456789",
  "customer_name": "John Doe",
  "customer_email": "john.doe@example.com",
  "order_date": "2024-04-02T10:00:00",
  "items": [
    {
      "item_id": "SKU123",
      "item_name": "Example Product 1",
      "quantity": 2,
      "unit_price": 25.99
    },
    {
      "item_id": "SKU456",
      "item_name": "Example Product 2",
      "quantity": 1,
      "unit_price": 39.99
    }
  ],
  "total_amount": 91.97,
  "payment_method": "Credit Card",
  "shipping_address": {
    "street": "123 Main St",
    "city": "Anytown",
    "state": "CA",
    "postal_code": "12345",
    "country": "USA"
  }
}

A user could specify the JSONPath expression $.payment_method to use the payment_method field (key) value in the outgoing EventBridge event detail-type field.

This functionality is useful in Kafka environments, where different events (Records) are written to the same Kafka topic. Currently, all these events would use the same topic name (or hardcoded string in the configuration) in the detail-type field. Allowing more fine-grained behavior enables flexible rule pattern matching and advanced IAM policies based on detail-type filter criteria, see https://twitter.com/embano1/status/1734260552640143817

Would you be willing to make the change?

No, but @maschnetwork was interested in doing it.

Additional context

• Follow-up from Custom Topic Naming Strategy #27
• Support sidelining large events/payloads to an external event store #110 uses JSONPath

[embano1]

To be discussed:

• configuration syntax
• allow JSONPath on Kafka Record key as well
• handle cases where specified JSONPath is not defined/null/empty (String)
• align with S3 offloading implementation e.g., disallow certain JSONPath expressions such as array indexes

[embano1]

cc/ @agebhar1