How to send kafka tombstones?

[radovsky1]

Hi! I have the following pipeline, the result, where mapper(javascript) creates an array of json objects, each of which has a {key}:{message to kafka}, faced with the question of how to send Kafka tombstones? tried {key}:{null}, {key}:{""}, everything is interpreted as a string, how to implement a message as zero bytes?

  processors:
    - try:
        - javascript:
            file: ${MAPPER_FILE:/config/mapper.js}
        - mapping: root = this.instruments
        - unarchive:
            format: json_array
        - unarchive:
            format: json_map
    - catch:
        - log:
            level: ERROR
            message: "processor failed: ${!error()}"
            fields_mapping: |
              root.value = this
        - mapping: root = deleted()
output:
  kafka:

[mihaitodor]

Hey @radovsky1 👋 I believe the body of the message needs to be empty in that case. Would you mind trying the kafka_franz output? It's based on a newer underlying driver called franz-go.

You can use a mapping processor with root = "" to empty out the message body and, if you need to have a Kafka key attached to the message, you can configure this field: https://docs.redpanda.com/redpanda-connect/components/outputs/kafka_franz/#key.

Please let me know if that doesn't do the trick and I'll dig deeper.