insert mutli index on the same topic

[PeterAmisy]

Hi,
First of all thank you for your amazing job.
I have a question. right now i'm tryin to insert mutli index from ES to one topic on Kafka.
I wonder if it's possible to do that. Because right now, as long as i'm using one parameters on "index.prefix" everything work perfectly. As soon as I'm tryin to put an extra index like => index.prefix= my_awesome_index, my_beauty_index (using yaml)
Nothing come from ES.
So my question is, Do I have to add an other connector or simply duplicate the line ?

[DarioBalinzo]

Hi,
you can't have multiple index.prefix, but you have two solutions:
in your example you can use my_ as index prefix, or alternatively you might create two instances of the connetor (each one using a single index.prefix).

Then, the output topic is based on the name of the input index. But if you need to put the output in a single topic, you might use the confluent connect transformer https://docs.confluent.io/platform/current/connect/transforms/regexrouter.html#regexrouter in order to rename the topic as you prefer.

Hope this helps

[liz72703]

Hi there,

Is there another means of doing this?

I need to send multiple indexes to one fixed topic name but the connector seems to be completely ignoring the regex configuration entries.

For example qradar-wcf-000011, qradar-wcf-000012, qradar-wcf-000013 (they increment on elasticsearch over time) all to a kafka topic called "qradar-wcf".

Thanks,
Liz

[hyperbaba]

Add the following to your config part :

"transforms": "dropTopicPrefix",
"transforms.dropTopicPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.dropTopicPrefix.regex": "(.*)qradar-wcf-(.*)",
"transforms.dropTopicPrefix.replacement": "qradar-wcf",

this will publish all the messages to the topic qradar-wcf