Connect topology shows wrong topics

[kanimaru]

Using:

• Kafka Connect Version : 0.11.0.0-cp1
• Kafka Connect UI Version : 0.9.2
• ElasticSearch (confluent) Worker

Configuration like this:

{
  "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "type.name": "action",
  "flush.timeout.ms": "10000",
  "topics": "action-log",
  "tasks.max": "1",
  "batch.size": "1000",
  "topic.key.ignore": "true",
  "topic.index.map": "action-log:log",
  "name": "my_logger",
  "connection.url": "http://somedomain:9200",
  "linger.ms": "10",
  "topic.schema.ignore": "false"
}

Following Topics will be recognized:

• action-log
• true
• action-log:log
• false

[andmarios]

Hey @kanimaru, the problem is that the topics field isn't standard in Kafka Connect, so we can't rely on it in order to detect the topics your connector uses.

Thus, trying to be smart, we will use any configuration option that includes topic in its name in order to detect topics. Alas in your case it backfires and Connect UI detects erroneous topic names.

I don't know whether there are any plans to fix this in the short term from the dev team.

[namoscato]

@andmarios, that is true for source connectors, but per Kafka documentation, sink connectors will always have a topics option. Thus, I believe an explicit assumption could be made for sinks.

Here are the topic-related configuration option(s) for some common source connectors:

Connector	Option
JdbcSourceConnector	topic.prefix
ReplicatorSourceConnector	topic.regex, topic.whitelist, topic.blacklist
FileStreamSource	topic
SpoolDirCsvSourceConnector	topic
FsSourceConnector	topic
FtpSourceConnector	ftp.monitor.tail, ftp.monitor.update
GitHubSourceConnector	topic
CloudPubSubSourceConnector	kafka.topic
IotHubSourceConnector	Kafka.Topic
CouchbaseSourceConnector	topic.name
ReplicateSourceConnector	topic.prefix
KineticaSourceConnector	kinetica.topic_prefix
HANASourceConnector	topics

Clearly some of these are more complex than others (i.e. connectors with configuration options similar to ReplicatorSourceConnector), but thoughts on updating the source connector "topic extraction" algorithm to first try and find a single configuration option that matches the following regular expression?

/(?:kafka[._])?topics?(?:[._]prefix)?$/

It seems like that would fix most of the issues.

[andmarios]

Thanks! This is very helpful indeed.

I tagged some members of the UIs dev team to have a look and implement it when possible.