Tasks are not created #383
Description
Hello,
We use kafka-connect to send data from a topic to ElasticSearch.
(Almost) Every morning, the connector doesn't work.
When I check the API, my connector is present, but 0 tasks are running, therefore the connector does "nothing" :
"name": "myConnector",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"type.name": "myView",
"behavior.on.null.values": "delete",
"tasks.max": "2",
"topics": "mytopic",
"drop.invalid.message": "true",
"key.ignore": "false",
"schema.ignore": "true",
"behavior.on.malformed.documents": "warn",
"topic.index.map": "mytopic:myindex",
"value.converter.schemas.enable": "false",
"name": "myConnector",
"connection.url": "http://elasticsearch:9200",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.storage.StringConverter"
},
"tasks": [],
"type": "sink"
}
the /status endpoint shows no errors. but still no tasks:
{
"name": "myConnector",
"connector": {
"state": "RUNNING",
"worker_id": "100.105.89.94:8083"
},
"tasks": [],
"type": "sink"
}
No errors are shown in the logs of the kafka-connect worker, at this point we have to delete the connector and re-create it every morning. Then tasks are usually created in the process, until the next day or so when the tasks seem to disapear.
Second problem, which might be related to the first one.
When ElasticSearch goes down, my tasks are in "error" (connection refused to the elastic search node), which is normal. But after ElasticSearch is back online, my tasks are still in the same error state. Here too I need to manually delete/re-create the connector to restore his tasks.
Is there some kind of retry mecanism which recreate tasks in error ? The documentation only mention "request retry" which is not the same.