ES 5.0 compatibility

[shikhar]

Let's track known issues here.

1. The mapping from Schema.Type.STRING to string fails because is is not a valid ES type anymore (https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_50_mapping_changes.html)

[ewencp]

@shikhar Would probably be good to get the system tests to run against ES 5.0 too, will probably reveal many issues.

[jozanek]

So ES 5.x is not yet supported by this library?

[otisg]

Same question here...

[azell]

I believe that one of the issues with using the string type in ES 5 is combining it with null_value. Omitting the null_value setting in Mapping.java allows ES 5 to create the mapping.

[ewencp]

It's not fully supported yet (though note that you can still use it -- just the incompatibility noted above). This should get closed once ES 5 support is resolved.

[rusrushal13]

Is this issue is still open?
I would like to work on it 😄

[babadofar]

elasticsearch 5.0 will silently convert string to the new text field data type, but in 6.0 this fails miserably.
This line:

kafka-connect-elasticsearch/src/main/java/io/confluent/connect/elasticsearch/ElasticsearchSinkConnectorConstants.java

Line 36 in e5aec60

public static final String STRING_TYPE = "string";

[wicknicks]

For ES 6.0 and above, the string type has been completely removed. In order to get the existing elasticsearch-connector to work with ES, we can set schema.ignore to true in the connector configuration. This will skip creating the mappings which causes the tasks to fail.

Importing data into ES will work with the above workaround. When the mappings are not specified, ES server will detect the type using the field mapping rules specified here. A string in your JSON document will most likely be converted into a text type and a keyword type (if it's within the length limits). This will allow full text searches on the analyzed text.

[synhershko]

See #154 (comment)

[rayokota]

This is fixed with #169