Support for Elasticsearch 6.0

[tamilhce]

Elastic-search is going to remove mapping type from version 6.0
[(https://www.elastic.co/blog/removal-of-mapping-types-elasticsearch)]
what are the currently supported Elasticsearch versions?
will kafka-connect-elasticsearch work seamlessly with Elasticsearch version 6.0?

[babadofar]

Mapping types are actually still there in elasticsearch 6.0, but you can only have one type pr.index now, which kind of makes the whole concept of types redundant.
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html

However, I had some problems using elasticsearch 6.0 even with just one type. Not sure why yet, trying to dig up the reason.

[TylerBrock]

Hi everyone. I'd love to help, do we know what we need to do here to get support for 6.0?

[babadofar]

It might work if you allow elasticsearch handle the schema mapping . ( set schema.ignore:true)
Otherwise, the types are defined her

kafka-connect-elasticsearch/src/main/java/io/confluent/connect/elasticsearch/ElasticsearchSinkConnectorConstants.java

Line 36 in ebf12ba

public static final String STRING_TYPE = "string";

in particular, the elasticsearch data type for String is now Text (or Keyword). More details here: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/breaking_50_mapping_changes.html
There may be other issues as well.

[synhershko]

This connector code is very outdated (only supports ES 2.4.x and before) and does not apply ES best practices where applicable (for instance, it insists on setting document IDs, doesn't support time-based indices, doesn't support HTTP(S) authentication and more).

To use with ES 5.x and above, you need to create the index and it's mappings yourself and use type: text and type: keyword instead of type: string. This will prevent Connect from attempting to create the mapping itself. I can offer a PR for supporting ES 5.x and above if it has any chance of being accepted (unfortunately it looks like this project is stale).

The single type per index at this point in time is just a nuance, just make sure to set type.name to something broad and general (I usually use doc). Only once Elasticsearch 7.x comes out code changes will be required.

[rmoff]

I'm re-titling this as it's encompassing several issues.

[rayokota]

This is fixed with #169

[fabriziofortino]

We are running this connector with schemaIgnore: "true" against Elasticsearch 6.5.

Although the ingestion works, we have noticed that for each document sent the following WARN is printed in the es client logs:

[WARN ][o.e.d.c.ParseField ] Deprecated field [_version_type] used, expected [version_type] instead
[WARN ][o.e.d.c.ParseField ] Deprecated field [_version] used, expected [version] instead

This is a bit annoying since it creates a lot of noise in the logs. Digging a bit in the source code I have noticed the dependency with jest version 2.4.0. This matches with ES 2.0 in the compatibility table.

Is there any plan to upgrade to a newer version or (even better) use the Java High Level Rest Client now available from elastic?

@rayokota