DOCS-208 - Connect Elastic Quickstart not working

[joel-hamill]

https://confluentinc.atlassian.net/browse/DOCS-208

• related CC-1385: Enhance connector to use text type with ES 5+ #169

Staging server http://staging-docs.confluent.io.s3-website-us-west-2.amazonaws.com/PR/693/3.3.0/connect/connect-elasticsearch/docs/elasticsearch_connector.html#quick-start

[joel-hamill]

@ewencp @kaiwaehner i haven't been able to actually get this to work locally. can you take a look? the staging server is here.

[kaiwaehner]

Hi Joel,

I checked your link...

I can only confirm that it worked for me using standalone mode started via regular Kafka script. I did not start it using Confluent CLI and distributed mode. I think this did not work for me either (IIRC).

Also not sure, but probably best to start Elastic as first step (not sure if you did)...

Kai

[joel-hamill]

@kaiwaehner

I did not start it using Confluent CLI and distributed mode.

in your comments here didn't you use the CLI to start Schema Registry; and start Kafka Connect as standalone? my instructions are:

1. Start Schema Registry, Kafka, ZK using the CLI
2. Start Kafka Connect as standalone
   ...

[ewencp]

@joel-hamill I tagged the connect team for review, they should be able to help review and get this into a good state.

[kkonstantine]

Why are we advising in the quickstart to use a mix of CLI and connect-standalone?

My impression is that the old quickstart just contained a stray reference to connect-standalone that we should remove it. But what is wrong with using the CLI from start to finish like we do with our other connectors?

Asking just before I try to run the quickstart again with the CLI. I believe that if there's something buggy there we just need to fix that, instead of using standalone connect.

[kkonstantine]

I just ran the quickstart. My understanding is that the CLI is not the problem.

The problem is compatibility with ES 6.x.x.

If I use ES 5.x.x the quickstart works as it used to. Given that, I've added suggestions inline, but that's the main caveat currently.

[kkonstantine]

We should start everything with the CLI. Here we just need:
$ confluent start

[joel-hamill]

CP installed and running is now listed as a prerequisite (like the other quick starts).

[kkonstantine]

We should have here what we have for the other connectors, letting users know that the need to start all services with the CLI

[kkonstantine]

Needs to include startup message for kafka-rest and connect, as shown when we run confluent start

[joel-hamill]

CP installed and running is now listed as a prerequisite (like the other quick starts).

[kkonstantine]

We need to remove or mention that we start everything with the CLI.

[kkonstantine]

This output won't apply when the CLI is used to start the Connect worker.

[kkonstantine]

Does not apply now.

[kkonstantine]

optional: Maybe mention that we use standalone mode? Because there's also distributed (without CLI) that loads the connector hitting the REST endpoint (commonly with a curl command).

If you don't think we overstress the meaning in the sentence this could be:
For non-CLI users, you can load the Elasticsearch connector by running Connect in standalone mode with this command:
or similar.

[kkonstantine]

Here's the caveat, and probably why it didn't work for others so far.

This example (the three records above) work with the connector when elasticsearch is of version 5.x.x (tried it with 5.6.6). With version 6.x.x (tried 6.1.2) we are hitting this issue: #154

Somehow we need to make this quickstart functional before we fix the issue above. That's either by mentioning the compatibility with ES 5 only or by changing the example not to use string types (which is not great if the users attempt to extend their examples following this quickstart and try string types later on).

[joel-hamill]

i will add the limitation to version 5.x.x

[kkonstantine]

The output I had after successfully running the connector with ES 5.6.6 was:

curl -XGET 'http://localhost:9200/test-elasticsearch-sink/_search?pretty'
{
  "took" : 39,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-elasticsearch-sink",
        "_type" : "kafka-connect",
        "_id" : "test-elasticsearch-sink+0+0",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value1"
        }
      },
      {
        "_index" : "test-elasticsearch-sink",
        "_type" : "kafka-connect",
        "_id" : "test-elasticsearch-sink+0+2",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value3"
        }
      },
      {
        "_index" : "test-elasticsearch-sink",
        "_type" : "kafka-connect",
        "_id" : "test-elasticsearch-sink+0+1",
        "_score" : 1.0,
        "_source" : {
          "f1" : "value2"
        }
      }
    ]
  }
}

[kaiwaehner]

Yeah, I just used a mix of CLI and standalone as the other one did not work :-) Agree that we should use CLI for all... Much better experience.

I think using Elastic 5 for now is fine for the quickstart. (highlighting that Elastic 6 does not work until an open issue is solved so that other do not try this out for hours)

We could move to Elastic 6 when the bug kkonstantine mentioned is fixed.

[joel-hamill]

@kaiwaehner @kkonstantine is there a known Elastic issue for 6.x that i can link to for more information?

[kkonstantine]

LGTM after recent changes.

[rayokota]

This is fixed with #169