Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConnectionFailure can not reach Elasticsearch Cluster (Fd1.2.2, ESplugin 2.11.0, ES6.2.4, https) #439

Closed
amphibithen opened this issue Jun 20, 2018 · 7 comments

Comments

@amphibithen
Copy link

commented Jun 20, 2018

Problem

I'm able to connect to ES 6.2.4 using FluentD .12, but when upgrading to FluentD 1.2.2 which uses ES plugin 2.11.0 it is throwing errors.

2018-06-20 14:10:42 +0000 [warn]: #0 got unrecoverable error in primary and no secondary error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>\"elasticsearch.myurl.com\", :port=>443, :scheme=>\"https\"})!"

Steps to replicate

Dockerfile

FROM fluent/fluentd:v1.2.2

COPY fluent.conf /fluentd/etc/

RUN apk add --update --virtual .build-deps \
        sudo build-base ruby-dev \
    && sudo gem install \
         fluent-plugin-elasticsearch \
         fluent-plugin-record-modifier \
    && sudo gem sources --clear-all \
    && apk del .build-deps \
    && rm -rf /var/cache/apk/* \
              /home/fluent/.gem/ruby/2.3.0/cache/*.gem

ENV ELASTICSEARCH_HOST=elasticsearch.myurl.com \
    ELASTICSEARCH_PORT=443 \
    ELASTICSEARCH_PROTOCOL=https \
    cluster=test \
    vpc=test \
    host=test

fluent.conf

<source>
  @type forward
  port 24224
</source>

<filter **>
  @type record_modifier
  <record>
    host "#{ENV['host']}"
    cluster "#{ENV['cluster']}"
    vpc "#{ENV['vpc']}"
  </record>
</filter>

<match **>
  @type copy
  <store>
    # reference https://github.com/uken/fluent-plugin-elasticsearch
    @type elasticsearch
    logstash_format true
    flush_interval 1s
    host "#{ENV['ELASTICSEARCH_HOST']}"
    port "#{ENV['ELASTICSEARCH_PORT']}"
    scheme "#{ENV['ELASTICSEARCH_PROTOCOL']}"
    ssl_verify false
    index_name fluentd
    type_name fluentd
    include_tag_key true
    tag_key container_image
    request_timeout 15s
  </store>
</match>

testing to see if its dns issue

[root@ip-10-149-5-110 fluentd]# docker exec -it -u root fluentd122 /bin/ash
/ # apk add --no-cache curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/3) Installing libssh2 (1.8.0-r2)
(2/3) Installing libcurl (7.60.0-r1)
(3/3) Installing curl (7.60.0-r1)
Executing busybox-1.27.2-r11.trigger
OK: 27 MiB in 30 packages
/ # curl -k https://elasticsearch.myurl.com
{
  "name" : "oD2svwu",
  "cluster_name" : "elasticsearch-aws",
  "cluster_uuid" : "z5LTNZOwQsWXM0ZIBh01qQ",
  "version" : {
    "number" : "6.2.4",
    "build_hash" : "ccec39f",
    "build_date" : "2018-04-12T20:37:28.497551Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Expected Behavior or What you need to ask

it works seamlessly similar to fluentd v0.12

Using Fluentd and ES plugin versions

/ # fluentd --version
fluentd 1.2.2

ES 6.2.4

/ # fluent-gem list

*** LOCAL GEMS ***
cool.io (1.5.3)
dig_rb (1.0.1)
elasticsearch (6.1.0)
elasticsearch-api (6.1.0)
elasticsearch-transport (6.1.0)
excon (0.62.0)
faraday (0.15.2)
fluent-plugin-elasticsearch (2.11.0)
fluent-plugin-record-modifier (1.1.0)
fluentd (1.2.2)
http_parser.rb (0.6.0)
json (2.1.0)
msgpack (1.2.4)
multi_json (1.13.1)
multipart-post (2.0.0)
oj (3.3.10)
openssl (default: 2.0.7)
psych (default: 2.2.2)
serverengine (2.0.6)
sigdump (0.2.4)
strptime (0.2.3)
thread_safe (0.3.6)
tzinfo (1.2.5)
tzinfo-data (1.2018.5)
yajl-ruby (1.4.0)

logs on startup

[root@ip-10-149-5-110 fluentd]# docker run -it --rm --name fluentd122 -p 24224:24224 -e ELASTICSEARCH_PROTOCOL=https -e ELASTICSEARCH_HOST=elasticsearch.myurl.com -e ELASTICSEARCH_PORT=443 -e cluster=test -e vpc=test -e host=$HOSTNAME -v /etc/hosts:/etc/hosts fluentd122
2018-06-20 15:01:20 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2018-06-20 15:01:20 +0000 [info]: 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2018-06-20 15:01:20 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    port 24224
  </source>
  <filter **>
    @type record_modifier
    <record>
      host ip-10-149-5-110.myurl.com
      cluster test
      vpc test
    </record>
  </filter>
  <match **>
    @type copy
    <store>
      @type "elasticsearch"
      logstash_format true
      flush_interval 1s
      host "elasticsearch.myurl.com"
      port 443
      scheme "https"
      ssl_verify false
      index_name "fluentd"
      type_name "fluentd"
      include_tag_key true
      tag_key "container_image"
      <buffer>
        flush_interval 1s
      </buffer>
    </store>
  </match>
</ROOT>
2018-06-20 15:01:20 +0000 [info]: starting fluentd-1.2.2 pid=5 ruby="2.4.4"
2018-06-20 15:01:20 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2018-06-20 15:01:21 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.0'
2018-06-20 15:01:21 +0000 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'
2018-06-20 15:01:21 +0000 [info]: gem 'fluentd' version '1.2.2'
2018-06-20 15:01:21 +0000 [info]: adding filter pattern="**" type="record_modifier"
2018-06-20 15:01:21 +0000 [info]: adding match pattern="**" type="copy"
2018-06-20 15:01:21 +0000 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
2018-06-20 15:01:21 +0000 [info]: adding source type="forward"
2018-06-20 15:01:21 +0000 [info]: #0 starting fluentd worker pid=15 ppid=5 worker=0
2018-06-20 15:01:21 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
2018-06-20 15:01:21 +0000 [info]: #0 fluentd worker is now running worker=0
2018-06-20 15:01:23 +0000 [warn]: #0 got unrecoverable error in primary and no secondary error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>\"elasticsearch.myurl.com\", :port=>443, :scheme=>\"https\"})!"
2018-06-20 15:01:23 +0000 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_3f9f50aa7e48/56f141257abb9bc8b3d3ec8e7e4088c9.log
2018-06-20 15:01:24 +0000 [warn]: #0 got unrecoverable error in primary and no secondary error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>\"elasticsearch.myurl.com\", :port=>443, :scheme=>\"https\"})!"
2018-06-20 15:01:24 +0000 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_3f9f50aa7e48/56f1412763dd3d8b61f2ad94b4186d33.log

@amphibithen

This comment has been minimized.

Copy link
Author

commented Jun 20, 2018

I also pointed this to my ES 5.6.3 cluster and get the same error. So i'm sure it's an issue with my configuration and how it's not a 1:1 match between FluentD .12 and 1.2, but looking for some guidance here...

@amphibithen

This comment has been minimized.

Copy link
Author

commented Jun 20, 2018

I have figured this out after many hours. Apparently there is an additional required field when using https:

ssl_version TLSv1_2 # or [SSLv23, TLSv1, TLSv1_1]

documentation says "if you want to include this" but it appears it's actually required. after adding that, i was able to connect. i just so happened to have some other changes to tweak as a result:

2018-06-20 18:04:51 +0000 [info]: #0 Connection opened to Elasticsearch cluster => {:host=>"elasticsearch.myurl.com", :port=>443, :scheme=>"https"}
2018-06-20 18:04:51 +0000 [info]: #0 Detected ES 6.x: ES 7.x will only accept `_doc` in type_name.

and after changing type_name then everything works as expected. Is the documentation wrong or is the code wrong for ssl_version being required?

@cosmo0920

This comment has been minimized.

Copy link
Collaborator

commented Jun 21, 2018

Because this plugin documentation does not assume TLS environment by default.

@cosmo0920

This comment has been minimized.

Copy link
Collaborator

commented Jun 21, 2018

Is the documentation wrong or is the code wrong for ssl_version being required?

Documentation is correct.
Some plugin users still use ES 2.x.
We cannot specify ssl_version TLSv1_2 by default.

@amphibithen

This comment has been minimized.

Copy link
Author

commented Jun 25, 2018

Would it make sense to change documentation to say "if ssl enabled, it is required to set ssl_version"? I think this is important because forcing a ssl version is not a standard requirement for systems connecting over https, it's typically optional for "added security".

@cosmo0920

This comment has been minimized.

Copy link
Collaborator

commented Jun 25, 2018

Would it make sense to change documentation to say "if ssl enabled, it is required to set ssl_version"?

I think that updating document seems to be good.
Moreover, I want to display warnings or log when user enable ssl and ES 6 or above to be detected.

@tombailey

This comment has been minimized.

Copy link

commented Jun 29, 2018

I agree with @amphibithen, I also had this issue so it should be mentioned in the docs somehow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.