Timeout Error

[georg94]

After starting logstash it runs for a short time and then ends with the following error message:

Faraday::TimeoutError: Timeout::Error
              call at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/adapter/net_http.rb:56
    build_response at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139
       run_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/connection.rb:377
   perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/http/faraday.rb:24
              call at org/jruby/RubyProc.java:271
   perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/base.rb:187
   perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/transport/http/faraday.rb:20
   perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.2/lib/elasticsearch/transport/client.rb:102
   perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.2/lib/elasticsearch/api/namespace/common.rb:21
      get_template at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.2/lib/elasticsearch/api/actions/indices/get_template.rb:28
  template_exists? at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:132
  template_install at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:21
          register at /opt/logstash/lib/logstash/outputs/elasticsearch.rb:259
              each at org/jruby/RubyArray.java:1613
      outputworker at /opt/logstash/lib/logstash/pipeline.rb:220
     start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152

The config file looks like this

input {
  stdin {
    type => "stdin-type"
  }

  file {
    type => "syslog"
    path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
    start_position => "beginning"
  }

  file {
    type => "logstash"
    path => [ "/var/log/logstash/logstash.log" ]
    start_position => "beginning"
  }
}

filter {
  if [type] == "docker" {
    json {
      source => "message"
    }
    mutate {
      rename => [ "log", "message" ]
    }
    date {
      match => [ "time", "ISO8601" ]
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }

  elasticsearch {
    embedded => true
    host => "10.20.30.59"
    port => "9200"
    protocol => "http"
  }
}

Logstash is running in a docker container with OpenJDK 64-Bit 7u71-2.5.3-2

[jsvd]

Is the port 9200 of 10.20.30.59 accessible from that docker container?

[georg94]

Yes it is. In the short time period logstash is running I even can access the API
curl "[host-ip]:9200/_search?pretty=true" gives me

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

[georg94]

It seems to be a problem with the embedded elasticsearch.
When I only output to stdout logstash keeps running.

[jsvd]

Having trouble replicating that, I just started logstash with a shorter config and it worked fine (I was able to query the server w/ curl):

input { stdin {} }
output {
  stdout { codec => rubydebug   }
  elasticsearch {
    embedded => true
    host => "192.168.1.76"
    port => "9200"
    protocol => "http"
  }
}

[suyograo]

@georg94 can you start LS with --debug command line flag to get more verbose logs?

[georg94]

I already did, but the error I get is the same

[georg94]

This is weired. I just set up docker on another server and it is working.
So I guess you can close this issue.

[jsvd]

Glad it's sorted out :)

Closing.

[georg94]

I found the problem:
On the not working enviorment http_proxy, https_proxy and no_proxy were set. Logstash/Elasticsearch seem not to use the no_proxy env so the requests were redirected over the proxy who threw an error page.
I executed unset http_proxy && unset https_proxy before starting logstash and it is working now on both systems.

[jsvd]

Interesting, thanks for debugging this! I'll open an issue to investigate the no_proxy handling