Nodes with inaccessible http.host and connection sniffing

[jimmyjones2]

I've setup a client node on my Kibana server, bound to loopback for security, but ever since got lots of errors from my perl script doing the data ingest. I added a network interface on the private cluster network to the Kibana server, alongside its normal interface. However it seems that the Perl client, connected to the cluster network, gets this inaccessible IP when sniffing, tries to connect to it at some point and bombs out.

node.master: false
node.data: false
network.host: "10.0.0.4"
http.host: "127.0.0.1"

Repro:

#!/usr/bin/perl

use Search::Elasticsearch;

my $es = Search::Elasticsearch->new( cxn_pool => 'Sniff', nodes => 'localhost:9200' );
for (1..10) {
  $es->index( index => 'test', type => 'test', body => { foo => 'bar' } );
}

As a workaround I can remove the http.host and leave it on the private interface, but I don't really want indexing traffic going through my underpowered web server! Shouldn't the Perl client ignore the inaccessible address rather than bombing out?

[clintongormley]

Hi @jimmyjones2

Please turn on the logging, which will explain what is happening internally:

my $es = Search::Elasticsearch->new( 
    cxn_pool => 'Sniff', 
    nodes => 'localhost:9200', 
    log_to => 'Stderr'
);

[jimmyjones2]

Hey @clintongormley

Okay, to easily reproduce am running Elasticsearch as normal, then starting another on the same host with the following config (where 10.0.0.1 is unreachable/not running Elasticsearch):

node.master: false
node.data: false
http.publish_host: "10.0.0.1"

Output is:

No live nodes available. Trying seed nodes.
Sniffing [http://localhost:9200]
Current cxns: ["http://10.0.0.1:9201","http://192.168.1.134:9200"]
Next sniff at: Fri Jun 19 20:23:05 2015
Require sniff before next request
[Timeout] ** [http://10.0.0.1:9201]-[599] Could not connect to '10.0.0.1:9201': Connection timed out, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 8. With vars: {'status_code' => 599,'request' => {'body' => {'foo' => 'bar'},'method' => 'POST','qs' => {},'serialize' => 'std','path' => '/test/test','mime_type' => 'application/json','ignore' => []}}

[Timeout] ** [http://10.0.0.1:9201]-[599] Could not connect to '10.0.0.1:9201': Connection timed out, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 8. With vars: {'status_code' => 599,'request' => {'body' => {'foo' => 'bar'},'method' => 'POST','qs' => {},'serialize' => 'std','path' => '/test/test','mime_type' => 'application/json','ignore' => []}}

[clintongormley]

You're using perl 5.20 or above? I think you're running into this bug: https://rt.cpan.org/Public/Bug/Display.html?id=103878

All I can suggest is that you use a different back end or a different version of Perl. I'd suggest NetCurl or Hijk

[clintongormley]

@jimmyjones2 out of interest, what OS are you on?

[clintongormley]

Also see chansen/p5-http-tiny#52

[jimmyjones2]

Above was Fedora 21 (v5.18.4), also tried on CentOS 6 (v5.10.1):

No live nodes available. Trying seed nodes.
Sniffing [http://localhost:9200]
Current cxns: ["http://10.0.0.1:9201","http://192.168.234.130:9200"]
Next sniff at: Sat Jun 20 04:38:48 2015
Marking [http://10.0.0.1:9201] as dead. Next ping at: Sat Jun 20 04:35:09 2015
Require sniff before next request
[http://10.0.0.1:9201] [Cxn] ** [http://10.0.0.1:9201]-[599] Could not connect to '10.0.0.1:9201': Connection refused, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 7. With vars: {'request' => {'body' => {'foo' => 'bar'},'qs' => {},'method' => 'POST','serialize' => 'std','path' => '/test/test','ignore' => [],'mime_type' => 'application/json'},'status_code' => 599}

Retrying request on a new cxn
Sniffing [http://192.168.234.130:9200]
Current cxns: ["http://192.168.234.130:9200","http://10.0.0.1:9201"]
Next sniff at: Sat Jun 20 04:39:09 2015
Marking [http://10.0.0.1:9201] as dead. Next ping at: Sat Jun 20 04:35:30 2015
Require sniff before next request
[http://10.0.0.1:9201] [Cxn] ** [http://10.0.0.1:9201]-[599] Could not connect to '10.0.0.1:9201': Connection refused, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 7. With vars: {'request' => {'body' => {'foo' => 'bar'},'qs' => {},'method' => 'POST','serialize' => 'std','path' => '/test/test','ignore' => [],'mime_type' => 'application/json'},'status_code' => 599}

Retrying request on a new cxn
Sniffing [http://192.168.234.130:9200]
Current cxns: ["http://192.168.234.130:9200","http://10.0.0.1:9201"]
Next sniff at: Sat Jun 20 04:39:30 2015
Marking [http://10.0.0.1:9201] as dead. Next ping at: Sat Jun 20 04:35:51 2015
Require sniff before next request
[http://10.0.0.1:9201] [Cxn] ** [http://10.0.0.1:9201]-[599] Could not connect to '10.0.0.1:9201': Connection refused, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 7. With vars: {'request' => {'body' => {'foo' => 'bar'},'qs' => {},'method' => 'POST','serialize' => 'std','path' => '/test/test','ignore' => [],'mime_type' => 'application/json'},'status_code' => 599}

Retrying request on a new cxn
Sniffing [http://192.168.234.130:9200]
Current cxns: ["http://10.0.0.1:9201","http://192.168.234.130:9200"]
Next sniff at: Sat Jun 20 04:39:51 2015
Marking [http://10.0.0.1:9201] as dead. Next ping at: Sat Jun 20 04:36:12 2015
Require sniff before next request
[Cxn] ** [http://10.0.0.1:9201]-[599] Could not connect to '10.0.0.1:9201': Connection refused, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 7. With vars: {'request' => {'path' => '/test/test','serialize' => 'std','ignore' => [],'mime_type' => 'application/json','body' => {'foo' => 'bar'},'qs' => {},'method' => 'POST'},'status_code' => 599}

[Cxn] ** [http://10.0.0.1:9201]-[599] Could not connect to '10.0.0.1:9201': Connection refused, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 7. With vars: {'request' => {'path' => '/test/test','serialize' => 'std','ignore' => [],'mime_type' => 'application/json','body' => {'foo' => 'bar'},'qs' => {},'method' => 'POST'},'status_code' => 599}

[jimmyjones2]

Back on Fedora 21 with cxn => 'NetCurl':

No live nodes available. Trying seed nodes.
Sniffing [http://localhost:9200]
Current cxns: ["http://10.0.0.1:9201","http://192.168.1.134:9200"]
Next sniff at: Sat Jun 20 13:22:36 2015
Require sniff before next request
[Timeout] ** [http://10.0.0.1:9201]-[509] 28: Timeout was reached, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 8. With vars: {'status_code' => 509,'request' => {'qs' => {},'ignore' => [],'body' => {'foo' => 'bar'},'mime_type' => 'application/json','path' => '/test/test','serialize' => 'std','method' => 'POST'}}

[Timeout] ** [http://10.0.0.1:9201]-[509] 28: Timeout was reached, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at es.pl line 8. With vars: {'status_code' => 509,'request' => {'qs' => {},'ignore' => [],'body' => {'foo' => 'bar'},'mime_type' => 'application/json','path' => '/test/test','serialize' => 'std','method' => 'POST'}}

[clintongormley]

Hmm ok - I see what you mean here. I'm thinking I should add a ping into the Sniff cxn_pool on first use.

[clintongormley]

Also see #73

[jimmyjones2]

Also it tried 10.0.0.1:9200 twice and then failed, why is it not trying 192.168.1.134:9200, which would have worked?