"No alive nodes found in your cluster" using v2.1.4 release with ES 1.7

[rjbaker]

Hi There,

I'm getting the following error when attempting to connect to ES 1.7 to run a scroll/scan operation.

I'm using an ES cluster hosted by Qbox.io accessed via a client node. I'm able to connect fine using a standard rest client and also the 1.0 branch of the client library.

Fatal error: Uncaught exception 'Elasticsearch\Common\Exceptions\NoNodesAvailableException' with message 'No alive nodes found in your cluster' in /Users/richardbaker/Documents/Development/project/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ConnectionPool/StaticNoPingConnectionPool.php:51
Stack trace:
#0 /Users/richardbaker/Documents/Development/project/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php(71): Elasticsearch\ConnectionPool\StaticNoPingConnectionPool->nextConnection()
#1 /Users/richardbaker/Documents/Development/project/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php(89): Elasticsearch\Transport->getConnection()
#2 /Users/richardbaker/Documents/Development/project/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(230): Elasticsearch\Transport->performRequest('GET', '/_search/scroll...', Array, NULL, Array)
#3 /Users/richardbaker/Documents/Development/project/vendor/react/promise/src/Fulf in /Users/richardbaker/Documents/Development/project/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ConnectionPool/StaticNoPingConnectionPool.php on line 51

My code is basically ripped from the client library example:

require 'vendor/autoload.php';

$source_hosts = [
    'https://user:long$passwor?d@clientnode-instance-hostname.qbox.io:443'
];

$source_client = Elasticsearch\ClientBuilder::create()
    ->setHosts($source_hosts)->build();

// Scan/Scroll Query
$source_params = [
    "search_type" => "scan",    // use search_type=scan
    "scroll" => "30s",          // how long between scroll requests. should be small!
    "size" => 50,               // how many results *per shard* you want back
    "index" => "alerts-1-1",
    "type" => "alert",
    "body" => [
        "query" => [
            "filtered" => [
                "query" => [
                    "match_all" => []
                ],
                "filter" => [
                    "range" => [
                        "timestamps.created" => [
                            "gte" => "now-10d"
                        ]
                    ]
                ]
            ]
        ]
    ]
];

$total = 0;
$docs = $source_client->search($source_params);
$scroll_id = $docs['_scroll_id'];

while (true)
{
    // Execute a Scroll request
    $response = $source_client->scroll([
            "scroll_id" => $scroll_id,  //...using our previously obtained _scroll_id
            "scroll" => "30s"           // and the same timeout window
        ]
    );

    // Check to see if we got any search hits from the scroll
    if (count($response['hits']['hits']) > 0)
    {
        $total += count($response['hits']['hits']);

        // Get new scroll_id
        // Must always refresh your _scroll_id!  It can change sometimes*/
        $scroll_id = $response['_scroll_id'];
        echo "Completed: " . $total . PHP_EOL;
    }
    else
    {
        // No results, scroll cursor is empty.  You've exported all the data
        break;
    }
}

[polyfractal]

Hmm. Does it fail for regular operations (generic search, etc), or just scan/scroll?

Can you try running this:

require 'vendor/autoload.php';

$source_hosts = [
    'https://user:long$passwor?d@clientnode-instance-hostname.qbox.io:443'
];

$source_client = Elasticsearch\ClientBuilder::create()
    ->setHosts($source_hosts)->build();

$source_params = [
    "index" => "alerts-1-1",
    "type" => "alert",
    'client' => [
        'verbose' => true
    ]
];

try {
  $docs = $source_client->search($source_params);
  print_r($docs);
catch (Exception $e) {
  $last = $source_client->transport->getLastConnection()->getLastRequestInfo();
  $last['response']['error'] = [];
  print_r($last);
}

The getLastRequestInfo() call spews a huge amount of output (the promises are recursive/nested, so they dump a ton of output), which is why I'm nulling out the error section. I'm mainly interested in the top-level request and response sections.

Some other questions:

• Can you access the cluster via a cli curl command?
• Are you on Redhat? Is SELinux enabled? By default, SELinux is configured to disallow curl from hitting non-standard ports.
• IPtables? Firewall?

[rjbaker]

Thanks! No firewall here, just running on my mac.

The client node is accessed using https on port 443. Curl commands work fine, and i'm able to connect via the client node fine.

Interesting output, looks like it's trying to resolve a private cluster hostname or something?

Array
(
    [request] => Array
        (
            [http_method] => GET
            [scheme] => https
            [uri] => /alerts-1-1/alert/_search
            [body] => 
            [headers] => Array
                (
                    [host] => Array
                        (
                            [0] => :9200
                        )

                )

            [client] => Array
                (
                    [curl] => Array
                        (
                            [107] => 1
                            [10005] => user:redacted-long-pa$$word
                        )

                    [verbose] => 1
                )

        )

    [response] => Array
        (
            [transfer_stats] => Array
                (
                    [url] => https://34fG:9200/alerts-1-1/alert/_search
                    [content_type] => 
                    [http_code] => 0
                    [header_size] => 0
                    [request_size] => 0
                    [filetime] => -1
                    [ssl_verify_result] => 0
                    [redirect_count] => 0
                    [total_time] => 0.014194
                    [namelookup_time] => 0
                    [connect_time] => 0
                    [pretransfer_time] => 0
                    [size_upload] => 0
                    [size_download] => 0
                    [speed_download] => 0
                    [speed_upload] => 0
                    [download_content_length] => -1
                    [upload_content_length] => -1
                    [starttransfer_time] => 0
                    [redirect_time] => 0
                    [redirect_url] => 
                    [primary_ip] => 
                    [certinfo] => Array
                        (
                        )

                    [primary_port] => 0
                    [local_ip] => 
                    [local_port] => 0
                    [error] => Could not resolve host: 
                    [errno] => 6
                )

            [curl] => Array
                (
                    [error] => Could not resolve host: 
                    [errno] => 6
                )

            [effective_url] => https://34fG:9200/alerts-1-1/alert/_search
            [status] => 
            [reason] => 
            [body] => 
            [headers] => Array
                (
                )

            [error] => Array
                (
                )

        )

)

[polyfractal]

o_O

That's very strange, I have no idea tbh. Is there anything suspicious in your /etc/hosts file that may be redirecting? Or something like Anvil for hosting local sites on your mac?

I noticed the redacted password in your example has a question mark...does your real password have one? I ask because I just tried the fake user/pass and got a quirky result:

'https://user:long$passwor?d@clientnode-instance-hostname.qbox.io:443'

Array
(
    [request] => Array
        (
            [http_method] => GET
            [scheme] => http
            [uri] => //user:long$passwor/alerts-1-1/alert/_search
            [body] => 
            [headers] => Array
                (
                    [host] => Array
                        (
                            [0] => https:9200
                        )
                )
...

E.g. it looks like the ? causes PHP's parse_url() to terminate early, as if the question mark was actually the ? signifying the start of url parameters.

Also, I noticed the $...that could cause problems if you are using double-quotes and PHP is trying to substitute in a variable for the $ section.

• Make sure you are single-quoting your URL
• Try adding a / at the end of the URL, that seems to make PHP's parse_url() happier if there is a question mark in the password. E.g. 'https://user:long$passwor?d@clientnode-instance-hostname.qbox.io:443/'

[rjbaker]

Thanks for your help. I think i've figured it out. My password actually contains an @ symbol.

With the v1 client i'm passing the credentials like so:

$params['hosts'] =  [
        'client-node-hostname.qbox.io:443'
    ];

$params['connectionParams']['auth'] = array(
    'user',
    'long-p@ssword',
    'Basic'
);

$source_client = new Elasticsearch\Client($params);

The trailing slash did the trick! 👍

[polyfractal]

Ahh. Unfortunately, no, it doesn't look like there is an easy way to do it in 2.0. You could do it on a per-request basis, but that's rather gross. E.g.:

$search = [
    "index" => "my_index",
    "type" => "my_type",
    'client' => [
        'curl' => [
          CURLOPT_HTTPAUTH => CURLAUTH_BASIC,
          CURLOPT_USERPWD => "$user:$pass"
        ]
    ]
];
$docs = $client->search($search);

But this is something that should be supported by the client. Lemme hack at it for a few minutes, I think I can introduce a setCustomCurl() method to the builder, as well as an array version for setHosts().

[ikwattro]

parse_url() should just work fine with this :

$x = parse_url("http://user:long-p@ssword@localhost:8000");
print_r($x);

Array
(
    [scheme] => http
    [host] => localhost
    [port] => 8000
    [user] => user
    [pass] => long-p@ssword
)

[polyfractal]

Hm yep, you're right. I'm not sure what's going on, hard to know without seeing the password. There is clearly something being parsed incorrectly.

@rjbaker Did you check for double-quoting issue and/or question mark?

What version of PHP are you using?

[rjbaker]

I'm currently just hacking on my mac, rather than a vm so PHP is a little outdated at 5.5.29.

To confirm, the exact issue is caused by a combination of special characters in my password that parse_url fails to parse correctly. Oddly, adding a trailing slash to the end of the complete URL resolves this. So, not really anything to do with ES client, apologies!

[polyfractal]

Bizarre! I'll comb through the PHP bug tracker, I wonder if you just uncovered a new bug :)

I'll also keep working on the two features I mentioned, since I think it would be useful to be able to explicitly set custom curl options for the entire client (rather than per-request).

[rjbaker]

Yeah that's quite a nice feature. It would be cool if you could grab an instance of the ring php client request handler, or pass some options directly to it. The AWS SDK does similar with guzzle.

[polyfractal]

Opened #382 to track that feature (on the road now, but should be able to polish it off when I find some free time). Gonna close this ticket. :)