Redis cluster not considering host protocol for masters

[mensler]

I'm connecting to an AWS ElastiCache Redis cluster using in-transit encryption (TLS) by adding the tls:// protocol to the host (see example below). I guess that the protocol is not taken into account when connecting to a node.

If I connect to a cluster without encryption it works as expected, it also works when connecting to an encrypted non-clustered Redis instance.

Expected behaviour

Using methods like keys() on the cluster or ping()/scan() on each master should return the correct results.

Actual behaviour

keys() returns the error "RedisCluster::keys(): Can't send KEYS to masterhost-1.euc1.cache.amazonaws.com:6379".

ping() on a master returns "Unable to send commnad at the specificed node" (there are typos in this error message ;-))

I'm seeing this behaviour on

• OS: Ubuntu 16.04
• Redis: 5.0.4
• PHP: 7.0
• phpredis: 5.0.0

Steps to reproduce, backtrace or example script

$RedisCluster = new RedisCluster(null, ['tls://<cluster-endpoint>.euc1.cache.amazonaws.com:6379'], 5, 5, false, '<auth-pw>');
$RedisCluster->keys('*');

foreach ($RedisCluster->_masters() as $master) {
	$RedisCluster->ping($master);
}

I've checked

• There is no similar issue from other users
• Issue isn't fixed in develop branch

[yatsukhnenko]

@mensler unfortunately only Redis class supports specifying protocol right now so you can't use it with RedisCluster.

[mensler]

OK, thanks for this hint.
Any chance that this will get implemented in the cluster class, too?

[yatsukhnenko]

I will try to take a look next week

[mensler]

Hi @yatsukhnenko!
Did you make any progress on this?

[yatsukhnenko]

@mensler unfortunately no. I made a quick glance, understood that this will not be so easy as I thought and haven't returned to this.

[vanderlee]

Any idea if this will be fixed any time soon?

[yatsukhnenko]

@mensler @vanderlee could you test changes from issue-1607 branch?

[valero90]

Hi @yatsukhnenko it still doesn't work :( I tried to ping the masters but it doesn't work

PHP Fatal error: Uncaught RedisException: read error on connection to <myservice>.euw1.cache.amazonaws.com:6379 in redisTest.php:9

[yatsukhnenko]

@valero90 could you give me access to the node for testing?

[valero90]

@yatsukhnenko not really, it’s an AWS instance of ElastiCache and there’s not external access to the nodes, sorry :(

[yatsukhnenko]

I've tried to connect to tls://github.com:443 and it worked fine so it may be something AWS specific which I can't handle without access to AWS.

[vnrld]

Perhaps this can help, as I have the same problem with the AWS Elasticache TLS connector with RedisCluster:

ElastiCache configuration
Field                           : Description                                           : Value

Name                            : The identifier for the cluster                        : name
Status                          : The status of this Cluster                            : available
Configuration Endpoint          : The configuration endpoint of the cluster             : <cluster-id>.euc1.cache.amazonaws.com:6379
Primary Endpoint                : Primary endpoint of the cluster                       : -
Reader                          : EndpointReader endpoint of the cluster                : -
Node type                       : The type of the Node in your cluster                  : cache.t3.micro
Engine                          : Engine on the cluster                                 : Clustered Redis
Engine Version Compatibility    : Version compatibility of the engine                   : 5.0.6

Shards                          : The number of Shards in a Redis Cluster               : 1
Number of Nodes                 : The number of Nodes in the cluster                    : 2 nodes
Encryption in-transit:          : Status of enabling encryption of data on-the-wire     : Yes
Encryption at-rest              : Status of enabling encryption for data stored on disk : Yes
Redis AUTH                      : Status of Redis AUTH                                  : No

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/encryption.html
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/at-rest-encryption.html

Executed code:

Redis:

<?php
declare(strict_types=1);

$hosts = ['tls://<cluster-id>.euc1.cache.amazonaws.com'];

$redis = new Redis();
$redis->connect($hosts[0]);

$redis->set('test:time', 'Time: ' . date('Y-m-d H:i:s'));
$redis->set('test:time2', 'Time: ' . date('Y-m-d H:i:s', strtotime('-10 years')));

print_r($redis->keys('*'));

Result:

/tmp $ php cc.php

Fatal error: Uncaught RedisException: MOVED 11175 <cluster-id-node-1>.euc1.cache.amazonaws.com:6379 in /tmp/cc.php:13
Stack trace:
#0 /tmp/cc.php(13): Redis->set('test:time', 'Time: 2020-05-2...')
#1 {main}
  thrown in /tmp/cc.php on line 13
/tmp $ php cc.php
Array
(
    [0] => test:time
    [1] => test:time2
)

Redis Cluster

<?php

declare(strict_types=1);

$hosts = ['tls://<cluster-id>.euc1.cache.amazonaws.com:6379'];

$redis = new RedisCluster(null, $hosts);

$redis->set('test:time', 'Time: ' . date('Y-m-d H:i:s'));
$redis->set('test:time2', 'Time: ' . date('Y-m-d H:i:s', strtotime('-10 years')));

print_r($redis->keys('*'));

Result:

/tmp $ php cc2.php

Fatal error: Uncaught RedisClusterException: Error processing response from Redis node! in /tmp/cc2.php:9
Stack trace:
#0 /tmp/cc2.php(9): RedisCluster->set('test:time', 'Time: 2020-05-2...')
#1 {main}
  thrown in /tmp/cc2.php on line 9

[yatsukhnenko]

@vnrld thanks for info. I pushed new change into issue-1607 branch. Could you test it?

[vnrld]

@yatsukhnenko I did tests, and it does not work:

Redis extension information

redis
Redis Support	enabled
Redis Version 	5.2.0
Redis Sentinel Version 	0.1
Git revision 	$Id: c09504a67a05a1f5df9a5e01b3bd0e7a0434f820 $
Available serializers 	php, json

Connecting to the cluster configuration endpoint with TLS prefix

/var/www/html $ cat /tmp/c1.php && php /tmp/c1.php
<?php
$hosts = ['tls://clustercfg.<cluster-id>.euc1.cache.amazonaws.com:6379'];

$redis = new RedisCluster(null, $hosts);

$redis->set('test:time', 'Time: ' . date('Y-m-d H:i:s'));
$redis->set('test:time2', 'Time: ' . date('Y-m-d H:i:s', strtotime('-10 years')));

print_r($redis->keys('*'));

Fatal error: Uncaught RedisClusterException: Error processing response from Redis node! in /tmp/c1.php:6
Stack trace:
#0 /tmp/c1.php(6): RedisCluster->set('test:time', 'Time: 2020-05-2...')
#1 {main}
  thrown in /tmp/c1.php on line 6

Connecting to the cluster configuration endpoint without TLS prefix

/var/www/html $ cat /tmp/c2.php && php /tmp/c2.php
<?php
$hosts = ['clustercfg.<cluster-id>.euc1.cache.amazonaws.com:6379'];

$redis = new RedisCluster(null, $hosts);

$redis->set('test:time', 'Time: ' . date('Y-m-d H:i:s'));
$redis->set('test:time2', 'Time: ' . date('Y-m-d H:i:s', strtotime('-10 years')));

print_r($redis->keys('*'));

Fatal error: Uncaught RedisException: socket error on read socket in /tmp/c2.php:4
Stack trace:
#0 /tmp/c2.php(4): RedisCluster->__construct(NULL, Array)
#1 {main}

Next RedisClusterException: Couldn't map cluster keyspace using any provided seed in /tmp/c2.php:4
Stack trace:
#0 /tmp/c2.php(4): RedisCluster->__construct(NULL, Array)
#1 {main}
  thrown in /tmp/c2.php on line 4

Connecting to all cluster nodes with TLS prefix

/var/www/html $ cat /tmp/c3.php && php /tmp/c3.php
<?php
$hosts = ['tls://<node-id>-0001-001.<cluster-id>.euc1.cache.amazonaws.com:6379', 'tls://<node-id>-0001-002.<cluster-id>.euc1.cache.amazonaws.com:6379'];

$redis = new RedisCluster(null, $hosts);

$redis->set('test:time', 'Time: ' . date('Y-m-d H:i:s'));
$redis->set('test:time2', 'Time: ' . date('Y-m-d H:i:s', strtotime('-10 years')));

print_r($redis->keys('*'));

Fatal error: Uncaught RedisClusterException: Error processing response from Redis node! in /tmp/c3.php:6
Stack trace:
#0 /tmp/c3.php(6): RedisCluster->set('test:time', 'Time: 2020-05-2...')
#1 {main}

Any ideas?

Regards,
Lukas

[vnrld]

Linux: alpine
And the PHP config (if this helps)

PHP Version 7.4.6

Configure Command 	'./configure' '--build=x86_64-linux-musl' '--with-config-file-path=/usr/local/etc/php' '--with-config-file-scan-dir=/usr/local/etc/php/conf.d' '--enable-option-checking=fatal' '--with-mhash' '--enable-ftp' '--enable-mbstring' '--enable-mysqlnd' '--with-password-argon2' '--with-sodium=shared' '--with-pdo-sqlite=/usr' '--with-sqlite3=/usr' '--with-curl' '--with-libedit' '--with-openssl' '--with-zlib' '--with-pear' '--enable-fpm' '--with-fpm-user=www-data' '--with-fpm-group=www-data' '--disable-cgi' 'build_alias=x86_64-linux-musl'

Server API 	FPM/FastCGI

Virtual Directory Support 	disabled

Additional .ini files parsed 	/usr/local/etc/php/conf.d/docker-php-ext-pdo_mysql.ini, /usr/local/etc/php/conf.d/docker-php-ext-soap.ini, /usr/local/etc/php/conf.d/docker-php-ext-sodium.ini, /usr/local/etc/php/conf.d/opcache.ini, /usr/local/etc/php/conf.d/redis.ini

PHP API 	20190902

PHP Extension 	20190902

Zend Extension 	320190902

Zend Extension Build 	API320190902,NTS

PHP Extension Build 	API20190902,NTS

Debug Build 	no

Thread Safety 	disabled

Zend Signal Handling 	enabled

Zend Memory Manager 	enabled

Zend Multibyte Support 	provided by mbstring

IPv6 Support 	enabled

DTrace Support 	disabled

Registered PHP Streams	https, ftps, compress.zlib, php, file, glob, data, http, ftp, phar

Registered Stream Socket Transports	tcp, udp, unix, udg, ssl, tls, tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3

Registered Stream Filters	zlib.*, convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, dechunk

[yatsukhnenko]

@vnrld if I understand correctly tls connection works because set command works without errors. I will look at keys implementation, may be something wrong is there

[vnrld]

@yatsukhnenko: The error is here: #0 /tmp/c1.php(6): RedisCluster->set('test:time', 'Time: 2020-05-2...') and this is the "set" method

[yatsukhnenko]

@vnrld I looked at your example incorrectly. I'll try to install Redis 6 or stunnel and replicate issue locally

[yatsukhnenko]

@vnrld I added test for tls connection and it works on Travis for Redis + stunnel. If the latest changes in issue-1607 branch doesn't work for you this means that problem is really AWS specific and I can't fix it without access

[herrbpl]

I concur that error is there. Looks like when cluster nodes info is retrieved with CLUSTER SLOTS command,

phpredis/cluster_library.c

Line 595 in 5ca4141

clusterReply* cluster_get_slots(RedisSock *redis_sock)

there is no indication in response, if they use TLS or not.

CLUSTER SLOTS
*3
*4
:0
:5461
*3
$72
test-backoffice-0001-001.test-backoffice.ozh0ii.euc1.cache.amazonaws.com
:6379
$40
dd824332f1f1d032bdbab457d0f3fd8950d49f5c
*3
$72
test-backoffice-0001-002.test-backoffice.ozh0ii.euc1.cache.amazonaws.com
:6379
$40
c32714bc9587625ea54a5260a281f66f3d5ee8e3
*4
:10923
:16383
*3
$72
test-backoffice-0003-001.test-backoffice.ozh0ii.euc1.cache.amazonaws.com
:6379
$40
0ad03698d324b1f052d566b2e12a710b448990c5
*3
$72
test-backoffice-0003-002.test-backoffice.ozh0ii.euc1.cache.amazonaws.com
:6379
$40
86a13188bcbf252e7acc67a44bc2adff8f73d552
*4
:5462
:10922
*3
$72
test-backoffice-0002-001.test-backoffice.ozh0ii.euc1.cache.amazonaws.com
:6379
$40
86d9fd2c77cd6769675338db1dc45f43821247ab
*3
$72
test-backoffice-0002-002.test-backoffice.ozh0ii.euc1.cache.amazonaws.com
:6379
$40
dc678b693513fd957f0cd73c168b905e62e78c71

This information should be added from seed connection string when building sockets for actual nodes:

phpredis/cluster_library.c

Line 698 in 5ca4141

host = r3->element[0]->str;

Right now, anything returned from seed info is without indication of whether TLS is used or not, so when actual connection is made, it fails.

[yatsukhnenko]

@herrbpl thanks👍, I've got access to aws and already found this problem

[dennis-hh]

Sorry for necromancing this issue, but did I understand correctly that this is an AWS issue, not one in the library? Is there any solution in sight?
I'm connecting to an AWS redis cluster with tls and password like so:

$cluster = [
    getenv('ELASTICACHE_ENDPOINT') . ':' . getenv('ELASTICACHE_PORT')
];

$cluster = new RedisCluster(
    null,
    $cluster,
    1.5,
    1.5,
    false,
    getenv('ELASTICACHE_TOKEN')
);

This works, or at least there's no error.

$result = $cluster->get("foo");

This results in the following error:

Fatal error: Uncaught RedisException: read error on connection to ***-0002-001.***.euc1.cache.amazonaws.com:6379 in /var/www/html/public/redis.php:15 Stack trace: #0 /var/www/html/public/redis.php(15): RedisCluster->get('foo') #1 {main} Next RedisException: read error on connection to ***.euc1.cache.amazonaws.com:6379 in /var/www/html/public/redis.php:15 Stack trace: #0 /var/www/html/public/redis.php(15): RedisCluster->get('foo') #1 {main} Next RedisClusterException: Can't communicate with any node in the cluster in /var/www/html/public/redis.php:15 Stack trace: #0 /var/www/html/public/redis.php(15): RedisCluster->get('foo') #1 {main} thrown in /var/www/html/public/redis.php on line 15

Would it be possible to assume tls is used based on the protocol prefix of the cluster node(s)? I know that ioredis is able to connect to Elasticache clusters with tls and password enabled.

[yatsukhnenko]

@dennis-hh no, this is phpredis issue and I'm going to fix it after 5.3.0 GA

[yatsukhnenko]

I added new parameter to RedisCluster::__construct to specify stream ssl/tls context.
Important: seeds must be specified in host:port without scheme

$cluster = new RedisCluster(null, [
    '<node-id>-0001-001.<cluster-id>.euc1.cache.amazonaws.com:6379',
    '<node-id>-0001-002.<cluster-id>.euc1.cache.amazonaws.com:6379',
], 0, 0, false, null, [
    'verify_peer' => false,
]);

Could somebody test this changes?

[yatsukhnenko]

Changes were merged into develop branch

[Diamonds0a]

I just tested latest develop with a 3 node redis cluster in AWS with TLS. Working great!