Environment variable collision with Kubernetes

[floflock]

• Laravel Version: 5.6
• PHP Version: 7.2.8
• Database Driver & Version: mariadb
• Redis Package: predis/predis

Description:

I don't know, if this issue is related to the Laravel framework, but this is the output, if I am running my application as a kubernetes pod. The following log is from my laravel.log after hitting the application. The app tries to fetch some cached data from redis, but you have to look at the hostname in the brackets. [tcp://redis:tcp://10.7.240.204:6379] -> 'redis' is the internal hostname from my ENV-file and the IP is the IP coming from kubernetes. The correct string would be [tcp://redis:6379], all data coming from my ENV-file.

Maybe some of you have a solution or maybe it's a framework bug. By the way: the application is running on my docker-compose setup very well, without hostname issues.

[2018-07-28 20:48:53] production.ERROR: Connection timed out [tcp://redis:tcp://10.7.240.204:6379] {"exception":"[object] (Predis\\Connection\\ConnectionException(code: 110): Connection timed out [tcp://redis:tcp://10.7.240.204:6379] at /var/www/vendor/predis/predis/src/Connection/AbstractConnection.php:155)
[stacktrace]
#0 /var/www/vendor/predis/predis/src/Connection/StreamConnection.php(128): Predis\\Connection\\AbstractConnection->onConnectionError('Connection time...', 110)
#1 /var/www/vendor/predis/predis/src/Connection/StreamConnection.php(178): Predis\\Connection\\StreamConnection->createStreamSocket(Object(Predis\\Connection\\Parameters), 'tcp://redis:tcp...', 4)
#2 /var/www/vendor/predis/predis/src/Connection/StreamConnection.php(100): Predis\\Connection\\StreamConnection->tcpStreamInitializer(Object(Predis\\Connection\\Parameters))
#3 /var/www/vendor/predis/predis/src/Connection/AbstractConnection.php(81): Predis\\Connection\\StreamConnection->createResource()
#4 /var/www/vendor/predis/predis/src/Connection/StreamConnection.php(258): Predis\\Connection\\AbstractConnection->connect()
#5 /var/www/vendor/predis/predis/src/Connection/AbstractConnection.php(180): Predis\\Connection\\StreamConnection->connect()
#6 /var/www/vendor/predis/predis/src/Connection/StreamConnection.php(288): Predis\\Connection\\AbstractConnection->getResource()
\7 /var/www/vendor/predis/predis/src/Connection/StreamConnection.php(394): Predis\\Connection\\StreamConnection->write('*4\
\5\
\ETEX\
...')
#8 /var/www/vendor/predis/predis/src/Connection/AbstractConnection.php(110): Predis\\Connection\\StreamConnection->writeRequest(Object(Predis\\Command\\StringSetExpire))
#9 /var/www/vendor/predis/predis/src/Client.php(331): Predis\\Connection\\AbstractConnection->executeCommand(Object(Predis\\Command\\StringSetExpire))
#10 /var/www/vendor/predis/predis/src/Client.php(314): Predis\\Client->executeCommand(Object(Predis\\Command\\StringSetExpire))
#11 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(96): Predis\\Client->__call('setex', Array)
#12 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(108): Illuminate\\Redis\\Connections\\Connection->command('setex', Array)
#13 /var/www/vendor/laravel/framework/src/Illuminate/Cache/RedisStore.php(93): Illuminate\\Redis\\Connections\\Connection->__call('setex', Array)
#14 /var/www/vendor/laravel/framework/src/Illuminate/Cache/Repository.php(195): Illuminate\\Cache\\RedisStore->put('QbR4kcNJjGvBFKp...', 'a:1:{s:6:\"_flas...', '120')
#15 /var/www/vendor/laravel/framework/src/Illuminate/Session/CacheBasedSessionHandler.php(66): Illuminate\\Cache\\Repository->put('QbR4kcNJjGvBFKp...', 'a:1:{s:6:\"_flas...', '120')
#16 /var/www/vendor/laravel/framework/src/Illuminate/Session/Store.php(129): Illuminate\\Session\\CacheBasedSessionHandler->write('QbR4kcNJjGvBFKp...', 'a:1:{s:6:\"_flas...')
#17 /var/www/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(87): Illuminate\\Session\\Store->save()
#18 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(218): Illuminate\\Session\\Middleware\\StartSession->terminate(Object(Illuminate\\Http\\Request), Object(Illuminate\\Http\\Response))
#19 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(189): Illuminate\\Foundation\\Http\\Kernel->terminateMiddleware(Object(Illuminate\\Http\\Request), Object(Illuminate\\Http\\Response))
#20 /var/www/public/index.php(60): Illuminate\\Foundation\\Http\\Kernel->terminate(Object(Illuminate\\Http\\Request), Object(Illuminate\\Http\\Response))
#21 {main}
"}

These are my ENV-Settings:

REDIS_HOST=redis
REDIS_PASSWORD=null
REDIS_PORT=6379

[it-can]

what redis driver do you use in Laravel? Predis or phpredis?

[floflock]

Predis and the setup in the documentation of laravel.

[floflock]

Interesting thing I found out now:

When i call var_dump(config('database.redis'));, this is the output:

array(2) {
  ["client"]=>
  string(6) "predis"
  ["default"]=>
  array(4) {
    ["host"]=>
    string(5) "redis"
    ["password"]=>
    NULL
    ["port"]=>
    string(22) "tcp://10.7.248.97:6379"
    ["database"]=>
    int(0)
  }
}

But this is in my config/database.php:

'redis' => [

        'client' => 'predis',

        'default' => [
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => 0,
        ],

    ],

[floflock]

Appendix: Looks like Kubernetes and Laravel are using the same ENV-variable but in a different way.

Because I have another container, the database, called mariadb and look at this:

echo $MARIADB_PORT
tcp://10.7.253.115:3306

So, my issue is specific, when you have a container called redis for redis purposes and using Laravel in another container. Then, the Laravel container has set the ENV-Variable from Kubernetes REDIS_PORT=tcp:://your-ip:port but Laravel uses the ENV-Variable this way: REDIS_PORT=6379

[tillkruss]

I'm not sure what we can do here, if Kubernetes changes environment variables you have to either open an issue with them, or maybe just add your own DatabaseServiceProvider that fixes the port.

[floflock]

I think, the framework is fine, nothing to do here. My solution was to change the config in config/database.php like that:

    'redis' => [

        'client' => 'predis',

        'default' => [
            'scheme' => 'tcp',
            'host' => env('REDIS_SERVICE_HOST', env('REDIS_HOST','127.0.0.1')),
            'port' => env('REDIS_SERVICE_PORT', env('REDIS_PORT',6379)),
            'password' => env('REDIS_PASSWORD', null),
            'database' => 0,
        ],

    ],

Now, the config checks first, if the REDIS_SERVICE_HOST and REDIS_SERVICE_PORT are present as ENV variable. This is the case, if you have a container in a docker/kubernetes cluster which is called REDIS.

Advantage of this solution is, that REDIS_SERVICE_HOST returns the IP address of the container, not a hostname. Therefore, there is no dns resolution anymore for this internal connections.

[stanliwise]

This issue also resurface if you host your laravel projects using same php files. even when multiple instance of php are running. environment variable collision occurs.

I had to name all variable differently across all projects.