RFC: Adding support for `.env` files

[josegonzalez]

This is something that could help local developers configure their setups much quicker. With the added support for DSNs in the core, it is now quite easy to get setup.

Why

This isn't a new way to configure applications. Many of us are already taking advantage of this in production through process managers such as Circus, SupervisorD, and Upstart. As well, all popular webservers support the usage of environment variables (SetEnv in Apache, and various ways with Nginx). In production at least, environment variables allow the changing of configuration without a deploy, which can be a godsend if, for example, your database goes away and you need to swap to a backup.

The other issue I'd like to tackle is different development environments for teams. It's annoying to have to change hardcoded config when moving from one database config to another (we don't all have my_app:secret setup), and having a .env file can obviate the need for people stepping over each other while changing config.

How

We can use josegonzalez/php-dotenv, a library I maintain, or vlucas/phpdotenv, which is used in Laravel. Both solve similar problems in different ways. The advantage with the former is that we have more control over features we want/need, though to be honest both would work adequately.

Using my library, we can add the following to our bootstrap.php file:

try {
    josegonzalez\Dotenv\Loader::load([
        'filepath' => __DIR__ . DS . '.env',
        'toServer' => true,
        'skipExisting' => ['toServer'],
        'raiseExceptions' => true
    ]);
} catch (InvalidArgumentException $e) {
    // do nothing in case the file doesn't exist
}

Here is a sample .env file that I use for vagrant based installations:

export APP_NAME=app

export DEBUG=2
export SECURITY_SALT="DYhG93b0qyJfIxfs2guVoUubWwvniR2G0FgaC9mi"
export SECURITY_CIPHER_SEED="76859309657453542496749683645"

export DATABASE_URL="mysql://my_app:secret@localhost/my_app?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"
export DATABASE_TEST_URL="mysql://my_app:secret@localhost/my_app?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"

export CACHE_URL="file:///CACHE/?prefix=APP_NAME_&duration=DURATION"
export CACHE_CAKE_CORE_URL="file:///CACHE/persistent/?prefix=myapp_cake_core_&duration=DURATION&serialize=true"
export CACHE_CAKE_MODEL_URL="file:///CACHE/models/?prefix=myapp_cake_model_&duration=DURATION&serialize=true"

export LOG_DEBUG_URL=file:///LOGS/?levels[]=notice&levels[]=info&levels=[]=debug
export LOG_ERROR_URL=file:///LOGS/?levels[]=warning&levels[]=error&levels[]=critical&levels[]=alert&levels[]=emergency

export EMAIL_URL="mail://user:secret@localhost:25/?client=null&timeout=30&tls=null"

I do a few replacements in my app.php file like so:

'_cake_model_' => [
    'url' => str_replace(
        ['/CACHE/', 'DURATION'],
        [CACHE, '+2 minutes'],
        env('CACHE_CAKE_MODEL_URL')
    ),
],

I can likely make the library smarter and have it do it's replacements not only from bash but also from PHP constants or existing environment variables.

Errata

We'd need to add documentation on how .env files work. I have quite a bit of it in the readme for my project, but we'd want to add this to the main docs as well.

Note that this would maybe increase the complexity of setting up an app. We don't want to promote a .env file in production, but we do want to show users how to configure apps in production. Something to consider.

Developers might think this is something we've stolen from other frameworks, though this sort of thing has been around for quite a while. Rails has had automatic support for DSNs in environment variables since at least 3.x, and the friendsofcake/app-template project has used them for at least a year. As well, most/all of the core team has used this sort of functionality in the past, though maybe not with a php library to support their dev cycle.

Does adding this break BC? We've not yet made a release of the app composer project. Should we wait until a 3.1 CakePHP release before doing this? Or are we allowed to make changes like this without waiting for a major release.

Finally, does everyone desire this change? I know this makes it easy for me to write my book, and probably makes tutorial writing slightly easier, but maybe this isn't right for our users. Thoughts?

[markstory]

How is a dotenv file better than other config file formats. Given we'd need to have/use a parser that supports basic bash syntax the adds complexity over something like an ini file.

Could env files be integrated as config file reader for configure?

[jadb]

If we are to add .env support, I'd go with @josegonzalez's library for the exact same reasons he mentions and, having used it for some time now, I really like how configurable it is.

I also think that, similarly to what was done for cakephp/codeception, adding it in a transparent way (only the ones who require the plugin explicitly after install as --dev would get the functionality by checking for it), i.e.:

if (class_exists('\josegonzalez\Dotenv\Loader', true)) {
....
}

[jadb]

To outline the reasons I prefer it is conditional to being explicitly required by the end-user:

1 - if not, a lot of users will end up having it but not knowing what it does or not using it.
2 - keeps documentation to the package instead of adding more documentation to the already extensive one that cake has (the bigger the less translations will be complete).

[markstory]

Is there really much value to .env files if they are being parsed and handled by PHP? Seems like a fairly inefficient way to do configuration.

I totally buy in on using environment variables for config, I just don't think that parsing effectively bash files in PHP is a good idea. Could we approach this problem in a different way by having conventions around ENV vars that a config reader could then convert into structured data?

[josegonzalez]

Well it works well for much of our config, just not stuff like the error handler. I don't think having some convention like _ => . would be wise either.

For what it's worth, laravel basically calls env with smart defaults.

[jadb]

IMO, env vars aren't for everyone. They've been working great for me and a lot of the core contributors I know and thus why I am biased on this RFC.

Here's how I do it since our env() doesn't support defaults: https://github.com/gourmet/platform/blob/master/config/database.php#L11-L14

But I do remember having hacked my own for passing defaults as well. Basically, the idea was to allow configuration to be made either way (using Configure or env vars). Something like this:

// basics.php
function read($key, $default = null)
{
    $result = env(strtoupper(str_replace('.', '_', $key));
    if (is_null($result) && Configure::check($key)) {
        $result = Configure::read($key);
    }
    if (is_null($result) && !is_null($default)) {
        $result = $default;
    }
    return $result;
}

Does that resemble what you are suggesting @markstory ?

[markstory]

@jadb Not really, I was more wondering/proposing that env variables be sucked into Configure and renamed similar to _ => .. The read() function you have could work well, but I don't know how that would work with the rest of the app skeleton and other parts of CakePHP that heavily rely on Configure::read().

[ionas]

Rails is doing a similar thing with secrets.yml
http://guides.rubyonrails.org/4_1_release_notes.html#config-secrets-yml
(It is not an .env but it seems to solve similar problems).

[josegonzalez]

@markstory The only problem with that is Error.exceptionRenderer being inflected the way it is.

So I guess the two changes that can come about are:

• Adding a default key to the env() global
• Adding a Configure implementation that reads in env vars and normalizes them to cakephp config.

I'm not sure how I feel about the second, but the first sounds like it could be generally useful and I'd be okay with that one.

[robertpustulka]

Wouldn't it be easier just to have default app.php config and override it with dev.php / production.php / backup.php / etc ? The name of a config file could be resolved from env.

[josegonzalez]

I'm not a fan of environment-specific config files. Having done so at scale, I can say that road is the road to failure. If you're going to resolve a filename based on env var, then why not just support the env vars themselves.

[jadb]

I side with @josegonzalez. Again, having used both scenarios, environment variables all the way (added benefit: automatic support for heroku-style of hosting).

[AD7six]

I'm not a fan of environment-specific config files

Me neither. There is this environment, and environments that are irrelevant to the current install.

[lorenzo]

Should we start listing the env names that we would use by conventions?

[markstory]

@josegonzalez If the biggest blocker around env var inflection is the exception handler, couldn't we have that code read both the new and 'old' configuration names?

[lorenzo]

I'd like that solution @markstory

[josegonzalez]

I added support to php-dotenv to parse .env file output into the cakephp output:

<?php
// contents of a config class
$dotenv = new josegonzalez\Dotenv\Loader('path/to/.env');
$dotenv->setFilters(['josegonzalez\Dotenv\Filter\UnderscoreArrayFilter']);
$dotenv->parse();
$dotenv->filter();
return $dotenv->toArray();

A bit possible now I guess. Not sure anymore if we should add it?

[markstory]

@josegonzalez We could include it for 3.1 as an alternative to using the current php config files.

[htstudios]

Would love that.

[jadb]

👍

[netstyler]

👍

[josegonzalez]

We'd need to have a patch for the aforementioned Error.exceptionRenderer issue. @markstory would this still be able to make it into 3.1 in the app repo if I made the above change?

[markstory]

I was hoping to do the 3.1 RC this weekend. Adding configuration fallbacks sounds like the kind of thing we can squeeze in during an RC.

[josegonzalez]

Just did a simple test, and it looks like even with that, there would still be quite a few special cases we'd need to handle, so not worth doing in the core at all. I'll think on it a bit more.

[markstory]

What are the other special cases @josegonzalez ?

[josegonzalez]

Anything with className. Also, the special cache configs with underscores, amongst others...

[markstory]

Hrm.

[burzum]

👍 Would be nice to see that in the core.

[josegonzalez]

I would rather we not prefix with CAKEPHP everywhere, especially for stuff like datastores, where we could use DATABASE_URL or CACHEL_DEFAULT_URL.

[burzum]

N00b question: How does this work with multiple apps on the same machine? I've never used the .env files but I'm teased to give it a try.

Is the content of the .env file written to the global environment or just made available for env() within the Cake app? Because if I have two apps and both use the same names they'll get overriden?

[josegonzalez]

Your apache/nginx configuration would pass parameters to the fasctgi/fpm php process with these values. For background tasks, you would source in a .env file with config specific to your app before running the command.

[cake17]

@josegonzalez 👍 for URL instead of DSN suffix. For the prefix, I would prefer writing a prefix CAKEPHP_ that reminds us that we give this variable to the cakephp app configuration, that itself will take cake of the service's config.

But I can understand this can be too much verbose, and I'm not against removing the CAKEPHP_ prefix and name the env variables CACHE_DEFAULT_URL and DATABASE_URL.

3 env variables we could add (with or without CAKEPHP_ prefix`):

• CAKEPHP_TIMEZONE_DEFAULT (by default UTC)
• CAKEPHP_LOCALE_DEFAULT (by default en_US)
• CAKEPHP_ENCODING_DEFAULT (by default UTF-8)

[josegonzalez]

The standard is sans prefix for most of these things. There isn't a SYMFONY_DATABASE_URL or a FLASK_DATABASE_URL. Just DATABASE_URL.

[cake17]

@josegonzalez so no CAKEPHP_ prefix is good to me

[josegonzalez]

Note: One thing I've noticed is that for some keys in the app.php file, there aren't "sane" defaults anywhere. For instance, we have bare calls to App.encoding across the app, so if you removed it, you could potentially break your application.

A few keys that don't follow our "normal" naming schema:

• App.wwwRoot
• App.baseUrl
• App.fullBaseUrl
• App.imageBaseUrl
• App.cssBaseUrl
• App.jsBaseUrl
• Error.errorLevel
• Error.skipLog (also this is an array!)
• Error.exceptionRenderer
• Email.default.headerCharset
• Anything with className (though you can get around this by using a url)
• The prefix EmailTransport. The whole Email vs EmailTransport is confusing btw. Should have been EmailProfile and Email, respectively.

For the most part, people don't need to modify the above via env vars, so thats nice. The real issue is just setting defaults for things that matter, like debug or Datasources.default.url. Luckily, the url keys are merged with the rest. If they are empty, they won't result in a bad app config, assuming your defaults are ok.

[markstory]

EmailProfile and EmailTransport would have been better in hindsight. What about those variables isn't normal? Is it that they have multi word keys with camelBacked names?

[josegonzalez]

Yeah, they end up getting multi-word keys if you do naive splitting.

Here is what I have now:

#!/usr/bin/env bash
# in config/.env

export APP_NAME=myapp

export DEBUG=true
export SECURITY_SALT="SOME_SALT1"

export DATABASE_URL="mysql://user:password@localhost/${APP_NAME}?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"
export TEST_DATABASE_URL="mysql://user:password@localhost/test_${APP_NAME}?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"

export CACHE_DURATION="2+ minutes"
export CACHE_DEFAULT_URL="file:///tmp/cache?prefix=${APP_NAME}_&duration=${CACHE_DURATION}"
export CACHE_CAKECORE_URL="file:///tmp/cache/persistent?prefix=${APP_NAME}_cake_core_&duration=${CACHE_DURATION}&serialize=true"
export CACHE_CAKEMODEL_URL="file:///tmp/cache/models?prefix=${APP_NAME}_cake_model_&duration=${CACHE_DURATION}&serialize=true"

export LOG_DEBUG_URL=file://logs?levels[]=notice&levels[]=info&levels[]=debug&file=debug
export LOG_ERROR_URL=file://logs?levels[]=warning&levels[]=error&levels[]=critical&levels[]=alert&levels[]=emergency&file=error

export EMAILTRANSPORT_DEFAULT_URL="mail://user:secret@localhost:25/?client=null&timeout=30&tls=null"

<?php
// in config/env.php
use josegonzalez\Dotenv\Loader;
use Cake\Utility\Hash;

$config = [];
if (!env('APP_NAME')) {
    $dotenv = new Loader([
        __DIR__ . DS . '.env',
        __DIR__ . DS . '.env.default',
    ]);
    $dotenv->setFilters([
        'josegonzalez\Dotenv\Filter\LowercaseKeyFilter',
        'josegonzalez\Dotenv\Filter\UppercaseFirstKeyFilter',
        'josegonzalez\Dotenv\Filter\UnderscoreArrayFilter',
        function ($data) {
            $keys = [
                'Debug' => 'debug',
                'Emailtransport' => 'EmailTransport',
                'Database' => 'Datasources.default',
                'Test.database' => 'Datasources.test',
                'Test' => null,
                'Cache.duration' => null,
                'Cache.cakemodel' => 'Cache._cake_model_',
                'Cache.cakecore' => 'Cache._cake_core_',
            ];
            foreach ($keys as $key => $newKey) {
                if ($newKey === null) {
                    $data = Hash::remove($data, $key);
                    continue;
                }
                $value = Hash::get($data, $key);
                $data = Hash::remove($data, $key);
                $data = Hash::insert($data, $newKey, $value);
            }
            return $data;
        }
    ]);
    $dotenv->parse();
    $dotenv->filter();
    $config = $dotenv->toArray();
}
return $config;

// modified my config/bootstrap.php
try {
    Configure::config('default', new PhpConfig());
    Configure::load('app', 'default', false);
    Configure::load('env', 'default', true);
} catch (\Exception $e) {
    die($e->getMessage() . "\n");
}

This will load up my env from a .env file, but only when APP_NAME is not set as an environmnent variable. This allows me to skip setup completey and just use my config/app.php, which has been modified to read from the aforementioned environment variables directly using env('KEY', 'DEFAULT').

A few things I think we can do to ease this:

1. Use env() to read in a few keys in our app.default.php. I use it for anything that takes a url, as well as the security salt and debug value.
2. Potentially add my sample app.env.php and a commented out Configure::load() statement.

I'm happy to make a PR for 1, but understand us not wanting to do 2 (and honestly think such a thing belongs in a custom cakephp project skeleton).

Thoughts?

[markstory]

I think 1 sounds reasonable and mirrors what I've seen done in other frameworks. Just so I'm clear the would app.php contain something like:

return [
  'debug' => env('CAKEPHP_DEBUG', true),
  'Datasource' => [
    'default' => [
      'url' => env('CAKEPHP_DATASOURCE_DEFAULT')
    ]
  ]
];

[cake17]

@josegonzalez I'm 👍 for env() implementation.

Could we add other key configs in app.php like:

• date_default_timezone_set
• default_locale
  Those keys would then be configurable with env variables.

[josegonzalez]

I don't think we should promote changing the default timezone. Server/database time should always be in UTC to prevent errors in application configuration. I'd be happy to discuss this in depth, but I stand by that rule and so won't make any effort to do otherwise.

Regarding default_locale, that makes sense to me. I'll add it as a new config under App.

[bravo-kernel]

👏

[arturmamedov]

@josegonzalez By using your configurations i have the error:

The "_cake_core_" cache configuration does not exist. 
and 
The "_cake_model_" cache configuration does not exist. 

And its because i commented export CACHE_CAKECORE_URL and export CACHE_CAKEMODEL_URL, can be this the issue and why?

Thank you very much, very usefull thing the .env configuration!

[anhtuank7c]

@arturmamedov
This is not support area.
Please go to IRC, Slack

[inoas]

@josegonzalez the env loader is still not part of cakephp, right?
Maybe it should? @ https://twitter.com/symfonydocs/status/823824851290955776