New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Adding support for `.env` files #224

Closed
josegonzalez opened this Issue Mar 3, 2015 · 63 comments

Comments

Projects
None yet
@josegonzalez
Member

josegonzalez commented Mar 3, 2015

This is something that could help local developers configure their setups much quicker. With the added support for DSNs in the core, it is now quite easy to get setup.

Why

This isn't a new way to configure applications. Many of us are already taking advantage of this in production through process managers such as Circus, SupervisorD, and Upstart. As well, all popular webservers support the usage of environment variables (SetEnv in Apache, and various ways with Nginx). In production at least, environment variables allow the changing of configuration without a deploy, which can be a godsend if, for example, your database goes away and you need to swap to a backup.

The other issue I'd like to tackle is different development environments for teams. It's annoying to have to change hardcoded config when moving from one database config to another (we don't all have my_app:secret setup), and having a .env file can obviate the need for people stepping over each other while changing config.

How

We can use josegonzalez/php-dotenv, a library I maintain, or vlucas/phpdotenv, which is used in Laravel. Both solve similar problems in different ways. The advantage with the former is that we have more control over features we want/need, though to be honest both would work adequately.

Using my library, we can add the following to our bootstrap.php file:

try {
    josegonzalez\Dotenv\Loader::load([
        'filepath' => __DIR__ . DS . '.env',
        'toServer' => true,
        'skipExisting' => ['toServer'],
        'raiseExceptions' => true
    ]);
} catch (InvalidArgumentException $e) {
    // do nothing in case the file doesn't exist
}

Here is a sample .env file that I use for vagrant based installations:

export APP_NAME=app

export DEBUG=2
export SECURITY_SALT="DYhG93b0qyJfIxfs2guVoUubWwvniR2G0FgaC9mi"
export SECURITY_CIPHER_SEED="76859309657453542496749683645"

export DATABASE_URL="mysql://my_app:secret@localhost/my_app?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"
export DATABASE_TEST_URL="mysql://my_app:secret@localhost/my_app?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"

export CACHE_URL="file:///CACHE/?prefix=APP_NAME_&duration=DURATION"
export CACHE_CAKE_CORE_URL="file:///CACHE/persistent/?prefix=myapp_cake_core_&duration=DURATION&serialize=true"
export CACHE_CAKE_MODEL_URL="file:///CACHE/models/?prefix=myapp_cake_model_&duration=DURATION&serialize=true"

export LOG_DEBUG_URL=file:///LOGS/?levels[]=notice&levels[]=info&levels=[]=debug
export LOG_ERROR_URL=file:///LOGS/?levels[]=warning&levels[]=error&levels[]=critical&levels[]=alert&levels[]=emergency

export EMAIL_URL="mail://user:secret@localhost:25/?client=null&timeout=30&tls=null"

I do a few replacements in my app.php file like so:

'_cake_model_' => [
    'url' => str_replace(
        ['/CACHE/', 'DURATION'],
        [CACHE, '+2 minutes'],
        env('CACHE_CAKE_MODEL_URL')
    ),
],

I can likely make the library smarter and have it do it's replacements not only from bash but also from PHP constants or existing environment variables.

Errata

We'd need to add documentation on how .env files work. I have quite a bit of it in the readme for my project, but we'd want to add this to the main docs as well.

Note that this would maybe increase the complexity of setting up an app. We don't want to promote a .env file in production, but we do want to show users how to configure apps in production. Something to consider.

Developers might think this is something we've stolen from other frameworks, though this sort of thing has been around for quite a while. Rails has had automatic support for DSNs in environment variables since at least 3.x, and the friendsofcake/app-template project has used them for at least a year. As well, most/all of the core team has used this sort of functionality in the past, though maybe not with a php library to support their dev cycle.

Does adding this break BC? We've not yet made a release of the app composer project. Should we wait until a 3.1 CakePHP release before doing this? Or are we allowed to make changes like this without waiting for a major release.

Finally, does everyone desire this change? I know this makes it easy for me to write my book, and probably makes tutorial writing slightly easier, but maybe this isn't right for our users. Thoughts?

@josegonzalez josegonzalez changed the title from Adding support for `.env` files to RFC: Adding support for `.env` files Mar 3, 2015

@markstory

This comment has been minimized.

Member

markstory commented Mar 3, 2015

How is a dotenv file better than other config file formats. Given we'd need to have/use a parser that supports basic bash syntax the adds complexity over something like an ini file.

Could env files be integrated as config file reader for configure?

@markstory markstory added this to the 3.0.0 milestone Mar 3, 2015

@jadb

This comment has been minimized.

Member

jadb commented Mar 3, 2015

If we are to add .env support, I'd go with @josegonzalez's library for the exact same reasons he mentions and, having used it for some time now, I really like how configurable it is.

I also think that, similarly to what was done for cakephp/codeception, adding it in a transparent way (only the ones who require the plugin explicitly after install as --dev would get the functionality by checking for it), i.e.:

if (class_exists('\josegonzalez\Dotenv\Loader', true)) {
....
}
@jadb

This comment has been minimized.

Member

jadb commented Mar 3, 2015

To outline the reasons I prefer it is conditional to being explicitly required by the end-user:

1 - if not, a lot of users will end up having it but not knowing what it does or not using it.
2 - keeps documentation to the package instead of adding more documentation to the already extensive one that cake has (the bigger the less translations will be complete).

@markstory

This comment has been minimized.

Member

markstory commented Mar 3, 2015

Is there really much value to .env files if they are being parsed and handled by PHP? Seems like a fairly inefficient way to do configuration.

I totally buy in on using environment variables for config, I just don't think that parsing effectively bash files in PHP is a good idea. Could we approach this problem in a different way by having conventions around ENV vars that a config reader could then convert into structured data?

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Mar 3, 2015

Well it works well for much of our config, just not stuff like the error handler. I don't think having some convention like _ => . would be wise either.

For what it's worth, laravel basically calls env with smart defaults.

@jadb

This comment has been minimized.

Member

jadb commented Mar 3, 2015

IMO, env vars aren't for everyone. They've been working great for me and a lot of the core contributors I know and thus why I am biased on this RFC.

Here's how I do it since our env() doesn't support defaults: https://github.com/gourmet/platform/blob/master/config/database.php#L11-L14

But I do remember having hacked my own for passing defaults as well. Basically, the idea was to allow configuration to be made either way (using Configure or env vars). Something like this:

// basics.php
function read($key, $default = null)
{
    $result = env(strtoupper(str_replace('.', '_', $key));
    if (is_null($result) && Configure::check($key)) {
        $result = Configure::read($key);
    }
    if (is_null($result) && !is_null($default)) {
        $result = $default;
    }
    return $result;
}

Does that resemble what you are suggesting @markstory ?

@markstory

This comment has been minimized.

Member

markstory commented Mar 3, 2015

@jadb Not really, I was more wondering/proposing that env variables be sucked into Configure and renamed similar to _ => .. The read() function you have could work well, but I don't know how that would work with the rest of the app skeleton and other parts of CakePHP that heavily rely on Configure::read().

@ionas

This comment has been minimized.

Contributor

ionas commented Mar 20, 2015

Rails is doing a similar thing with secrets.yml
http://guides.rubyonrails.org/4_1_release_notes.html#config-secrets-yml
(It is not an .env but it seems to solve similar problems).

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Mar 22, 2015

@markstory The only problem with that is Error.exceptionRenderer being inflected the way it is.

So I guess the two changes that can come about are:

  • Adding a default key to the env() global
  • Adding a Configure implementation that reads in env vars and normalizes them to cakephp config.

I'm not sure how I feel about the second, but the first sounds like it could be generally useful and I'd be okay with that one.

@robertpustulka

This comment has been minimized.

Member

robertpustulka commented Apr 1, 2015

Wouldn't it be easier just to have default app.php config and override it with dev.php / production.php / backup.php / etc ? The name of a config file could be resolved from env.

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Apr 1, 2015

I'm not a fan of environment-specific config files. Having done so at scale, I can say that road is the road to failure. If you're going to resolve a filename based on env var, then why not just support the env vars themselves.

@jadb

This comment has been minimized.

Member

jadb commented Apr 2, 2015

I side with @josegonzalez. Again, having used both scenarios, environment variables all the way (added benefit: automatic support for heroku-style of hosting).

@AD7six

This comment has been minimized.

Member

AD7six commented Apr 2, 2015

I'm not a fan of environment-specific config files

Me neither. There is this environment, and environments that are irrelevant to the current install.

@lorenzo

This comment has been minimized.

Member

lorenzo commented Apr 2, 2015

Should we start listing the env names that we would use by conventions?

@markstory

This comment has been minimized.

Member

markstory commented Apr 2, 2015

@josegonzalez If the biggest blocker around env var inflection is the exception handler, couldn't we have that code read both the new and 'old' configuration names?

@lorenzo

This comment has been minimized.

Member

lorenzo commented Apr 2, 2015

I'd like that solution @markstory

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented May 24, 2015

I added support to php-dotenv to parse .env file output into the cakephp output:

<?php
// contents of a config class
$dotenv = new josegonzalez\Dotenv\Loader('path/to/.env');
$dotenv->setFilters(['josegonzalez\Dotenv\Filter\UnderscoreArrayFilter']);
$dotenv->parse();
$dotenv->filter();
return $dotenv->toArray();

A bit possible now I guess. Not sure anymore if we should add it?

@markstory

This comment has been minimized.

Member

markstory commented May 24, 2015

@josegonzalez We could include it for 3.1 as an alternative to using the current php config files.

@htstudios

This comment has been minimized.

Contributor

htstudios commented Aug 12, 2015

Would love that.

@jadb

This comment has been minimized.

Member

jadb commented Aug 13, 2015

👍

1 similar comment
@netstyler

This comment has been minimized.

netstyler commented Aug 26, 2015

👍

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Aug 26, 2015

We'd need to have a patch for the aforementioned Error.exceptionRenderer issue. @markstory would this still be able to make it into 3.1 in the app repo if I made the above change?

@markstory

This comment has been minimized.

Member

markstory commented Aug 26, 2015

I was hoping to do the 3.1 RC this weekend. Adding configuration fallbacks sounds like the kind of thing we can squeeze in during an RC.

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Aug 26, 2015

Just did a simple test, and it looks like even with that, there would still be quite a few special cases we'd need to handle, so not worth doing in the core at all. I'll think on it a bit more.

@markstory

This comment has been minimized.

Member

markstory commented Aug 27, 2015

What are the other special cases @josegonzalez ?

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Aug 27, 2015

Anything with className. Also, the special cache configs with underscores, amongst others...

@markstory

This comment has been minimized.

Member

markstory commented Aug 27, 2015

Hrm.

@burzum

This comment has been minimized.

Member

burzum commented Sep 13, 2015

👍 Would be nice to see that in the core.

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Oct 21, 2015

I would rather we not prefix with CAKEPHP everywhere, especially for stuff like datastores, where we could use DATABASE_URL or CACHEL_DEFAULT_URL.

@burzum

This comment has been minimized.

Member

burzum commented Oct 21, 2015

N00b question: How does this work with multiple apps on the same machine? I've never used the .env files but I'm teased to give it a try.

Is the content of the .env file written to the global environment or just made available for env() within the Cake app? Because if I have two apps and both use the same names they'll get overriden?

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Oct 21, 2015

Your apache/nginx configuration would pass parameters to the fasctgi/fpm php process with these values. For background tasks, you would source in a .env file with config specific to your app before running the command.

@cake17

This comment has been minimized.

Member

cake17 commented Oct 22, 2015

@josegonzalez 👍 for URL instead of DSN suffix. For the prefix, I would prefer writing a prefix CAKEPHP_ that reminds us that we give this variable to the cakephp app configuration, that itself will take cake of the service's config.

But I can understand this can be too much verbose, and I'm not against removing the CAKEPHP_ prefix and name the env variables CACHE_DEFAULT_URL and DATABASE_URL.

3 env variables we could add (with or without CAKEPHP_ prefix`):

  • CAKEPHP_TIMEZONE_DEFAULT (by default UTC)
  • CAKEPHP_LOCALE_DEFAULT (by default en_US)
  • CAKEPHP_ENCODING_DEFAULT (by default UTF-8)
@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Oct 22, 2015

The standard is sans prefix for most of these things. There isn't a SYMFONY_DATABASE_URL or a FLASK_DATABASE_URL. Just DATABASE_URL.

@cake17

This comment has been minimized.

Member

cake17 commented Oct 22, 2015

@josegonzalez so no CAKEPHP_ prefix is good to me

@markstory markstory modified the milestones: 3.1.2, 3.1.3 Nov 18, 2015

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Dec 25, 2015

Note: One thing I've noticed is that for some keys in the app.php file, there aren't "sane" defaults anywhere. For instance, we have bare calls to App.encoding across the app, so if you removed it, you could potentially break your application.

A few keys that don't follow our "normal" naming schema:

  • App.wwwRoot
  • App.baseUrl
  • App.fullBaseUrl
  • App.imageBaseUrl
  • App.cssBaseUrl
  • App.jsBaseUrl
  • Error.errorLevel
  • Error.skipLog (also this is an array!)
  • Error.exceptionRenderer
  • Email.default.headerCharset
  • Anything with className (though you can get around this by using a url)
  • The prefix EmailTransport. The whole Email vs EmailTransport is confusing btw. Should have been EmailProfile and Email, respectively.

For the most part, people don't need to modify the above via env vars, so thats nice. The real issue is just setting defaults for things that matter, like debug or Datasources.default.url. Luckily, the url keys are merged with the rest. If they are empty, they won't result in a bad app config, assuming your defaults are ok.

@markstory

This comment has been minimized.

Member

markstory commented Dec 25, 2015

EmailProfile and EmailTransport would have been better in hindsight. What about those variables isn't normal? Is it that they have multi word keys with camelBacked names?

@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Dec 25, 2015

Yeah, they end up getting multi-word keys if you do naive splitting.

Here is what I have now:

#!/usr/bin/env bash
# in config/.env

export APP_NAME=myapp

export DEBUG=true
export SECURITY_SALT="SOME_SALT1"

export DATABASE_URL="mysql://user:password@localhost/${APP_NAME}?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"
export TEST_DATABASE_URL="mysql://user:password@localhost/test_${APP_NAME}?encoding=utf8&timezone=UTC&cacheMetadata=true&quoteIdentifiers=false&persistent=false"

export CACHE_DURATION="2+ minutes"
export CACHE_DEFAULT_URL="file:///tmp/cache?prefix=${APP_NAME}_&duration=${CACHE_DURATION}"
export CACHE_CAKECORE_URL="file:///tmp/cache/persistent?prefix=${APP_NAME}_cake_core_&duration=${CACHE_DURATION}&serialize=true"
export CACHE_CAKEMODEL_URL="file:///tmp/cache/models?prefix=${APP_NAME}_cake_model_&duration=${CACHE_DURATION}&serialize=true"

export LOG_DEBUG_URL=file://logs?levels[]=notice&levels[]=info&levels[]=debug&file=debug
export LOG_ERROR_URL=file://logs?levels[]=warning&levels[]=error&levels[]=critical&levels[]=alert&levels[]=emergency&file=error

export EMAILTRANSPORT_DEFAULT_URL="mail://user:secret@localhost:25/?client=null&timeout=30&tls=null"
<?php
// in config/env.php
use josegonzalez\Dotenv\Loader;
use Cake\Utility\Hash;

$config = [];
if (!env('APP_NAME')) {
    $dotenv = new Loader([
        __DIR__ . DS . '.env',
        __DIR__ . DS . '.env.default',
    ]);
    $dotenv->setFilters([
        'josegonzalez\Dotenv\Filter\LowercaseKeyFilter',
        'josegonzalez\Dotenv\Filter\UppercaseFirstKeyFilter',
        'josegonzalez\Dotenv\Filter\UnderscoreArrayFilter',
        function ($data) {
            $keys = [
                'Debug' => 'debug',
                'Emailtransport' => 'EmailTransport',
                'Database' => 'Datasources.default',
                'Test.database' => 'Datasources.test',
                'Test' => null,
                'Cache.duration' => null,
                'Cache.cakemodel' => 'Cache._cake_model_',
                'Cache.cakecore' => 'Cache._cake_core_',
            ];
            foreach ($keys as $key => $newKey) {
                if ($newKey === null) {
                    $data = Hash::remove($data, $key);
                    continue;
                }
                $value = Hash::get($data, $key);
                $data = Hash::remove($data, $key);
                $data = Hash::insert($data, $newKey, $value);
            }
            return $data;
        }
    ]);
    $dotenv->parse();
    $dotenv->filter();
    $config = $dotenv->toArray();
}
return $config;
// modified my config/bootstrap.php
try {
    Configure::config('default', new PhpConfig());
    Configure::load('app', 'default', false);
    Configure::load('env', 'default', true);
} catch (\Exception $e) {
    die($e->getMessage() . "\n");
}

This will load up my env from a .env file, but only when APP_NAME is not set as an environmnent variable. This allows me to skip setup completey and just use my config/app.php, which has been modified to read from the aforementioned environment variables directly using env('KEY', 'DEFAULT').

A few things I think we can do to ease this:

  1. Use env() to read in a few keys in our app.default.php. I use it for anything that takes a url, as well as the security salt and debug value.
  2. Potentially add my sample app.env.php and a commented out Configure::load() statement.

I'm happy to make a PR for 1, but understand us not wanting to do 2 (and honestly think such a thing belongs in a custom cakephp project skeleton).

Thoughts?

@markstory

This comment has been minimized.

Member

markstory commented Dec 26, 2015

I think 1 sounds reasonable and mirrors what I've seen done in other frameworks. Just so I'm clear the would app.php contain something like:

return [
  'debug' => env('CAKEPHP_DEBUG', true),
  'Datasource' => [
    'default' => [
      'url' => env('CAKEPHP_DATASOURCE_DEFAULT')
    ]
  ]
];
@cake17

This comment has been minimized.

Member

cake17 commented Jan 17, 2016

@josegonzalez I'm 👍 for env() implementation.

Could we add other key configs in app.php like:

  • date_default_timezone_set
  • default_locale
    Those keys would then be configurable with env variables.
@josegonzalez

This comment has been minimized.

Member

josegonzalez commented Jan 17, 2016

I don't think we should promote changing the default timezone. Server/database time should always be in UTC to prevent errors in application configuration. I'd be happy to discuss this in depth, but I stand by that rule and so won't make any effort to do otherwise.

Regarding default_locale, that makes sense to me. I'll add it as a new config under App.

josegonzalez added a commit that referenced this issue Jan 17, 2016

josegonzalez added a commit that referenced this issue Jan 17, 2016

@markstory markstory modified the milestones: 3.1.3, 3.2.1 Feb 2, 2016

@bravo-kernel

This comment has been minimized.

Contributor

bravo-kernel commented Jun 3, 2016

👏

@arturmamedov

This comment has been minimized.

arturmamedov commented Jun 29, 2016

@josegonzalez By using your configurations i have the error:

The "_cake_core_" cache configuration does not exist. 
and 
The "_cake_model_" cache configuration does not exist. 

And its because i commented export CACHE_CAKECORE_URL and export CACHE_CAKEMODEL_URL, can be this the issue and why?

Thank you very much, very usefull thing the .env configuration!

@anhtuank7c

This comment has been minimized.

Contributor

anhtuank7c commented Jun 29, 2016

@arturmamedov
This is not support area.
Please go to IRC, Slack

@inoas

This comment has been minimized.

Contributor

inoas commented Jan 25, 2017

@josegonzalez the env loader is still not part of cakephp, right?
Maybe it should? @ https://twitter.com/symfonydocs/status/823824851290955776

@markstory markstory referenced this issue Mar 16, 2017

Closed

Is there a plan to support .env? #10420

2 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment