[DependencyInjection] Make accessing environment variables in configuration more convenient.

[magnusnordlander]

Q	A
Branch	master
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets
License	MIT
Doc PR

There have been several attempts to solve the issue of dynamic environment variables in Symfony (the issues are outlined in #10138, #7555). The latest stab at the issue that I am aware of was #16403, but that was closed as Fabien concluded that "a generic system is just not possible without rethinking the whole system", and that "another possibility is to only add support for some bundles like the DSN for the Doctrine bundle" (see #16403 (comment)).

In order to make it easier on bundles to support allowing environment variables in their configuration, I propose adding an expression language function to get environment variables from service expressions, and a trait to more easily parse such an expression from the bundle extension.

This could easily be made in a separate bundle (see my own https://github.com/fervo/EnvironmentBundle), but I feel that adoption will be far higher if we do it in core. Using the trait, configuration with an environment variable is quite simple, something along the following line works brilliantly:

fervo_pubnub:
    publish_key: "@=env('PUBNUB_PUBLISH_KEY')"
    subscribe_key: "@=env('PUBNUB_SUBSCRIBE_KEY')"

If the extension uses the configuration values as service arguments verbatim, all it has to do is something like the following:

        $def = $container->getDefinition('fervo_pubnub');
        $def->replaceArgument(0, $this->resolve($config['publish_key']));
        $def->replaceArgument(1, $this->resolve($config['subscribe_key']));

It does require opt-in from all bundles wishing to use this, and for all configuration values it is used on, but I think that is the best way forward.

[patrick-mcdougle]

Is it possible to include unit tests?

[stloyd]

Instead of strlen() > 0 you should use '' !== $value, also strict check for this: '@' === $value[0].

[magnusnordlander]

@patrick-mcdougle: Sure, I'll be happy to write some unit tests if you folks think this approach has merit :)

[wouterj]

I don't like the idea of having the same @ and @= syntax for configuration options as for service arguments. They are fundametally different.

If an option allows to pass an expression, it should just allow the expression without any need for crazy prefixes imo. See for instance the options allowing expressions in Symfony.

[magnusnordlander]

@wouterj Without any form of prefix? How would you distinguish between an expression and just a string value?

[dupuchba]

I really like the idea which is a step further in supporting the twelve factor-app. Would be nice to ear about @symfony/deciders point of view

[dzuelke]

+100000. Has @fabpot seen this? :) /cc @stof

[fabpot]

@magnusnordlander Can you give us a usage of the trait? I don't understand how that works. @= is already used by the YAML loader for expressions. For other configuration formats, we have some other ways. But AFAIU, your example being YAML, it's going to be caught bu the loader and converted right away. Am I missing something here?

[magnusnordlander]

@fabpot The YAML loader itself only runs in service definitions. This functionality is indeed USING the expression language. It simply registers a new expression function that gets compiled to a getenv call, and makes it easy to parse it from the extension.

[wouterj]

I don't really like the trait for the extension. Using @... is related to service references. Service references make no sense as configuration values. If a service reference is needed, the setting should only allow passing the service ID (like is done in many places already).

However, if an option has a scalar, I think it makes sense to allow passing an expression in there. I don't think we should reuse service configuration syntax here though.

[nicolas-grekas]

Since ES is a soft dep of DI, would it make sense to throw an exception here before triggering a fatal error when this syntax is used but ES is not available? We do so in other parts AFAIK

[magnusnordlander]

Okay, I'm absolutely open to an alternate syntax.

How about just skipping the @? That could be something like the following:

acme_bundle:
    some_value: "Just another scalar"
    some_other_value: "=env('PATH')" # or any other valid service container expression
    some_third_value: "== to escape strings starting with = (similar to the %% syntax)"

Reasonable?

[fabpot]

I still don't get it. =@ is already how to use an expression in the YAML format, it's not new in this PR.

[wouterj]

@fabpot @= is only used in the argument section of a service definition, right?

If I undestand this PR correctly, @magnusnordlander wants to support this format everywhere in the config. E.g. framework: { secret: "@= getenv('SF_SECRET')" }

[magnusnordlander]

@wouterj @fabpot That was my understanding as well of the current state of affairs.

And Wouter is mostly correct. I want to make it easy to opt in to supporting that format everywhere in the config.

[atorian]

What is the benefit of reading ENV on every run? And why SYMFONY__ENV vars can't be used instead?

[magnusnordlander]

Because in some deployment environments (Heroku, Docker etc.), environment variables can change after the container has been built. If you're using SYMFONY__ENV vars, they'll be cached in the dumped container, necessitating a cache rebuild on environment changes.

[atorian]

Docker env does not change after container was started.If it does - then issue is not in the symfony itself. However Symfony as it is now does not fit dockerised ENV.
This issue should be solved by your build process, not by framework.
Build your conteiner without prod cache and let it be generated on the first run and provide correct vars. Or use docker entypoint and clear prod cache on every container run.

[dzuelke]

It is an issue in Symfony. No other language or framework I know does such caching. The whole point of Docker, Heroku, and all the Twelve-Factor systems is that configuration happens through the environment, and immediately gets applied. Some people have cache warmup procedures that run for minutes, not seconds (don't ask me why), and doing that on each container run defeats the whole purpose.

[atorian]

12 factor approach does not tell anything about caching of the env vars. As well as about the approach when they should chenge between the deployments, but it mentions:

independently managed for each deploy

It means that there should be some deploy procedure, which includes clear prod cache.

As for cache warmers - why should framework add a workaround for bad app design?
If you want a it - implement it in your app/bundle but not in the framework.
Probably some event listener which will update config from env on every run will be a better solution.

And if your config changes between deployments - probably other config storage will suit better.

[magnusnordlander]

@atorian:

Probably some event listener which will update config from env on every run will be a better solution.

This is not possible without a major API break for DI extensions or serious runtime performance penalties.

probably other config storage will suit better

When using semantic bundle configuration, there is no possibility for another config storage without a major API break. All configuration gets processed by DI extensions and compiler passes, then cached in the compiled DIC.

Furthermore, I quite like the elegance of the Heroku build system. Your application is turned into an immutable slug of code, with an already warmed cache, configured only using environment variables. This should be preferable in a Dockerized environment as well, to avoid the overhead of warming the cache on start, and instead have an already hot version in your container.

[dzuelke]

12 factor approach does not tell anything about caching of the env vars

Yes it does. A build runs once, and is combined with configuration from the environment into a release. If the config changes, no new build is performed. That's factor number V: http://12factor.net/build-release-run

The whole purpose of env vars for runtime configuration is that you do not need to re-build your code. Emergency/maintenance/readonly modes, split factors for A/B testing, database credentials on failover - these are things you want, and should be able, to change quickly.

Docker has separate build time (ARG) and run time (ENV) config specifically because of this distinction. Symfony is the only framework that does caching of this kind, and that's because with PHP's execution model, the startup of a framework is expensive because it happens for each request. In languages with standards for native web servers and its communication with frameworks (Servlets, Rack, WSGI), this is not a concern.

Back when the config cache got designed for Symfony 2, the possibility of config to change without a code/cache re-build was simply not considered. That's no big deal, but it doesn't mean it doesn't need fixing.

[atorian]

@magnusnordlander

When using semantic bundle configuration, there is no possibility for another config storage without a major API break. All configuration gets processed by DI extensions and compiler passes, then cached in the compiled DIC.

Symfony guide us to use %parameters% for this. Not expressions. And it offers us SYMFONY_ENV_VARS to assign this parameters and an extension point in form of php version of parameters file, where you can use getenv('CUSTOM_ENV_VAR') and assign it to a parameter.

Consider prod cache as safety net. Imagine we've disabled caching for env vars - now, if we rename env var or remove it from your heroku admin panel - should all the hosted apps die? It means that to prevent it - we have to maintain both configurations in the code, what does not make any sense for our releases. Just brings more complexity and room for bugs.

What about rollback? If env is changed how to automate this without a cache? Sure it's possible to do it without a cache, but solution is already there.

Using cache we can easily change env and then trigger deploy procedure with new clean version of app. Or just clear the cache.

Another question is - what about other hostings? Heroku is not the only possible hosting.
Some apps run without docker in old fashioned way, somewhere on private servers.
Where deploy procedure is significantly different from what heroku has.

Especially it's important when app is deployed by some script which does not export its ENV to the global ENV, like chef for example. Then right after deployment app wont work, because outside of the script ENV is empty, correct? What about console scripts?

@dzuelke

Yes it does. A build runs once, and is combined with configuration from the environment into a release. If the config changes, no new build is performed. That's factor number V: http://12factor.net/build-release-run

It explicitly says - combine your code with config once on release(deploy). And once it combined - symfony caches the result. Then runtime stage does not change ENV anymore.
If you want app to react on that change - consider to run your deploy procedure and rebuild your app cache. Docker entry point script might help.

The whole purpose of env vars for runtime configuration is that you do not need to re-build your code. Emergency/maintenance/readonly modes, split factors for A/B testing, database credentials on failover - these are things you want, and should be able, to change quickly.

It looks like you trying to use ENV for your business logic configuration, when it should be used only for app/infrastructure layer configuration. Have you thought about using tools like ZooKeeper instead? Or implement your own admin backend?

12 factor app:

Builds are initiated by the app’s developers whenever new code is deployed. Runtime execution, by contrast, can happen automatically in cases such as a server reboot, or a crashed process being restarted by the process manager. Therefore, the run stage should be kept to as few moving parts as possible, since problems that prevent an app from running can cause it to break in the middle of the night when no developers are on hand.

[magnusnordlander]

Symfony guide us to use %parameters% for this. Not expressions. And it offers us SYMFONY_ENV_VARS to assign this parameters and an extension point in form of php version of parameters file, where you can use getenv('CUSTOM_ENV_VAR') and assign it to a parameter.

Symfony is not an infallible deity that gives gifts and offers us advice. Symfony is a set of tools, a very good set of tools, useful in a lot of cases, but which in this particular case is lacking. There's nothing wrong with wanting to make the framework better by making it work well under more circumstances.

What you are describing does not work well when you pass configuration via environment variables, and those environment variables can change at runtime. A major hosting provider uses this scheme. A very popular container format is better off using this scheme. There is no reason why we shouldn't attempt to accomodate it.

Consider prod cache as safety net. Imagine we've disabled caching for env vars - now, if we rename env var or remove it from your heroku admin panel - should all the hosted apps die?

Yes. I personally do not feel the need for any such safety nets.

What about rollback? If env is changed how to automate this without a cache? Sure it's possible to do it without a cache, but solution is already there.

That is the responsibility of the hosting provider using such a scheme (and indeed on Heroku, rollback is already very much possible, you just press a button).

Another question is - what about other hostings? Heroku is not the only possible hosting.
Some apps run without docker in old fashioned way, somewhere on private servers.
Where deploy procedure is significantly different from what heroku has.

If you're not passing configuration through environment variables, this does not affect you. If you are passing environment variables through IncenteevParameterHandler or SYMFONY__ENV, and want your environment parameters to be cached, this does not affect you. This PR is about adding the POSSIBILITY to use dynamic environment variables, something that has been requested for years, not about forcing anyone to do it.

[atorian]

@magnusnordlander

This PR is about adding the POSSIBILITY to use dynamic environment variables, something that has been requested for years, not about forcing anyone to do it.

i'm not against new features, i'm for correct responsibilities and obvious ways of usage.

Once there was a framework, which had multiple ways to implement same thing. And developers were not able to use their framework knowledge to start with maintenance of existing apps fast, because every previous developer had his own way.

Allowing this kind of configuration - we automatically allow it to be used in any other config.

I expect once this feature is added - people will use it instead of SYMFONY_VARS, as it look simpler on development stage. And, because developers does not read documentation, but copy examples, until they stacked for a day with debug - it will lead to a lot of confusion, when those vars will disappear at some moment.

It will need dangerous mark in documentation. People will use them in different config files and it will become a hell, to find which bundle needs which env var. Because people use short cuts quite often.
And if this feature will be added - it will immediately appear in many bundles.

It's hard to find a good bundle now. And it will become even harder.

Symfony already has issues with developer experience. Should we add more?

So if configuration should change during runtime - maybe it would be better to use another configuration provider, and don't use ENV in this case? And probably, we have to improve sf2-3 docs and explain what should be in ENV, and what should not?

Anyway, with all being said - let others decide: @stof ?

[magnusnordlander]

So, personal opinions on the respective merits of dynamic environment variables aside, let me try to restate the issue and what this PR attempts to do to alleviate it.

Presuppositions

• Because of how DI extensions and config passes are allowed to alter semantic bundle configuration, a generic solution to allow dynamic environment variables is impossible without major BC breaks (see [DependencyInjection] added a way to reference dynamic environment variables in service definitions #16403)
• We do not want to break BC, thus a generic solution is out of the question.
• Because a generic solution is out of the question, the best we can do is to have every bundle that wants to support using dynamic environment variables opt in to it.
• I have written a DI expression language function to get the dynamic value of an environment variable. Using this function, a bundle can fairly easily allow the user to use dynamic environment variables.

The problem with just adding the function

Using the function in a DI Extension is fairly easy, and could look something like this:

        $def = $container->getDefinition('fervo_pubnub');
        $def->replaceArgument(0, new Expression('env("PUBNUB_USERNAME")');

That's easy from a bundle author's point of view. The trouble comes when you're actually writing the configuration tree. How do you want to allow your users to pass you either a regular configuration value (be it a constant or a DI parameter), or an environment variable? Do you invent a new syntax (sort of like some bundles have done when referencing a service ID in their configuration)? Should you be really explicit and have a structure with type and value (again, like some bundles do when you are allowed to reference a service)?

I believe that in order for this to be widely adopted, there needs to be a convention on how to accept different types of configuration values.

My proposal

I propose that we standardize on one syntax for this. To be honest, I don't really care which syntax we choose, as long at it works. A non-exhaustive list of suggestions include:

fervo_pubnub:
    username: "@=env('PUBNUB_USERNAME')" # Allow any DI expression
    password: "@=service('config_storage').get('pubnub.pass')" # This would also allow you to store config in e.g. a database
    publish_key: "=env('PUBNUB_PUBLISH_KEY')" # Alternate syntax for the above, less confusing perhaps?
    subscribe_key: some_key # With either of these syntaxes, passing constant strings or DI parameters still works without any change
    secret_key: # Alternate syntax using a struct instead
        type: envvar
        value: PUBNUB_SECRET_KEY

And for XML:

    <fervo-pubnub:config>
        <credentials 
            username="=env('PUBNUB_USERNAME')"
            password="%di_parameter_as_usual%"
            id="and of course const strings still work"
        /><!-- One option -->
        <secret_key type="expression">env('PUBNUB_SECRET_KEY')</secret_key><!-- Another option -->
        <notification_key>no change here</notification_key>
   </fervo-pubnub:config>

These are just examples. The PR does have a suggested syntax, but I'm by no means opposed to changing it to some other syntax.

I do hope this clears up the intent and purpose of this PR.

[magnusnordlander]

Actually, the env expression function is mostly syntactic sugar. If you allow full DI expressions, you could just as easily do service('my_env_getter_service').getEnv('FOO'), it's just that env('FOO') looks a lot nicer in the config.

[stof]

@nicolas-grekas reading the env directly is fine IMO. this is what I did in https://github.com/Incenteev/DynamicParametersBundle. No need to create a dedicated service.

Btw, the function reading environment variable in the expression language works fine. My bundle failed to solve this use case, because of the point solved here: allowing DI extensions to accept expressions directly instead of trying to hook in the existing usage of parameters to make them dynamic.

So a big 👍 for adding the ExpressionNode, and then a 👍 for adding a way to read the environment (and IMO, we should not add a public method in the container for that)

[stof]

wrong number of arguments. the distinction is between 2 and 3 here

[magnusnordlander]

Thanks, fixed.

[magnusnordlander]

@stof: The only way I see around having a public method on the container is either a (possibly namespaced and/or static, but still globally available) function, inlining a quite big chunk of code, or coupling the expression language function to a specially "blessed" service in the container.

Out of these options, I would consider the public method to be the lesser of evils, but maybe I'm missing some option?

[magnusnordlander]

Actually, I've changed my mind, I'm making it into a static method instead.

[nicolas-grekas]

Instead of the Environment class, my personal preference to avoid a public method would to rebind this closure.
\Closure::bind(..., null, Container::class)

[magnusnordlander]

Ah, cool, I did not know that was possible. I'll try it out, and update the PR tomorrow.

[magnusnordlander]

That seems to work nicely.

[nicolas-grekas]

LGTM, could you add some test cases please?

[magnusnordlander]

There, now at least the interesting parts are being tested :)

[nicolas-grekas]

no need for this one: EL is always here, thx to composer require-dev
BUT you need to fix the lowest required EL version in Config's composer.json (to fix travis)

[magnusnordlander]

Thanks, fixed.

[nicolas-grekas]

👍

[dzuelke]

Can everyone here please review PR #19681 which contains a backward compatible alternative to this PR?

[nicolas-grekas]

The difference with #19681 from a user pov is the matter of responsibility: who says "here, use this env var". In this PR, env vars are referenced by bundle authors: the user can't use env vars where the bundle author did not allow to do so. This is the limitation that made me search for another way and led to #19681, where the user decides where dynamic env vars are injected.

Yet, this does not disqualify this PR: having an ExpressionNode in configurations might still be useful!
We just need to validate there is a use case for adding it in core (and change the title of this PR; if we agree that the env var use case is best solved by #19681).

[micha149]

#19681 works great from my point of view,

[fabpot]

Closing in favor of #19681