[DI] Allow processing env vars

[nicolas-grekas]

Q	A
Branch?	3.4
Bug fix?	no
New feature?	yes
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	see description
License	MIT
Doc PR	-

This PR is an updated version of #20276 (it embeds #23899 for now.)

It superscedes/closes:

• [DI] Add support for secrets #23621 ping @dunglas
• Runtime container parameter not found event filter #23669 ping @marfillaster
• [DependencyInjection] [DX] Support for JSON string environment variables #23823 ping @Pierstoval
• add support for composite environment variables #17689 ping @greg0ire
• [DI] ENV parameters at runtime with PHP 7 strict types not working properly #20434 ping @sandrokeil
• Issue when using a SQLite database and the DATABASE_URL env var #23527 ping @javiereguiluz

#22151 is another story, so not fixed here.

The way it works is via %env(foo:BAR)% prefixes, where "foo" can be bound to any services you'd like.
By default, the following prefixes are supported:

• bool, int, float, string, base64
• const (for referencing PHP constants)
• json (supporting only json arrays for type predictability)
• file (eg for access to secrets stored in files.)
• resolve (for processing parameters inside env vars.)

New prefixes can be added by implementing the new EnvProviderInterface, and tagging with container.env_provider (see Rot13EnvProvider in tests.)

Prefixes can be combined to chain processing, eg.
%env(json:base64:file:FOO)% will be roughly equivalent to
json_decode(base64_decode(file_get_content(getenv('FOO')))).

[marfillaster]

👍 I totally approve this solution. The chained processing is just genius.

[ro0NL]

preferably the namespace parsing happens in Container::getEnv no? Basically you need to do the same check each time implementing it. What about getEnv($prefix, $name, $value, \Closure $getEnv = null)?

[nicolas-grekas]

namespace parsing already happens in Container::getEnv.
What you see here is forwarding, which is something not all providers are required to do.

[ro0NL]

never mind, thats simply $value = $getEnv($name) currently. Basically the trick is to not forget return $getEnv($name) for defaulting.

[ro0NL]

and i missed your comment in between.. but you're right the current implem is fine.

[yceruto]

Can we fix #23527 (comment) somehow?

[nicolas-grekas]

@yceruto here we are, with a new resolve handler to the rescue.

[mnapoli]

Is %env(json:base64:file:FOO)% really more understandable than json_decode(base64_decode(file_get_content(getenv('FOO'))))?
This is yet another custom syntax that Symfony users will need to learn, whereas plain old PHP would do the same with a few more characters. Maybe these kind of problems could be solved with #22407/#23834 and allowing to write some entries as PHP closures? This is a more complex solution though ("easy to say"), but the end result would be maybe much simpler for users?

[nicolas-grekas]

@mnapoli I think you're forgetting about how things work. Despite your claim in #22407, a PHP DSL would not help at all reading from the env. The reason is compilation of the container. The DSL is read once, at compile time. %env(FOO)% is the syntax that we already use to be able to declare that intention of compiling down to dynamic calls. The namespaced env vars only build on top of this to add more processing capabilities to env, nothing more. It's not only about syntactic sugar.

[mnapoli]

👍 sorry :)

[Pierstoval]

Does this mean that the service class existence is not resolved at compile time?

[nicolas-grekas]

Yes, as done commonly in other passes: when it's not required at compile time, we usually don't validate that kind of things.

[nicolas-grekas]

registered before optimization passes so that env values can be used at compile time (typically using $container->resolveEnvPlaceholders().) This makes env providers a bit less flexible (they can't easily be manipulated by DI extensions) but that's consistent with the provided feature to me (env vars should be light to fetch.)

[Pierstoval]

I cannot agree more on this PR 👍

[ro0NL]

The str_ireplace seems to trigger an unexpected Array to string conversion

$_ENV['REAL'] = '{"real":true}';
$c = new ContainerBuilder();
$c->setParameter('json_real', '%env(json:REAL)%');

$c->register('foo', 'stdClass')->setProperties(array(
    'json_real' => '%json_real%',
));

$c->compile(true);

---

Somehow real and default envs get mixed up, probably due reference generation in EnvBag::get()

$_ENV['REAL'] = 'runtime';
$c = new ContainerBuilder();
$c->setParameter('env(REAL)', 'compiletime');
$c->setParameter('real', '%env(REAL)%');

$c->register('teatime', 'stdClass')->setProperties(array(
    'real' => '%real%',
    'real_inline' => '%env(REAL)%',
));

$c->compile(true);

dump($c->get('teatime'));

{#136
  +"real": "runtime"
  +"real_inline": "compiletime"
}

This looks bad 🤔

---

What im worried about is the current default strategy. With this feature things may work unexpected due the exact lookup. setParameter('env(REAL)', ...) will not be used if combined with any prefix, you need to expose all variants :(

Also related; if in above example you leave out the default ("compiletime"); you'll get

The service "teatime" has a dependency on a non-existent parameter "env(real)"

Yet we did a real lookup already.. so what's the deal here?

---

Otherwise it seems to work just great :)

[nicolas-grekas]

@ro0NL thanks for the detailed report. Good news, both (the 3 in fact) your examples apply to 3.3 also, which means you found bugs. Nothing related to this PR specifically. Those happen only in compile(true) mode, which I'm going to fix in another PR on 3.3.

[kbond]

What is the reason for no bool prefix?

[nicolas-grekas]

@kbond no reason, fixed.
Now with a new const filter also.

[dunglas]

Awesome, can't wait to get this merged :)

[dunglas]

Why not /* private */ const ALLOWED_TYPES?

[nicolas-grekas]

because that wouldn't be really private, whereas a private static has no such downside

[dunglas]

Can be stored in a var to prevent an extra function call.

[dunglas]

What's the benefit of making this function static, as it is already private?

[nicolas-grekas]

none, but another reviewer asked for it, and there is neither no benefit not doing it static :)

[dunglas]

Shouldn't symfony/config become a hard dependency?

[nicolas-grekas]

a exception is now thrown if Config is missing

[fabpot]

I don't like the "provider" name here as implementations do not provide anything. They process env vars, they filter them (Twig's name). So, I would use something like EnvProcessorInterface and container.env_processor.

[fabpot]

To be even more precise, it's processors for env vars. So, could be EnvVarProcessorInterface.

[nicolas-grekas]

done

[fabpot]

Probably a smell that this class should be renamed, or at least some part of it extracted into a more generic class.

[nicolas-grekas]

not a blocker here, and no better idea :)

[fabpot]

Does support for a length really useful? What is the use case? Apart from the unit test that uses /dev/urandom?

[nicolas-grekas]

removed

[fabpot]

Should be EnvVarProcessorInterface (same below)

[fabpot]

should be renamed as well

[fabpot]

Still some provider usages (like variable names for instance).

[fabpot]

Thank you @nicolas-grekas.

[sroze]

Great job @nicolas-grekas 👏