New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC - envProcessor to decrypt secrets for simple applications #27351

Open
jderusse opened this Issue May 23, 2018 · 13 comments

Comments

Projects
None yet
8 participants
@jderusse
Contributor

jderusse commented May 23, 2018

Description
For simple architecture (without Vault or Amazon KMS, ...) the list of secrets have to be written down somewhere.
Could be file/env variables on the host server or in config files of deployment tool like ansible/puppet/docker/kubernetes...
The application end up with 2 repository to synchronize. In the worse case, the secrets are pushed along side the code (and sadly, many time in the parameters.yml itself)

The purpose of this RFC is to expose a simple way for developers to safely commits secrets and simplifying deployments.

Based on asymmetric encryption, a public key could be committed in the project which allow developers to generate new encrypted secrets.
Deployments, "just" have to deploy the private key

Nothing to synchronize anymore, adding/updating a secret does not requires to register this value of that secret in a dedicated tool.

Example

$ ./bin/console secret:encrypt "foo bar"
MIIBoQYJKoZIhvcNAQcDoIIBkjCCAY4CAQAxg...
# services.yaml
parameters:
  default_jwk: '%env(json:APP_JWK)%' # This variable contains the private key used to decrypt secrets
  # default_jwk: '%env(json:file:APP_JWK_FILE)%' # with APP_JWK_FILE=/opt/application/.jwk.json

# config/packages/prod/doctrine.yaml
parameters:
  env(DATABASE_PASSWORD): 'MIIBoQYJKoZIhvcNAQcDoIIBkjCCAY4CAQAxg...'
  database_password: '%env(decrypt:default_jwk:DATABASE_PASSWORD)%'
@stof

This comment has been minimized.

Member

stof commented May 23, 2018

the main reason for the %env()% syntax is that these special parameters are resolved at runtime without needing a cache clearing (allowing to change them at runtime easily). If all the settings are committed in the repo anyway (requiring a deployment to change them), I'm not sure the env processor is the right way to handle this. use case.

@nicolas-grekas

This comment has been minimized.

Member

nicolas-grekas commented May 23, 2018

Following @stof's commend, does this have to be an env processor?
Can't we make a command that would be used when setting up the prod env, and that would decrypt+export the secrets (in a file) so that they could be consumed as real env vars using the "file:" processor?

@jderusse

This comment has been minimized.

Contributor

jderusse commented May 23, 2018

In fact, half of the secret is committed, to be revealed, the private key is required. This key should exists only at runtime.

The encrypted secret could also be available only at runtime (if it's committed in a Dockerfile for instance)

Using env preprocessor is also simpler for developers as documentation and recipes used that syntax.

@jderusse

This comment has been minimized.

Contributor

jderusse commented May 23, 2018

Thinking about the @nicolas-grekas comment.

When a developer want to add a new secrets:

$ ./bin/console secret:encrypt-string "my secret" --name bar 
$ ./bin/console secret:encrypt-file /path/to/file --name foo --bucket prod1

This command generate a new file (uses https://tools.ietf.org/html/rfc7516)

$ cat ./app/config/secrets/prod1/foo.jwe
{
    "header": "eyJhbGciOiAiUlNBLU9BRVAiLCAiZW5jIjogIkExMjhDQkMtSFMyNTYifQ",
    "cek": "SsLgP2bNKYDYGzHvLYY7rsVEBHSms6_jW...",
    "iv": "Awelp3ryBVpdFhRckQ-KKw",
    "ciphertext": "1MyZ-3nky1EFO4UgTB-9C2EHpYh1Z-ij0RbiuuMez70nIH7uqL9hlhskutO0oPjqdpmNc9glSmO9pheMH2DVag",
    "tag": "Xccck85XZMvG-fAJ6oDnAw"
}

In order to use it in production, developer have to decrypt secrets first

$ ./bin/console secret:decrypt --key /path/to/key --bucket prod1

This command dump the revealed secrets

$ cat ./var/cache/secrets.json
{
    "foo": "raw content of original file"
}

Application can use it

parameters:
  %env(APP_SECRETS)%: %kernel.cache_dir%/secrets.json
  bar: %env(key:foo:json:file:APP_SECRETS)%

How to configure several environment? for instance (preprod vs prod or prod-eu vs prod-us) ?
=> the encrypt and decrypt command expose a "--bucket" parameter (default value is "default").
Encrypted file will be stored in the "name of bucket" folder.

How to not break dev environment?
=> developpers could configure a dedicated "dist" bucket without encryption key which would generate plain text secrets files

Going further with semantic configuration

  secrets:
    default:
      key:
        public: %kernel.config_dir%/secrets/keys/{bucket_name}.jwk.json
        private: /opt/application/secret-key.jpsk.json
      encrypted_folder: %kernel.config_dir%/secrets/buckets/{bucket_name}/
    buckets:
      default:
        key: false
        encrypted_folder: %kernel.config_dir%/secrets/buckets/dist/
      prod1:
        key:
            public: %kernel.config_dir%/secrets/keys/prod1
            private: /opt/application/secret-key
        encrypted_folder: %kernel.config_dir%/secrets/buckets/prod1/
      prod2: ~

Comming to the conclusion that it could be done in a dedicated bundle...

@javiereguiluz

This comment has been minimized.

Member

javiereguiluz commented May 24, 2018

I'm sharing some resources about how other frameworks have solved this very same problem, so we can take ideas from them:

@javiereguiluz javiereguiluz added this to the next milestone May 24, 2018

@tvlooy

This comment has been minimized.

Contributor

tvlooy commented May 24, 2018

@nicolas-grekas encrypt+export to a file seems bad because then you have a file with plain text credentials on your filesystem again. If someone can read the file you lose your credentials. It's better to keep the encrypted values on your filesystem at most and only use decrypted values in the runtime. I think credentials should not be in the environment because if someone leaves a phpinfo, there will be plaintext credentials and/or decryption key in there. So why use the environment anyway and not just the runtime?

@nicolas-grekas

This comment has been minimized.

Member

nicolas-grekas commented May 25, 2018

If someone can read the file you lose your credentials

if someone can read your files, they can get the private key and decrypt the data on their own, so that this makes no difference, isn't it?

the extra file makes it a bit more difficult to deal with secrets, as it requires one extra step to work. Looking at other frameworks, it looks like they decode the secrets at runtime, so maybe this is not an issue after all and we should get rid of the file for better DX?

@jderusse I'm not sure about the "bucket" thing. Can't we deal with this using just standard parameters to vary the path of the stored secrets by region or whatever? You must have a parameter already that gives e.g. the region, isn't it? One less concept to grasp would be better :)

@tvlooy

This comment has been minimized.

Contributor

tvlooy commented May 25, 2018

+1 for no extra files. Decode secrets at runtime seems like what we need to do

About the decryption key. How do you think we should protect the key? From CLI, the file itself should be available obviously. But, from FPM I would take the same approach like how webservers do it with SSL keys. It would make sense if the file path is defined in FPM with an ini setting, and only accessible by the parent process. The parent should load the key and make the key contents available in the runtime (not the environment vars) but the file itself is not available from FPM.

If someone has file access from FPM, he can download your encrypted secrets but not your decryption key.

I am in the process of making this a solution so your opinion has great value to me.

@jderusse

This comment has been minimized.

Contributor

jderusse commented Jun 27, 2018

The more I think about it, the more I love the simplicity of

# service.yaml
framework:
  secrets:
    foo:
      type: RSA
      key:
        public: %kernel.config_dir%/secrets/keys/public.jwk.json
        private: /opt/application/secret-key.jpsk.json #could be defined in an env variable to
      encrypted_folder: %kernel.config_dir%/secrets/
    bar:
      type: vault
      endpoint: %env(VAULT_DSN)%
      token: %env(VAULT_TOKEN)%

third-party-bundle:
 aws-secret:
    prefix: /prod/baz
  
parameters:
  %env(SECRET_KEY)%: foo
  bar: %env(secret:foo:SECRET_KEY)%

This add an opportunity to let third party registering their own secret mechanism...

@NAYZO

This comment has been minimized.

Contributor

NAYZO commented Jun 28, 2018

@jderusse Good idea! but it could be better without env vars.

Actually, I still don't get how do you find deploying using environment variables is straight forward... Since in the best case scenario when adding/deleting or editing an env variable you have to update the /etc/environment or .bashrc file or any equivalent + sourcing the file to pick up the changes AND editing the webserver config + reload/restart !!
Which to me is way overhead then a simple c:c command!

Or I'm missing something here ?

@keichinger

This comment has been minimized.

Contributor

keichinger commented Jun 29, 2018

@NAYZO for our 4.x projects we're copying the environment variables all over the place, which is madness compared to the old parameters.yaml approach:

  • We pass down environment variables to Symfony via Nginx's fastcgi_param APP_ENV "prod";
  • We explicitly load a revived config/parameters.yaml for console commands
  • We explicitly declare environment variables in our own custom deploy.sh script where you literally do a export APP_ENV="prod", which is only exported and and available during the script execution

We basically never touch /etc/environment or any of the ~/.bashrc, ~/.bash_profile scripts, unless it's something we configure globally (like private NPM or private Packagist related things) that affects all projects that run on the server.

I can't remember what exactly about .env wasn't working for us, but it was painful when we were evaluating the 4.0 betas, which is why we're still doing the above madness.

@NAYZO

This comment has been minimized.

Contributor

NAYZO commented Jun 29, 2018

@keichinger if I get this correctly you have multiple projects on the same server! That is odd to me since you can get some env vars collision between projects like database credentials or APP_ENV (prod/test/dev) for instance. etc..
IMO, to manage multiple projects on the same server with env vars you have to use Docker or similar approach so that you can be sure no env vars collision could take place.
And this is an other downfall of the env vars.

@keichinger

This comment has been minimized.

Contributor

keichinger commented Jun 29, 2018

That is odd to me since you can get some env vars collision between projects like database credentials or APP_ENV (prod/test/dev) for instance. etc..

Each project is running in its own PHP-FPM pool (e.g. projectA runs on fpm-port 9000 while projectB runs on 9001), so we have a perfect isolation between projects since Nginx and PHP-FPM won't leak them in any way.

So far we've had a good run with this approach. We're kinda eyeing to upgrade our infrastructure to a Kubernetes cluster but for that we'd need some additional research and prototyping. Since we're an agency it needs to work for all of our projects in a seemless and reliable way while making sure our hosting prices won't go nuts, otherwise customers will want to host it elsewhere, which makes it harder to unify the deployment processes across hundreds of projects :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment