Add new encrypted variable without ability to decrypt

[kevlened]

I'd like contributors to be able to encrypt new variables for a target environment without the ability to decrypt other variables.

For example, let's say we want to update the variable BIG_SECRET in production. Today, it seems the contributor would need the DOTENV_KEY_PRODUCTION key from .env.keys and the DOTENV_VAULT_PRODUCTION variable from .env.vault. Because all the secrets are encrypted together, it seems a contributor would be able to read all the variables when given access to the key. It follows that a contributor can't add only BIG_SECRET to the vault without access to everything in production.

Is my understanding correct?

[motdotla]

with the current implementation, yes, your understanding is correct.

this is something i haven't given much thought yet, but one thought is this feature could be made available via a hosted service that holds your keys safely. then a teammate could be granted permission to write without seeing the key (the system/program would see it only)

but can you elaborate more on your use case? you mention contributors - are you using it with an open source or closed source repo? on GitHub or somewhere else? are these contributors approved by you or they could be anyone who generates a PR with a patch or something?

[kevlened]

Closed source repo on github, deployed to azure. Updates are all approved.

The concern is primarily around contractors who may have temporary access to the repo. If they introduce a new secret, it’s easy to rotate one when they finish, but annoying to rotate all of them.

[kevlened]

There’s also the issue of trusting the engineer’s machine itself. Granted, that’s a larger problem, but using a cloud KMS reduces the affected surface area.

[motdotla]

give dotenvx hub a try. exploring some of this there. it's very much in beta but currently it can:

1. securely hold your .env.keys
2. securely keep your .env.keys in sync with any members of your GitHub organization
3. display decrypted .env.vault values per Pull Request (make a change to .env.vault and it provides a UI to view the decrypted changes - side by side with your code changes)

i suppose it is a few steps away from:

1. permitting who on your team gets access to which keys (dev, prod, etc) in .env.keys
2. permitting who can push/pull/sync and who cannot
3. permitting a UI to append a single new production environment variable and that generates a PR with the .env.vault updated (your contractor would never know the contents or the key. just a UI to generate .env.vault changes.)

$ dotenvx hub login

[kevlened]

Seems like an interesting project, but unfortunately this is a sensitive application with rigorous vendor requirements, so another vendor is out of scope at the moment.

Basically, if someone has access to the repo, I don’t mind them having access to a credential that lets you encrypt using a KMS, because that’s functionally equivalent to a public key. Therefore, it would be safe to embed this limited credential to Azure in the repo.

[motdotla]

all makes sense, but one further question:

dotenvx would have to hold/manage this credential (so it could proxy to the true encryption key) so wouldn't you still be at the mercy of the same vendor requirements? vetting dotenvx as a vendor?

[kevlened]

Do you mean auditing this library?

For the purposes of SOC 2 and other credentials, the vendor of third party code isn’t audited as closely as infrastructure, even though the code is risky. For third party code, the risk is mitigated by vulnerability scanners; while for SaaS, the risk is mitigated by one of these certs (which encompasses pen tests, certified vendors, etc).

That’s why vetting a new library is easy, but vetting a new SaaS is more difficult.

Did I understand your question correctly?

[motdotla]

i'm saying that it's not possible to do this without it being a SaaS. The SaaS (whether a KMS or the current Hub approach) is still a SaaS holding your .env.keys and obfuscating the encryption key to part of your team in some manner.

Correct? or are you envisioning some other approach where maybe you store the .env.keys in Microsoft's or AWS's KMS? And dotenvx has a mechanism to fetch them just-in-time, hold them in memory, do the encryption set function on your contractor's machine, and then flush the memory?

[kevlened]

You're absolutely right; it's not possible to handle this key management securely without a SaaS.

I was imagining using Azure's KMS (because it's already a vetted vendor with HSMs) to generate a single public/private key per environment. The public key for each would be embedded in the repo and used to encrypt any variables for a desired environment. To decrypt the variables in a given environment, I'd assign a role to the vm with the power to decrypt using the private key assigned to the environment.

In this scenario, the plain text .env files never exist, except for local dev environments or non-secret variables.

I guess I was imagining a simplified version of sops that worked on a per-key basis, and doesn't require a roundtrip to the KMS when adding or updating a variable.

[motdotla]

this is edifying.

the current encryption algorithm (AES-256) is symmetric, but an asymmetric approach like you are mentioning could be very powerful and solve your request.

the public key like you mentioned could just live in the .env.vault file DOTENV_PUBLIC_KEY_DEVELOPMENT, DOTENV_PUBLIC_KEY_PRODUCTION, etc

it does complicate the formatting a bit - since 'encrypting' would actually be encrypting a KEY=value string and then appending that ciphertext, but that could be solvable enough with the library to compile all the appends together. dotenvx run would just have to add some smarts to incrementally decrypt each append.

but.. what i'm not sure how to solve is being able to set/override current keys without things getting kind of messy. it would technically work but would result in a decrypted .env file looking like this:

# .env
HELLO="World" # initially added
SECRET="other" # initially added
HELLO="Universe" # added on 2024-04-30
HELLO="Mom" # added on 2024-05-03
HELLO="Dad" # added on 2024-05-04

^ in this case the value is 'Dad' since the last value in a single .env 'wins'

i suppose the tool could then provide a cleanup method usable only by the individual holding the private encrypt/decrypt key, but that's starting to feel a bit unwieldy.

any thoughts around all this?

[kevlened]

Interesting. It seems like a ledger of changes could get unruly pretty fast.

The problem may be easier to solve if we assume a variable's value is secret, but a variable name is not. This is likely true for most use-cases, because if you have access to the vault file, you have access to the codebase, which references the variables by name.

Maybe a VAR_NAME=ciphertext scheme could work? Seems like you'd then need a way to distinguish which vars go to which environment.

It's also ok for this to be out-of-scope for dotenvx. My workaround for the moment is for contractors to just ask when they need new environment variables. If I'm the only one updating these variables, they never get access to the keys. However, that only solves half the problem (contractors accessing variables), and doesn't solve the other half (dev machines are compromised).

[motdotla]

seems like a ledger of changes could get unruly pretty fast

arguably yes, but thinking about it again, that also offers a helpful feature benefit - change history.

Maybe a VAR_NAME=ciphertext scheme could work

yes possibly but also possibly just shoving/appending it into the DOTENV_VAULT_DEVELOPMENT variable with some separator between appends.

then additionally modify the .env.keys to state the mechanism they are using - algorithm/asymmetric, etc. the keys are in uri format so can take on more metadata/instructions.

possibly this is something for the future, but going to table it for now given other priorities. thank you for the chat on this!

[kevlened]

a helpful feature benefit - change history

Storing the vault in git seems to satisfy this req, regardless of the strategy. A caveat is that encrypting VAR_NAME would obscure what was changed.

shoving/appending it into the DOTENV_VAULT_DEVELOPMENT

Ah, I understand what you were thinking now. This seems really useful as it doesn't require substantial changes to the existing format.

thank you for the chat on this!

Likewise! I'll leave this open, but feel free to close.

[nbercurrantsourcream]

+1 to asymmetric setup

Reading here about the dotenvx hub sounds great, but we would not be able to use it as our VCS is bound to other provider e.g. ( bitbucket/gitlab/selfhosted )

Having the ability to go with asymmetric encryption/decryption without hub would be amazing.

For cases like ours, where we would love any dev to append/change a key (plaintext) with encrypted value to the .env.production but only the target machine would have the other key to decrypt.

If we could have .vault per .env* file then

Maybe a VAR_NAME=ciphertext scheme could work

This would make it to see "a change" in PR without the need to decrypt to check which key was modified -> if I understood the append solution right.

Since you'd see the ciphertext modified and name of var plain text, it could also allow for the last added value for the key to be the one persisted avoiding the append log / need for cleanup

[motdotla]

Since you'd see the ciphertext modified and name of var plain text, it could also allow for the last added value for the key to be the one persisted avoiding the append log / need for cleanup

this is a good point since process could look like this:

# .env.production.vault
HELLO="ciphertextdfjkdfjkdfjdfj" # created by jack
HELLO="ciphertexteruieruieruier" # updated by jill

then just manually remove the prior one - like you would any key from a .env file.

# .env.production.vault
HELLO="ciphertexteruieruieruier" # updated by jill

[motdotla]

the above would be a departure from the current .env.vault implementation. some potential benefits and some potential downsides. would be great if others would weigh in:

(for the sake of discussion let's call it a .env.environment.crypt file)

benefits

• could effectively allow for mix of encrypted/decrypted values in the same value (dotenvx could recognize a prepend mechanism like HELLO="crypt:ciphertext")
• could effectively allow for mixing different encryption algos in the same .env.crypt file (HELLO="crypt:aes256:ciphertext")
• effectively permits anyone to contribute encrypted values without privileges to read them
• avoids need for a .env.example file because you could now effectively share your .env.crypt file with a mix of encrypted and raw values.

downsides

• maybe getting too mixed with complexity
• keys not encrypted - so if using a private repo and your .env.crypt file leaks an attacker knows where to attempt attacks
• arguably more could go wrong in production/devops because now juggling multiple keys and multiple .env.crypt files (the simplicity of a single .env.vault file and a set a single decryption key is appealing currently)

others thoughts?

example:

# .env.production.crypt
DOTENV_PUBLIC_KEY=“publicencryptkey”

HELLO=“World”
STRIPE_API_KEY=“encrypted(ciphertextgoesheredjfkdfjdkfjdkf)”
TWILIO_API_KEY=“encrypted(ciphertextgoeshereeruiqeruieqr)”
DASHBOARD_NAME=“Dashboard”

$ dotenvx set STRIPE_API_KEY sk_live_1234 --encrypt --env-file=.env.production

dotenvx set would:

1. check for DOTENV_PUBLIC_KEY in .env.production
2. use it to encrypt the sk_live_1234 payload
3. set that at STRIPE_API_KEY with encrypted(ciphertext)

dotenvx run would:

1. parse .env.* like usual
2. before inserting each KEY/value pair to process.env, it would check if the value started with encrypted{
3. if so, it would read ciphertext, decrypt it using the DOTENV_PRIVATE_KEY set on the server, and inject the decrypted value

[motdotla]

this approach would remove the need for paranoid mode - #178 - since everything would still just be contained in the .env file (if allowing for mixed raw and encrypted values)

[motdotla]

after some research i've chosen an encryption algorithm. secp256k1 curve. now working on a proof of concept implementation.

[motdotla]

work happening here: #197

it shows real promise to simplify things for the same benefits, plus the extra benefit of allowing non-trusted parties to contribute in a trustless manner to an open source repo's .env file.

the tooling is also arguably easier than having to manage a .env.vault file and grasp a new concept like that.

[nbercurrantsourcream]

One more benefit I can see is it will make it easier to deal with git merge conflicts since change will be limited to the key touched ( should be easier for git to pick up ) rather than the long vault string per env.

[motdotla]

on the left a .env file. on the right a public-key encrypted .env file.

[nbercurrantsourcream]

Exactly wht I was hoping for, this will make things so much easier 💯

[motdotla]

i'm working on run next. so that it works in a forgiving way:

• DOTENV_PRIVATE_KEY='<key>' dotenvx run -- node index.js
• dotenvx run -- node index.js (if .env.keys file present automatically reads DOTENV_PRIVATE_KEY from it)
• dotenvx run -f .env.production (automatically reads DOTENV_PRIVATE_KEY_PRODUCTION from .env.keys)
• DOTENV_PRIVATE_KEY_PRODUCTION='<key>' dotenvx run -- nodex index.js (notices key is production and loads from .env.production

but also might adjust the last option to instead use DOTENV_PRIVATE_KEY and instead embed the environment/filename in the key - like the .env.vault mechanism does. that has worked well.

[motdotla]

@kevlened what are your thoughts on this so far?

[kevlened]

This looks great!

i've chosen an encryption algorithm. secp256k1 curve

Great choice, as it's supported by the major KMSs, should you decide to add native cloud KMS support.

it will make it easier to deal with git merge conflicts since change will be limited to the key touched

A nice plus.

i'm working on run next

Fantastic!

• DOTENV_PRIVATE_KEY='<key>' dotenvx run -- node index.js
• ...
• ...
• DOTENV_PRIVATE_KEY_PRODUCTION='<key>' dotenvx run -- nodex index.js

Seems like your first and fourth usage examples are the same, but the first has less information.

use DOTENV_PRIVATE_KEY and instead embed the environment/filename in the key

It seems like this would be necessary to have enough information for your first usage example. However, one thing that's nice about DOTENV_PRIVATE_KEY_PRODUCTION is it's UI-friendly. At a glance, you'd know which keys are being used:

That said, if the env is encoded in the key AND the variable, it's not clear which would win. Perhaps it just throws an error to prevent accidents?

[motdotla]

UI-friendly. At a glance, you'd know which keys are being used

this is a good point. going to kick this around some more.

what UI is this from? VS Code?

[kevlened]

That particular UI is in Azure

[motdotla]

this is a good point. going to kick this around some more.

i gave this more thought and I think the cleanliness and most common happy path is to embed the environment/file-extension in the environment variable key so that a user can do this on their production servers:

scenario 1: .env.production

$ export DOTENV_PRIVATE_KEY_PRODUCTION="<key>"
$ dotenvx run -- node index.js
# this will smartly run the .env.production file (`_PRODUCTION` => `.env.production`)

scenario 2: .env.ci

$ export DOTENV_PRIVATE_KEY_CI="<key>"
$ dotenvx run -- node index.js
# this will smartly run the .env.ci file (`_CI` => `.env.ci`)

scenario 3: .env remote server

$ export DOTENV_PRIVATE_KEY="<key>"
$ dotenvx run -- node index.js
# this will smartly run the .env file (`` => `.env`)

i like this for the point you made @kevlened and because this is a 1 to 1 match with what you see in the .env.keys file:

# .env.keys file

# .env.production
DOTENV_PRIVATE_KEY_PRODUCTION="03e966744d2f4fcb1471f5dd2ecc38b0786cafb3ebe0e35e568e23de1eb78126"

# .env
DOTENV_PRIVATE_KEY="b7fe6cf1e5b064e0f3d0aa03f2bb601c15a6585c7cd9534595c87fe7ca4f7594"

i additionally like this because it unlocks support for multiple encrypted .env files in production (something the .env.vault mechanism cannot support)

scenario 4: multiple encrypted/mixed .env files in production

$ export DOTENV_PRIVATE_KEY="<key>"
$ export DOTENV_PRIVATE_KEY_PRODUCTION="<key>"
$ dotenvx run -f .env.production -f .env -- node index.js

furthermore, this will also support rotating private keys without downtime for any specific .env file:

$ export DOTENV_PRIVATE_KEY="<key>,<newKey>"
$ export DOTENV_PRIVATE_KEY_PRODUCTION="<key>"
$ dotenvx run -f .env.production -f .env -- node index.js

the only thing missing here is future options. those won't be able to be encoded in the key as a result, but i think adding additional environment variables will solve that elegantly for custom future cases. for example, we could support a different encryption algo with:

$ export DOTENV_PRIVATE_KEY_PRODUCTION="<key>"
$ export DOTENV_ENCRYPTION_PRODUCTION="x25519"
$ dotenvx run -- node index.js
# this will smartly run the .env.production file (`_PRODUCTION` => `.env.production`)

and one last note, i think these conceptually align well with the flags used in development which should make for less cognitive friction for developers.

for example:

$ dotenvx run -f .env.production  --overload -- node index.js

can be run instead using a DOTENV_PRIVATE_KEY_

$ DOTENV_PRIVATE_KEY_PRODUCTION="<key>" dotenvx run -- node index.js

they are functionally equivalent.

[motdotla]

these features are now in main and will be released as part of v0.38.0

the following commands will be deprecated and additionally moved under vault parent command:

• encrypt (deprecation notice and => vault encrypt)
• decrypt (deprecation notice and => vault decrypt)
• status(deprecation notice and => vault status)

when v1.0.0 is released, they will be removed to make room for additional tooling for this improved inline encryption approach.

thank you guys for all your thoughts on this. i think i'll look back on this thread someday as when dotenv's next evolution really found its footing.

[kevlened]

Great! Thanks for looking at this seriously; I'm looking forward to the release.

[motdotla]

placing screenshot here to be used in the README