Storing complex secrets

[jberglinds]

Solution: #749 (comment)

Hello,
I can't figure out how to store my Google API private key for use in a now deployment.
The key looks something like:

-----BEGIN PRIVATE KEY-----
lines
lines
lines
-----END PRIVATE KEY-----

What I've tried:

• now secret add google-private-key "<full key>"
  • Not possible because of dashes, Secret values with leading dashes are parsed as the incorrect number of args #80
• now secret add google-private-key -b $(echo "<full key>" | base64)
  • Only gives me "[object Object]" in js, see Secret values with leading dashes are parsed as the incorrect number of args #80
• Adding the key to a .env-file and then run with now --dotenv=.env
  • This seems to replace all newlines with whitespace making the auth with the key fail.
  • When running locally using the dotenv node package to set env vars from .env it works, it preserves newlines

How should it be done?

[sholladay]

I think your second example is almost correct, just omit the -b. I also used cat instead of echo below to ensure the newline problem isn't related to your terminal.

now secret add google-private-key "$(cat my-keyfile | base64)"

Unfortunately, whatever the intention behind the -b flag was, it seems to be quite broken.

https://github.com/zeit/now-cli/blob/f8c0288bb3e3f05b38ab6a75ba78d5dcc0e57c7a/src/providers/sh/commands/secrets.js#L236-L240

[malixsys]

Use @sholladay's method to put the secret in PROD:

now secret add google-private-key "$(cat my-keyfile | base64)"

then in now.json put:

{
  "env": {
    "google_private_key": "@google-private-key"
  }
}

then in your code:

const atob = require('atob');
const privateKey = atob(process.env.google_private_key);
// use privateKey

Locally, you can use the following to load your secrets:
In .env:

google-private-key=<btoa() of your key>

then use this script in package.json:

  "scripts": {
    "dev": "nodemon --exec \"node -r dotenv/config\" server.js",
  }

IMPORTANT!

Make sure to add .env to .gitignore and .npmignore!!

[Qix-]

Definitely - action item for this issue is to make sure we're elegantly handling different types of secrets. I'm not sure environment variables are a silver bullet here.

This definitely deserves some investigation.

[hakusaro]

Anything on this issue lately? Fixing this is kind of vital for apps like GitHub’s own Probot to be deployed successfully with Now.

[malixsys]

@hakusaro Did you try the workaround above? (e.g. for google-private-key)

[hakusaro]

I feel like I've been endlessly repeating this. I don't have this issue personally anymore, but it violates the principle of least surprise that the following works:

heroku config:set SOME_PRIVATE_KEY=[some multiline blob here]

but the following, which is stupidly close in syntax, does not:

now -e SOME_PRIVATE_KEY=[some multiline blob here]

Similarly, now doesn't echo back environment variables it sets, whereas heroku does. This means that a user has no idea that setting a private key failed. It just silently blows up.

[malixsys]

Sorry did not see it before... I agree with you on "principle of least surprise", but it is NOT a blocker, is it?

Fixing this is kind of vital for apps like GitHub’s own Probot to be deployed successfully with Now.

[hakusaro]

It's not a blocker by the strictest definition, but the probot maintainers (of which I am regrettably not) currently provide three solutions for deploying probot to the internet. They go to admirably extreme lengths to make the deployment process as simple as possible on the platforms they support.

Having this issue introduces enough friction that they would rather exclude the entire platform. I think the fear is that this response:

I'm not sure environment variables are a silver bullet here.
This definitely deserves some investigation.

and no movement on patching this gives an indication that now isn't frictionless enough to warrant further inclusion.

---

Speaking from my own experience, I knew already that probot was a GitHub product, so I took the suggestion to use now as an implicit recommendation that it was stable enough for GitHub to suggest using it. GitHub's reputation was carried over to it a little in my mind. I think the reason for the removal suggestion comes partially from this. It isn't a good user experience for someone to try to deploy a GitHub product on a platform using a workaround for something that should work.

The problem is that I have a soft spot for now in that I found it better than the other two options in a lot of areas. I'm sticking my head out because I don't want to see a stupid issue quash the potential joy that people get from using now. It should be a trivial enough fix to get this deployment to work, and I think that any ideological loss that happens from going with what heroku does is negligible compared to potential user loss due to this kind of issue.

Notice that I'm not even caring about Google API private keys. For every person who reports an issue or comments on an issue about a bug, there are ten others who sigh, drop the platform, and move on to the next. There's no way of telling how many people don't use now as a result of this than if it just worked.

[malixsys]

Indeed! Hope the path gets smoother... If only I had time to create a PR...

CC @rauchg ?

[TooTallNate]

As of #1033, the --base64 argument has been removed. As mentioned in the original post, it has been broken for some time (returns the string "[object Object]"). Aside from that, it's never been clear how exactly to get the binary data back out from the env var.

The recommended way to store complex and/or binary data with secrets is with the base64 CLI, but without --base64:

now secret add google-private-key "$(cat key.txt | base64)"

Then in your Node.js server code you can use Buffer to turn it back into a binary blob:

const key = new Buffer(process.env.GOOGLE_PRIVATE_KEY, 'base64');

Or if it's a Docker deployment then do something like:

echo "$GOOGLE_PRIVATE_KEY" | base64 -d > key.txt

I hope that helps!

[leo]

Closing this after @TooTallNate's answer – please do it like he described 😊

[Maxhodges]

great so everyone has to waste hours trying things that don't work as expected and consulting expensive consultants before hopefully discovering this issue and proposed workaround.

[VincentTam]

Thanks for raising this issue. I'm having the same problem.

Then in your Node.js server code you can use Buffer to turn it back into a binary blob:

const key = new Buffer(process.env.GOOGLE_PRIVATE_KEY, 'base64');

@TooTallNate new Buffer(...) has deprecated.

[mckernanin]

@VincentTam a node deprecation isn't an issue with now, and in the page you linked there's a guide on how to do this in a non-deprecated way, which looks like this:

const key = Buffer.from(process.env.GOOGLE_PRIVATE_KEY, 'base64');

[hleumas]

There is actually much easier way of storing complex secrets in now.

Just do this:

now secret add google-private-key -- "`< my-key-file`"

The -- is a bash convention to indicate the end of command options so this resolves the problem with key starting with ----.

I've tested this and it works for me. This also resolves #80.

[grikomsn]

Here's my take on storing Firebase private key on Vercel secrets:

---

Take 1: store long secrets with unescaped newlines (does not work) ❌

As like everyone else, tried storing the private key as-is and resulted with this error log which I assume is because the unescaped newlines.

View command line log

vc secrets add firebase-private-key "-----BEGIN PRIVATE KEY-----\nXXXXXXXXXXXXXXXXXXXXXXXX\n-----END PRIVATE KEY-----"
internal/validators.js:120
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Array
    at validateString (internal/validators.js:120:11)
    at Object.resolve (path.js:980:7)
    at Object.getGlobalPathConfig [as default] (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:221212:43)
    at Object.79919 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:221070:41)
    at __webpack_require__ (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:231483:43)
    at Object.13289 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:229150:34)
    at __webpack_require__ (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:231483:43)
    at Object.86137 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:229423:19)
    at __webpack_require__ (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:231483:43)
    at Module.69739 (/home/griko/.config/yarn/global/node_modules/vercel/dist/index.js:208259:13) {
  code: 'ERR_INVALID_ARG_TYPE'
}

---

Take 2: store the whole json as the secret (does work, but exceeds size limit) ❌

Tried storing the actual JSON content of the Firebase credentials to Vercel secrets by running something like:

vc secrets add firebase-creds-json "$(< ./creds.json)"

Then do a JSON.parse(process.env.FIREBASE_CREDS_JSON) when retrieving the env var. Technically it works, but when deploying you'll get the error of exceeding maximum env var size limit (similar to previous comment).

---

Take 3: store long secret as json-compliant secret (does work) 🎉🎉🎉

Since take 2 works but it exceeds the secret size limit, now I tried only storing the private key but also as a JSON-compliant value (either a basic quoted string or an object):

# as a single string value, note the extra quote
vc secrets add firebase-private-key '"-----BEGIN PRIVATE KEY-----\nXXXXXXXXXXXXXXXXXXXXXXXX\n-----END PRIVATE KEY-----"'

# as a json object
vc secrets add firebase-private-key '{"privateKey":"-----BEGIN PRIVATE KEY-----\nXXXXXXXXXXXXXXXXXXXXXXXX\n-----END PRIVATE KEY-----"}'

Then same as take 2, retrieve the value using JSON.parse(process.env.FIREBASE_PRIVATE_KEY) if it's a single string or const { privateKey } = JSON.parse(process.env.FIREBASE_CREDS_JSON) if it's a JSON object. Make sure you use double quotes in your .env file on local development:

# firebase env vars
FIREBASE_PRIVATE_KEY_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
FIREBASE_PRIVATE_KEY='"-----BEGIN PRIVATE KEY-----\nXXX\n-----END PRIVATE KEY-----"' # make sure to double wrap with quotes

---

Hope this helps! 🎉

[saitodisse]

Thanks @grikomsn, I finally could configure the GOOGLE_APPLICATION_CREDENTIAL without using the json file way.

@grikomsn 's Take 3 and this blog post

GOOGLE_APPLICATION_CREDENTIAL without JSON FILE

---

getAuthToken.ts

import { google } from 'googleapis'

export const SCOPES = ['https://www.googleapis.com/auth/spreadsheets']

// pega credentials das variáveis de ambiente
export async function getAuthToken() {
  if (typeof window !== 'undefined') {
    throw new Error('NO SECRETS ON CLIENT!')
  }

  const { privateKey } = JSON.parse(process.env.GOOGLE_PRIVATE_KEY || '{ privateKey: null }')
  const auth = new google.auth.GoogleAuth({
    scopes: SCOPES,
    projectId: process.env.GOOGLE_PROJECTID,
    credentials: {
      private_key: privateKey,
      client_email: process.env.GOOGLE_CLIENT_EMAIL,
    },
  })
  const authToken = await auth.getClient()
  return authToken
}

---

env.local

GOOGLE_PROJECTID=sheets-xxxxxxxxxxxxxx
GOOGLE_PRIVATE_KEY='{"privateKey":"-----BEGIN PRIVATE KEY.........-----END PRIVATE KEY-----\n"}'
GOOGLE_CLIENT_EMAIL=conta-sheets-de-servi-o@xxxxxxxxxxxxx.iam.gserviceaccount.com

[AarRidho]

I finally can store my private Firebase key on Vercel
Here's my take for Windows User:

• Using @grikomsn solution,
• Do not use cmd

I was using windows for my development and cmd for the default terminal. After 8 hours trying to repeat the same thing, I decided to use git bash instead and then it just works in less than 1 minute. 😄

[tlays]

Honestly, I've been using this solution proposed on SO https://stackoverflow.com/a/41044630/10833201 for quite some time and never had any issue with it.

In a nutshell:

• you add you pkey secret via the Vercel dashboard as usual
• then you can access it like this const pKey = process.env.FIREBASE_PRIVATE_KEY.replace(/\\n/g, '\n') (note the .replace(/\\n/g, '\n') pattern at the end)

PROS:

• you can use the Vercel dashboard to add secrets (no need to use the console)
• your PKEY string (however long) does NOT need to be in JSON, ou in double-double quotes as suggested above
• it keeps your code a bit cleaner too

CONS: none that I can think of.

Hope this helps.

[Zainab-Kalief]

@tlays life saver!!! Thank you

I like that I can add my env var from the Vercel dashboard while using this method. According to Vercel's CLI doc, storing env vars as secrets is deprecated. They recommend using their UI

[PierBover]

What I did was simply store the key of the service account via Vercel's UI inside double quotes:

"-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG ...... xesRXxMw8\n-----END PRIVATE KEY-----\n"

And then simply use JSON.parse(process.env.GOOGLE_SERVICE_ACCOUNT_KEY) which will correctly parse the line breaks to initialize the Google Cloud SDK.

[styfle]

Hello everyone 👋

We made some changes to the Environment Variables page today: https://vercel.com/changelog/environments-variables-per-git-branch

I think a lot of the confusion around these private keys is that the docs assume you are adding the key in a json file so it has escaped newlines instead of literal newlines.

However, the Environment Variable page accepts real newlines so you can hit Enter to add a newline where appropriate.

You can also use Vercel CLI to add Environment Variables with a value from a file. For example:

vercel env add FIREBASE_PRIVATE_KEY production < firebase.txt

This will use the contents of firebase.txt as if you copy/pasted it in the UI which is great for files with newlines.

Hope that helps 👍

[mate-h]

Private keys are too large for Lambda AWS. :(

Error! The total size of all Environment Variables (5.26KB) exceeds 4KB.
Learn More: https://vercel.com/docs/v2/platform/limits#environment-variables

Any way to copy a file to the output directory at build time but not have the file public?

[mattiasw]

Private keys are too large for Lambda AWS. :(

Error! The total size of all Environment Variables (5.26KB) exceeds 4KB.
Learn More: https://vercel.com/docs/v2/platform/limits#environment-variables

Any way to copy a file to the output directory at build time but not have the file public?

Here's a workaround for that problem: https://leerob.io/blog/vercel-env-variables-size-limit

[NickCrews]

Looks like people are tackling this a whole lot of different ways, but I think a common problem is not understanding how Vercel parses secrets when they are stored in the web UI. I think I have figured some things out. In summary what you put in the web editor is exactly what you get.

1. You should not include quotes in the web editor, or they will be taken as literal quotes. e.g. if you put "inquotes" in the web editor, then the value of your variable would have " on either side of it.
2. \n characters in the web editor are taken as a literal \ and n character. If you want a newline character, then there should be an actual line break in the web editor.

Putting these together, the format that worked for my PEM key looked like this:

So you could store your secrets like this, but it might be easier to use the same format in vercel's web editor as in your .env.local, in which case you should use the JSON-compatible form as described in part 3 of @grikomsn's comment above: #749 (comment)

[vvo]

As I struggled hard gluing pieces together and debugging Vercel env variables/JSON.parse etc I created an article that explains how to store the service account json file directly in Vercel and just use JSON.parse on it. Here it is: https://dev.to/vvo/how-to-add-firebase-service-account-json-files-to-vercel-ph5

tl;dr; remove line breaks (not the ones from the private_key value) and surround your env variable containing JSON with ' instead of " otherwise it will fail.

Dear Vercel: should the default of vercel env pull be to use single quotes instead of double quotes for env variable values? So this works by default instead of having to manually fix it.

[ian]

Any way to copy a file to the output directory at build time but not have the file public?

Here's a workaround for that problem: https://leerob.io/blog/vercel-env-variables-size-limit

I headed down this path and ended up writing my own implementation based on @leerob's solution article. Much like @mate-h I wanted a way to persist a file during artifact compilation. I'm hoping this is helpful to others looking for a solution.

next-secrets provides a quick dev-only interface to managing secrets, which are encrypted into a Redis directory and available through a HOC wrapper around API endpoints. It perists a .secrets file and in my testing it ass around 1-5ms of overhead for the request.

Repository: https://github.com/ian/next-secrets

Commnents and pull-requests appreciated!

[mate-h]

Any way to copy a file to the output directory at build time but not have the file public?

Here's a workaround for that problem: https://leerob.io/blog/vercel-env-variables-size-limit

In summary, as this article points out, my case was solved by encrypting the file. So I ran the AES on the private key, stored the encryption secret key in my vercel secrets, and decrypted it at runtime. This is a workaround because the file is still public, without sharing the contents of it. Using redis is smart, since it reduces the memory overhead!

[santhosh-umapathi-appymango]

As I struggled hard gluing pieces together and debugging Vercel env variables/JSON.parse etc I created an article that explains how to store the service account json file directly in Vercel and just use JSON.parse on it. Here it is: https://dev.to/vvo/how-to-add-firebase-service-account-json-files-to-vercel-ph5

tl;dr; remove line breaks (not the ones from the private_key value) and surround your env variable containing JSON with ' instead of " otherwise it will fail.

Dear Vercel: should the default of vercel env pull be to use single quotes instead of double quotes for env variable values? So this works by default instead of having to manually fix it.

This actually did the job in a clean and easy way. Thank you

[jamiegood]

Hi same issue here. Not enough space left to add a private key to Vercel Env Vars.

If I understand this approach here correctly? https://leerob.io/blog/vercel-env-variables-size-limit

Isn't this a big security hole? An api endpoint returns the decrypted private key. Anyone can look at the client side http request to the /api endpoint. See the encrypted key.
And see the response of the api endpoint and get the decrypted contents?

Seems to defeat the purpose of encrypting it in the first place.

[santhosh-umapathi-appymango]

Hi same issue here. Not enough space left to add a private key to Vercel Env Vars.

If I understand this approach here correctly? https://leerob.io/blog/vercel-env-variables-size-limit

Isn't this a big security hole? An api endpoint returns the decrypted private key. Anyone can look at the client side http request to the /api endpoint. See the encrypted key. And see the response of the api endpoint and get the decrypted contents?

Seems to defeat the purpose of encrypting it in the first place.

Check this comment, this was working like charm. https://dev.to/vvo/how-to-add-firebase-service-account-json-files-to-vercel-ph5

[jamiegood]

Thanks. Setting a Vercel Env Var does not work for me because Vercel has a max 4kb size for env vars and this google service account json key file is too big.

[ian]

@jamiegood take a look at https://github.com/ian/next-secrets and let me know if that's what you're looking for. This was our usecase too.

[jamiegood]

@ian I ended up freeing up some space for the next env vars. But that will not last long so will have a look at next-secrets above. Thanks for that.

[thanhtutzaw]

vc secrets add

How can I do this with Netlify ?
vc secrets add -> do I need to use Netlify cli ? When I set this secret key in Netlify env , build failed.

[SSylvain1989]

Honestly, I've been using this solution proposed on SO https://stackoverflow.com/a/41044630/10833201 for quite some time and never had any issue with it.

In a nutshell:

• you add you pkey secret via the Vercel dashboard as usual
• then you can access it like this const pKey = process.env.FIREBASE_PRIVATE_KEY.replace(/\\n/g, '\n') (note the .replace(/\\n/g, '\n') pattern at the end)

PROS:

• you can use the Vercel dashboard to add secrets (no need to use the console)
• your PKEY string (however long) does NOT need to be in JSON, ou in double-double quotes as suggested above
• it keeps your code a bit cleaner too

CONS: none that I can think of.

Hope this helps.

Thanks for the solution - work for me with ANY quote on vercel UI console

[leerob]

Hey folks, good news. We've made this much easier in two main ways:

1. You can now copy paste your .env file directly into the the input! (source)
2. When you paste in private key secrets, we'll automatically convert \n to new lines (source)

[ian]

Nice! @leerob I don't suppose the 4K max size has been solved has it?

[leerob]

It has! Very excited to share: https://vercel.com/changelog/16x-larger-environment-variable-storage-up-to-64kb 🎉

[simkessy]

This is the only thing that worked for me after hows on this.

https://dev.to/vvo/how-to-add-firebase-service-account-json-files-to-vercel-ph5

@grikomsn are you have to manually update the quotes in your env.local every time you do a vc env pull?

[vladshcherbin]

This one worked for me after trying multiple solutions from this issue (key in base64 format):

https://github.com/orgs/vercel/discussions/219#discussioncomment-128702