New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Managing secret keys #7

Open
mnapoli opened this Issue May 22, 2018 · 7 comments

Comments

Projects
None yet
3 participants
@mnapoli
Copy link
Owner

mnapoli commented May 22, 2018

The goal of this task is to make it easy to configure applications with secret keys. That may be through documentation or through tooling.

See https://serverless.com/blog/serverless-secrets-api-keys/ because that will help.

@mnapoli mnapoli added the enhancement label May 22, 2018

@bubba-h57

This comment has been minimized.

Copy link
Contributor

bubba-h57 commented Jan 7, 2019

We wrote a composer package https://github.com/stechstudio/laravel-env-security to manage secrets in Laravel. It would be easy enough to port that over to some bref specific tooling.

@sandrokeil

This comment has been minimized.

Copy link

sandrokeil commented Jan 7, 2019

I don't like env files but it could be reasonable for some projects. I use and prefer a kind of Managing Secrets for Larger Projects and Teams. Is it possible to store a config cache file at runtime in Lambda and it's populated for the next request? This has the advantage to only load "once" the secret keys.

With this, you can use the get_secret function in your PHP config file and if config cache file is available no AWS SSM API call is needed. See zend-config-aggregator for an example.

return [
   'my_secret' => get_secret('MY_SECRET_AWS_KEY'),
]

For development it is useful to use getenv in the get_secret() method to allow also to inject env variables. If no env variable is available a AWS SSM API call is executed. With this approach we have two possibilities combined.

@mnapoli

This comment has been minimized.

Copy link
Owner Author

mnapoli commented Jan 7, 2019

I was more thinking about documenting how to use AWS secret manager (as well as another similar tool I forgot the name). These are the recommended way to store and access secrets in Lambda.

@bubba-h57

This comment has been minimized.

Copy link
Contributor

bubba-h57 commented Jan 7, 2019

I see. That turned out to be expensive and not cost effective for us.

It cost $0.025 per 1,000 API calls to SSM.
It cost 1 API call to decrypt to one parameter.
We average about 6 secrets that need encryption per Lambda Function.
We average 100,000 Lambda executions a day.

Worst case scenario, that is 600,000 SSM API calls per day.
600 * $0.025 = an average cost of $15/day or $450/mnth

We were not seeing the worst case scenario, we did some caching, and tried to work around with it. But ultimately, it cost us far too much because our workload is not consistent throughout the day, and then I run the various bits of work in async parallel across multiple differing Lambda functions. AWS has expanded our concurrent jobs to 20K, and most of the work we do is in spikes of 10-15K jobs in parallel for 30 minutes or so and it can be an hour before we spike again. That means the majority are cold starts.

So we went with AWS KMS instead and found it much more cost efficient.

For one thing, it only costs $0.03 per 10,000 requests.

As we see it, the simplest way that KMS is used to better secure secrets in Lambda projects:

  1. A customer master key (CMK) is generated in KMS, with access only to production lambda functions (an IAM role) and the administrator account
  2. All static secrets are stored in an encrypted .env file.
  3. These .env files containing encrypted secrets (e.g. DB password, mail credentials) as well as any unencrypted non-secrets (e.g. DB user, DB host, etc) can then even be committed to your favorite version control system, if desired.
  4. Lambda bootstrap function is modified so that it uses the KMS decrypt function (part of AWS SDK) on the encrypted .env file and then exports the values to environment variables, ensuring unencrypted secrets are never stored anywhere but the memory.
  5. Because the bootstrap is doing the decryption only on cold starts, the values are effectively cached in memory, only costing 1 API call per cold start for all environment variables.

It is important to us that the secrets are never stored on disk, anywhere, at any time, unencrypted. It is also important to us to minimize the cost in API calls as well as the performance impact of decryption.

Thus, we came up with a way to manage encrypted .env files for this purpose.

I supose for smaller Lambda projects that only make a few 1,000 API calls a month, SSM could be cost effective. That is, simple enough to implement that it is worth the few pennies to use.

However, would never use a solution that writes the secrets to disk, even in the lambda environment, because it is critical security flaw that would result in Financial, Governmental, and Healthcare (and anyone else that requires rigorous security practices) institutions could never leverage it.

Addmittedly, there are always multiple ways to solve this sort of problem, and if we can come up with a more cost effective method, without sacrificing security, I will implement it as fast as possible. If for not other reason than it would have a positive impact on Signature Tech Studio's bottom line. :-)

@mnapoli

This comment has been minimized.

Copy link
Owner Author

mnapoli commented Jan 7, 2019

Thank you that is super useful! That shows that we definitely need a guide for all of that 😄

@sandrokeil

This comment has been minimized.

Copy link

sandrokeil commented Jan 7, 2019

To reduce SSM costs we can query 10 secrets at once via GetParametersByPath. Your security concerns are reasonable.

Here is an example to store the variables in the env vars so you can use getenv() in you config files or elsewhere.

$ssmPath = '[Your environment SSM path e.g. /production/awesome-app/]';

$ssm = new \Aws\Ssm\SsmClient([
    'region' => getenv('AWS_REGION'),
    'version' => getenv('SSM_VERSION'),
]);

$options = [
    'MaxResults' => 10,
    'Path' => $ssmPath,
    'Recursive' => true,
    'WithDecryption' => true,
];

do {
    $result = $ssm->getParametersByPath($options);

    $options['NextToken'] = $result->get('NextToken');

    foreach ($result->get('Parameters') as $parameter) {
        putenv(substr($parameter['Name'], strrpos($parameter['Name'], '/') + 1) . '=' . $parameter['Value']);
    }
} while ($result->get('NextToken'));
@bubba-h57

This comment has been minimized.

Copy link
Contributor

bubba-h57 commented Jan 7, 2019

GetParametersByPath ... that does put the costs back on par! As long as we have ten or less secret variable to manage, the performance would be on par as well (1 API call). I dig it.

mnapoli added a commit that referenced this issue Feb 7, 2019

Revert #202 for reasons explained in the PR (copied below)
I just realized that the code example actually replaces all the existing environment variables.

Additionally I'm not sure it's a good idea at all because it is not an atomic deployment: first cloudformation will deploy the function and then (seconds later) it defines the environment variables. For a short period the variables will not exist.

Until we document a proper solution (see #7) I think it's best to revert this bit because it will create confusion and frustration for users. Sorry for not catching that sooner!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment