Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(parameters): SecretsProvider support #1206

Merged
merged 6 commits into from
Jan 5, 2023

Conversation

dreamorosi
Copy link
Contributor

Description of your changes

This PR introduces the SecretsProvider class which will allow the future Parameters utility to support retrieving secrets from AWS Secrets Manager.

The SecretsProvider supports two retrieval modes, both for retrieving one secret at the time. This is in line with the current functionalities supported by the Parameters utility in Powertools for Python.

  • get this class method allows customers to retrieve a single secret by name and allows to specify aspects like the transformation to apply (none, binary, json, automatic) and max age (aka time to live for the item in the cache).

The class method will be used as such:

const secrets = new SecretsProvider();

const secret = secrets.get('foo');
console.log(secret); // value as string or binary
  • getSecret utility function that behaves the same as the the method, but doesn't require instantiating a class.
const secret = getSecret('foo');
console.log(secret); // value as string or binary

In both cases users can configure how long the secret should be cached in the Lambda's execution environment:

const secretOne = secrets.get('foo', { maxAge: 5000 }); // This will be cached for 5 seconds
// or
const secretTwo = getSecret('foo', { maxAge: 10000 }); // This will be cached for 10 seconds

Secrets Manger allows to store secrets both as SecretString and SecretBinary. Both of them can be base64-encoded.

Using the provider or helper function, users can optionally specify a transform (json, binary, or auto) and SecretsProvider will try to transform the string or binary accordingly. When using auto, the name of the secret must have a suffix that specifies the transform to use (i.e. my-secret.json will result in a json transform). If no transform is specified, the secret will be returned as-is (aka string or Uint8Array).

Examples:

Note
The examples below use the getSecret notation for brevity, but the are 1:1 equivalent to SecretsProvider.get(...)

// SecretString: foo.json - value: "{ \"hello\": \"world\" }" (stringified version of an object)
const a = getSecret('foo.json');
console.log(a); // "{ \"hello\": \"world\" }" (returned as-is)

const b = getSecret('foo.json', { transform: 'auto' });
console.log(b); // { hello: 'world' } (type object)

const c = getSecret('foo.json', { transform: 'json' });
console.log(c); // { hello: 'world' } (type object)

// SecretString: bar.binary - value: "Yw==" (base64 encoded string - 'hello world')
const a = getSecret('bar.binary');
console.log(a); // "Yw==" (returned as-is)

const b = getSecret('bar.binary', { transform: 'auto' });
console.log(b); // "hello world" (type string - base64 decoded)

const c = getSecret('bar.binary', { transform: 'binary' });
console.log(c); // "hello world" (type string - base64 decoded)

// SecretBinary: baz.binary - value: "Yw==" (base64 encoded string - 'hello world')
const a = getSecret('bar.binary');
console.log(a); // [104, 101, 108, 108, 111,  32, 119, 111, 114, 108, 100] (Uint8Array - returned as-is)

const b = getSecret('bar.binary', { transform: 'auto' });
console.log(b); // "hello world" (type string - base64 decoded)

const c = getSecret('bar.binary', { transform: 'binary' });
console.log(c); // "hello world" (type string - base64 decoded)

Finally, users can also customize the behavior of the underlying AWS SDK client calls by passing an optional sdkOptions of type GetSecretValueCommandInput in the second argument:

const secret = getSecret('bar.binary', {
  sdkOptions: {
    VersionId: 'test-version',
  }
});
// or
const secrets = new SecretsProvider();

const secret = secrets.get('bar.binary', {
  sdkOptions: {
    VersionId: 'test-version',
  }
});

Or, for class-based usage only, they can pass their own SDK client config (of type SecretsManagerClient) altogether while initialising the instance:

const secrets = new SecretsProvider({
  clientConfig: { region: 'eu-west-1' }
});

How to verify this change

See the newly added unit test cases & optionally compare also the API surface with the implementation found in Powertools for Python.

Related issues, RFCs

Issue number: #1175

PR status

Is this ready for review?: YES
Is it a breaking change?: NO

Checklist

  • My changes meet the tenets criteria
  • I have performed a self-review of my own code
  • I have commented my code where necessary, particularly in areas that should be flagged with a TODO, or hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding changes to the examples
  • My changes generate no new warnings
  • The code coverage hasn't decreased
  • I have added tests that prove my change is effective and works
  • New and existing unit tests pass locally and in Github Actions
  • Any dependent changes have been merged and published
  • The PR title follows the conventional commit semantics

Breaking change checklist

  • I have documented the migration process
  • I have added, implemented necessary warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@dreamorosi dreamorosi self-assigned this Dec 29, 2022
@dreamorosi dreamorosi linked an issue Dec 29, 2022 that may be closed by this pull request
4 tasks
@pull-request-size pull-request-size bot added the size/L PRs between 100-499 LOC label Dec 29, 2022
@github-actions github-actions bot added the feature PRs that introduce new features or minor changes label Dec 29, 2022
@dreamorosi dreamorosi added the parameters This item relates to the Parameters Utility label Dec 29, 2022
@dreamorosi dreamorosi requested a review from shdq January 5, 2023 08:36
shdq
shdq previously approved these changes Jan 5, 2023
Copy link
Contributor

@shdq shdq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

Should we have tests for the transformation in providers or having them in BaseProvider is enough?

I would add JSDoc for class and methods with a couple of examples (from the PR description).

It looks good to me.

Comment on lines 21 to 27
const sdkOptions: GetSecretValueCommandInput = {
SecretId: name,
};
if (options?.sdkOptions) {
this.removeNonOverridableOptions(options.sdkOptions as GetSecretValueCommandInput);
Object.assign(sdkOptions, options.sdkOptions);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explicit arguments passed to the constructor will take precedence over ones passed to the method.

I looked at the comment above the removeNonOverridableOptions function and instead of explicitly deleting properties, spread options.sdkOptions first (if any) and then overwrite them with name would work:

const sdkOptions: GetSecretValueCommandInput = {
  ...(options?.sdkOptions || {}),
  SecretId: name,
};

It may be less readable, but I like it more than duplicating removeNonOverridableOptions in every provider. Am I missing something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I have applied your suggestion. Thank you!

@dreamorosi
Copy link
Contributor Author

Great work!

Thank you for the review!

Should we have tests for the transformation in providers or having them in BaseProvider is enough?

I think testing in the BaseProvider should be enough for now, we'll add more testing in the integration tests against real end to end retrievals.

I would add JSDoc for class and methods with a couple of examples (from the PR description).

Definitely, we have a dedicated issue for it (#1043). When doing the planning I put it in a separate one because I thought it'd make more sense to do it once the code is more or less stable. I expect to do this sometime towards the end of the integration tests & before the beta release.

@dreamorosi dreamorosi merged commit 02516b7 into main Jan 5, 2023
@dreamorosi dreamorosi deleted the 1175-feature-request-implement-secretsprovider branch January 5, 2023 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature PRs that introduce new features or minor changes parameters This item relates to the Parameters Utility size/L PRs between 100-499 LOC
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature request: implement SecretsProvider
2 participants