AddOn - AWS Secrets Manager and Config Provider for Secret Store CSI Driver #471

anshrma · 2022-04-25T23:03:58Z

What does this PR do?

This PR adds two Kubernetes add-on , (1) AWS Secrets Manager and Config Provider for Secret Store CSI Driver (2) Secrets Store CSI Driver

Motivation

AWS partner needing this add on to complete their development work.
Closes #435

More

[ Y] Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
[ Y ] Yes, I have added a new example under examples to support my PR
[ NA ] Yes, I have created another PR for add-ons under add-ons repo (if applicable)
[ Y] Yes, I have updated the docs for this feature
Yes, I ran pre-commit run -a with this PR

Note: Not all the PRs required examples and docs except a new pattern or add-on added.

For Moderators

E2E Test successfully complete before merge?

Additional Notes

…ueprints into secrets-manager

vara-bonthu

Great work @anshrma ! LGTM 👍🏼

Few minor changes requested. We just need to run some E2E tests before merging the PR

examples/secrets-management-with-csi-secrets-driver-aws/variables.tf

modules/kubernetes-addons/csi-secrets-store-provider-aws/locals.tf

modules/kubernetes-addons/secrets-store-csi-driver/locals.tf

anshrma · 2022-04-26T21:31:55Z

Great work @anshrma ! LGTM 👍🏼

Few minor changes requested. We just need to run some E2E tests before merging the PR

Thanks @vara-bonthu - Addressed your feedbacks.

naris-silpakit

LGTM!

bryantbiggs

Thanks for the PR! Please update your branch with the latest on main as there have been a number of changes to the examples/ format

bryantbiggs · 2022-05-02T20:18:55Z

examples/secrets-management-with-csi-secrets-driver-aws/README.md

@@ -0,0 +1,131 @@
+# EKS Cluster Deployment with new VPC


Instead of creating a whole new example just for these two addons, instead can we just add their implementation into the https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/complete-kubernetes-addons

@askulkarni2 / @kcoleman731 what are your thoughts on this as we continue to expand the number of addons?

I agree with @bryantbiggs . We should start moving the individual Add-on examples to [complete-kubernetes-addons](https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/main/examples/complete-kubernetes-addons) . This will help us to manage limited exmaples

We can add some more details to docs .md files as well. New users should be able to goto docs to findout the usage of this Add-on.

bryantbiggs · 2022-05-02T20:20:00Z

examples/secrets-management-with-csi-secrets-driver-aws/data.tf

@@ -0,0 +1,11 @@
+data "aws_iam_policy_document" "secrets_management_policy" {


when you move this over to the complete-kubernetes-addons example, no need to put this in a separate file. Instead, locate it next to where it is used. Here that would be either before or after https://github.com/aws-ia/terraform-aws-eks-blueprints/pull/471/files#diff-6355537d0664dc024a0086c9581f3a707332dd67c563f98cbe5ee18d4fd0e44bR164

examples/secrets-management-with-csi-secrets-driver-aws/main.tf

examples/secrets-management-with-csi-secrets-driver-aws/secretconfig.yaml

modules/kubernetes-addons/secrets-store-csi-driver/locals.tf

modules/kubernetes-addons/secrets-store-csi-driver/main.tf

docs/add-ons/csi-secrets-store-provider-aws.md

examples/secrets-management-with-csi-secrets-driver-aws/README.md

askulkarni2

First of all thanks so much for this PR! The PR is looking good. There is one main outstanding design item. We probably want to keep the secrets provisioning design similar to how we do in CDK (if possible). Refer to https://github.com/aws-quickstart/cdk-eks-blueprints/blob/main/docs/extensibility.md#passing-secrets-to-add-ons.

docs/add-ons/csi-secrets-store-provider-aws.md

examples/secrets-management-with-csi-secrets-driver-aws/main.tf

askulkarni2 · 2022-05-03T01:17:23Z

examples/secrets-management-with-csi-secrets-driver-aws/main.tf

+# Creating IAM Role for Service Account
+#---------------------------------------------------------------
+
+module "iam_role_service_account" {


So, the problem here is that the addon itself has no clue what secrets are being retrieved. The way it works, secrets are retrieved by the AWS provider when a pod needing those secrets is started and removed afterwords. The service-account associated with the respective pods needs to have the needed IRSA config. These policies cannot be added to the addon. In CDK, we do this when the Team or AddOn secrets are defined. The same thing needs to happen here. That is to say, in order to consume secrets, there are two paths:

You need to define your secrets via Teams configuration (similar to how we do labels and quotas today and we will have to update the Teams module for this). We need to add the required policies to the IRSA config we define for the Team.

For add-ons, this will be a parameter for the add-on in which case we will add the required policies to the addons IRSA config.

CC: @kcaws @shapirov103 I am assuming we want to keep the design for secrets somewhat similar between the two projects.

modules/kubernetes-addons/csi-secrets-store-provider-aws/main.tf

modules/kubernetes-addons/secrets-store-csi-driver/locals.tf

modules/kubernetes-addons/secrets-store-csi-driver/main.tf

…ueprints into secrets-manager

modules/kubernetes-addons/csi-secrets-store-provider-aws/locals.tf

askulkarni2

@anshrma looking good, just one final change for using namespace resource instead of irsa module as suggested by Vara. Also please address the CI failures. Running pre-commit run -a should fix most of them.

docs/add-ons/csi-secrets-store-provider-aws.md

examples/secrets-management-with-csi-secrets-driver-aws/main.tf

modules/kubernetes-addons/csi-secrets-store-provider-aws/locals.tf

…ueprints into secrets-manager

…accelerator-for-terraform into secrets-manager

…ueprints into secrets-manager

anshrma · 2022-06-14T23:33:08Z

@anshrma looking good, just one final change for using namespace resource instead of irsa module as suggested by Vara. Also please address the CI failures. Running pre-commit run -a should fix most of them.

Thanks @askulkarni2 . Done with the changes requested and addressed the CI failures. Please let me know, if any other changes required.

BlZvi · 2022-06-28T17:08:31Z

Hey @anshrma, @askulkarni2 and @bryantbiggs, Whaat is missing in this PR to go forward?

updated

askulkarni2

Updated, tested and approved!

…Driver (aws-ia#471) Co-authored-by: Bryant Biggs <bryantbiggs@gmail.com> Co-authored-by: Apoorva Kulkarni <kuapoorv@amazon.com>

anshrma added 5 commits April 25, 2022 11:20

Initial commit for adding CSI AWS Secrets Driver

2bb4e9b

completing secrets manager example

ebd7361

instructions to use the add on

6b9fbc8

adding add-on doc

0258951

Merge branch 'main' of https://github.com/aws-ia/terraform-aws-eks-bl…

2835af5

…ueprints into secrets-manager

anshrma requested review from askulkarni2, kcoleman731 and vara-bonthu as code owners April 25, 2022 23:03

anshrma added 2 commits April 26, 2022 10:43

splitting secrets store csi driver and secrets manager implementation

2981ade

Merge branch 'main' of https://github.com/aws-ia/terraform-aws-eks-bl…

8cd92d5

…ueprints into secrets-manager

vara-bonthu approved these changes Apr 26, 2022

View reviewed changes

PR feedback

c950b24

terraform fmt fixes

60ac308

askulkarni2 requested review from bryantbiggs and naris-silpakit May 2, 2022 19:20

naris-silpakit approved these changes May 2, 2022

View reviewed changes

bryantbiggs previously requested changes May 2, 2022

View reviewed changes

askulkarni2 reviewed May 3, 2022

View reviewed changes

anshrma added 2 commits May 24, 2022 11:39

Merge branch 'main' of https://github.com/aws-ia/terraform-aws-eks-bl…

33a5493

…ueprints into secrets-manager

PR feedback

f354548

anshrma requested a review from Zvikan as a code owner May 24, 2022 21:46

anshrma had a problem deploying to EKS Blueprints Test May 24, 2022 21:46 Failure

creating a secret in example

3b8e7ea

anshrma had a problem deploying to EKS Blueprints Test May 24, 2022 23:58 Failure

vara-bonthu reviewed Jun 6, 2022

View reviewed changes

modules/kubernetes-addons/csi-secrets-store-provider-aws/locals.tf Outdated Show resolved Hide resolved

back to namespace creation and not using irsa config

582e92f

askulkarni2 reviewed Jun 9, 2022

View reviewed changes

anshrma added 4 commits June 9, 2022 09:59

pre-commit hook

8ed209b

Update README.md

5b54732

Merge branch 'main' of https://github.com/aws-ia/terraform-aws-eks-bl…

49928fa

…ueprints into secrets-manager

Merge branch 'secrets-manager' of https://github.com/anshrma/aws-eks-…

9236df9

…accelerator-for-terraform into secrets-manager

anshrma had a problem deploying to EKS Blueprints Test June 9, 2022 18:16 Failure

Merge branch 'main' of https://github.com/aws-ia/terraform-aws-eks-bl…

8e47d8f

…ueprints into secrets-manager

anshrma had a problem deploying to EKS Blueprints Test June 14, 2022 23:32 Failure

askulkarni2 added 2 commits June 28, 2022 14:47

Merge remote-tracking branch 'upstream/main' into secrets-manager

9993bb4

run pre-commit

811d2b9

askulkarni2 temporarily deployed to EKS Blueprints Test June 28, 2022 22:34 Inactive

edit example to fit the new style guideline

e777f75

askulkarni2 temporarily deployed to EKS Blueprints Test June 29, 2022 05:07 Inactive

askulkarni2 approved these changes Jun 29, 2022

View reviewed changes

update region

be58ce5

askulkarni2 temporarily deployed to EKS Blueprints Test June 29, 2022 05:12 Inactive

fix ci check to use CMK instead of default key

1b68a02

askulkarni2 temporarily deployed to EKS Blueprints Test June 29, 2022 18:28 Inactive

askulkarni2 added 2 commits June 29, 2022 12:36

add random provider

0426cac

fix for tflint failures

06b5b82

askulkarni2 temporarily deployed to EKS Blueprints Test June 29, 2022 20:40 Inactive

missing newline

afa51f0

askulkarni2 temporarily deployed to EKS Blueprints Test June 29, 2022 20:57 Inactive

askulkarni2 merged commit 472de70 into aws-ia:main Jun 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AddOn - AWS Secrets Manager and Config Provider for Secret Store CSI Driver #471

AddOn - AWS Secrets Manager and Config Provider for Secret Store CSI Driver #471

anshrma commented Apr 25, 2022 •

edited by bryantbiggs

Loading

vara-bonthu left a comment

anshrma commented Apr 26, 2022

naris-silpakit left a comment

bryantbiggs left a comment

bryantbiggs May 2, 2022

bryantbiggs May 2, 2022

vara-bonthu Jun 6, 2022

vara-bonthu Jun 6, 2022

bryantbiggs May 2, 2022

askulkarni2 left a comment

askulkarni2 May 3, 2022

askulkarni2 left a comment •

edited

Loading

anshrma commented Jun 14, 2022

BlZvi commented Jun 28, 2022

askulkarni2 left a comment

		@@ -0,0 +1,11 @@
		data "aws_iam_policy_document" "secrets_management_policy" {

AddOn - AWS Secrets Manager and Config Provider for Secret Store CSI Driver #471

AddOn - AWS Secrets Manager and Config Provider for Secret Store CSI Driver #471

Conversation

anshrma commented Apr 25, 2022 • edited by bryantbiggs Loading

What does this PR do?

Motivation

More

For Moderators

Additional Notes

vara-bonthu left a comment

Choose a reason for hiding this comment

anshrma commented Apr 26, 2022

naris-silpakit left a comment

Choose a reason for hiding this comment

bryantbiggs left a comment

Choose a reason for hiding this comment

bryantbiggs May 2, 2022

Choose a reason for hiding this comment

bryantbiggs May 2, 2022

Choose a reason for hiding this comment

vara-bonthu Jun 6, 2022

Choose a reason for hiding this comment

vara-bonthu Jun 6, 2022

Choose a reason for hiding this comment

bryantbiggs May 2, 2022

Choose a reason for hiding this comment

askulkarni2 left a comment

Choose a reason for hiding this comment

askulkarni2 May 3, 2022

Choose a reason for hiding this comment

askulkarni2 left a comment • edited Loading

Choose a reason for hiding this comment

anshrma commented Jun 14, 2022

BlZvi commented Jun 28, 2022

askulkarni2 left a comment

Choose a reason for hiding this comment

anshrma commented Apr 25, 2022 •

edited by bryantbiggs

Loading

askulkarni2 left a comment •

edited

Loading