How to add multiple secrets

[dnarwani]

I have a secret called

apiVersion: mumoshu.github.io/v1alpha1
kind: AWSSecret
metadata:
name: db-secrets
namespace: authentication
spec:
stringDataFrom:
secretsManagerSecretRef:
secretId: mysecret
versionId: ee113603-3254-478a-bb27-40027ae4ff60

I want to be to add multiple entries in 1 secret, is this possible?

[ghostsquad]

is this possible? I'd also like to know.

[mumoshu]

Hi! Unfortunately, it isn't possible today. Can we just extend the spec to accept two or more secretsManagerSecretRefs and merge them? On duplicate key, the latter and newer value for the key will win over the old one

Also I'm curious - what's the expected use-case of this?

[osterman]

what's the expected use-case of this?

Avoid polluting ASM with too many key/value pairs, and be able to atomic updates to a set of secrets that are related and should be versioned together.

See an example here: https://docs.aws.amazon.com/secretsmanager/latest/userguide/tutorials_basic.html

They provide an example like this one which has a username and password for a service:

$ aws secretsmanager get-secret-value --secret-id tutorials/MyFirstTutorialSecret --version-stage AWSCURRENT
{
    "ARN": "arn:aws:secretsmanager:region:123456789012:secret:tutorials/MyFirstTutorialSecret-jiObOV",
    "Name": "tutorials/MyFirstTutorialSecret",
    "VersionId": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE",
    "SecretString": "{\"username\":\"myserviceusername\",\"password\":\"MyVerySecureP@ssw0rd!\"}",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": 1522680764.668
}

So I think a similar expectation where we could serialize perhaps the secret as JSON when we write it to ASM. Then when the operator writes the secret, it becomes something like this (basing it off the example here):

{
  "kind": "Secret",
  "apiVersion": "v1",
  "metadata": {
    "name": "example",
    "namespace": "default",
    "selfLink": "/api/v1/namespaces/default/secrets/test",
    "uid": "82ef45ee-4fdd-11e8-87bf-00e092001ba4",
    "resourceVersion": "25758",
    "creationTimestamp": "2018-05-04T20:55:43Z"
  },
  "data": {"username":"myserviceusername", "password":"MyVerySecureP@ssw0rd!"}",
  "type": "Opaque"
}

maybe we would define the AWSSecret like this with a new deserialize flag:

apiVersion: mumoshu.github.io/v1alpha1
kind: AWSSecret
metadata:
  name: example
spec:
  stringDataFrom:
    secretsManagerSecretRef:
      secretId: prod/mysecret
      deserialize: true
      versionId: c43e66cb-d0fe-44c5-9b7e-d450441a04be

[osterman]

Actually, I think in your example, you already show how this is possible:

$ aws secretsmanager put-secret-value\
    --secret-id prod/mysecret \
    --secret-string '{"foo":"bar"}'`

So maybe it already works? =) @Nuru can you try?

e.g.

$ aws secretsmanager put-secret-value\
    --secret-id prod/mysecret \
    --secret-string '{"username":"keycloak", "password":"supersecret"}'`

(I think this is different from what @dnarwani is asking, but this works for our use-case)

[casey-robertson]

Just started investigating secretsmanager to move away from a sops-based workflow. Agree with @osterman that the above should just work. I am building secrets with Pulumi but I add this an example value:

const fruit = JSON.stringify({
  fruit: "Apple",
  size: "Medium",
  color: "Red",
});

This results in being able to do the following which aligns with how our applications currently use secrets. Right now they are encrypted in the repo (appsettings.json). Upon deploy, sops decrypts the secrets in place - that folder is then mounted as a Secrets volume in the pod.

aws secretsmanager get-secret-value --secret-id casey-config | jq -r .SecretString | jq
{
  "fruit": "Apple",
  "size": "Medium",
  "color": "Red"
}

Glad I stumbled across this because my first thought was to:

1. Add new step to CI/CD to take decrypted secrets and push them to SSM. Or push them encrypted and work out a plan to pull them all back globally into a single 'master' repo. Still trying to understand that concept.
2. Add new step to CI/CD that goes and gets the secrets from SM and drops them in the same folder/directory
3. Deprecate sops step.

End result is same file mounted as a secret in Kubernetes - however with the risks/caveats outlined in the readme about the CI/CD system having decrypted secrets etc. I'll still probably work on step 1 to do initial secret population.

[sh240293]

Hey, can any one suggest to use secrets from parameter store using this method. I have used this https://github.com/toVersus/aws-ssm-operator which points as a ref to this repo. But here i am not able to get secrets more than 10,
I have used path type and settings working fine, but there is a hard limit of 10 somewhere defined i think , Please help me if anyone knows how we can increase.

[mumoshu]

@sh240293 This is what it does and there's no hard limit of 10.

aws-secret-operator/controllers/awssecret_controller.go

Lines 57 to 115 in 564875e

	func (r *AWSSecretController) Reconcile(ctx context.Context, request reconcile.Request) (reconcile.Result, error) {
	reqLogger := log.WithValues("Request.Namespace", request.Namespace, "Request.Name", request.Name)
	// Fetch the AWSSecret instance
	instance := &mumoshuv1alpha1.AWSSecret{}
	err := r.Client.Get(ctx, request.NamespacedName, instance)
	if err != nil {
	if errors.IsNotFound(err) {
	// Request object not found, could have been deleted after reconcile request.
	// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
	// Return and don't requeue
	return reconcile.Result{}, nil
	}
	// Error reading the object - requeue the request.
	return reconcile.Result{}, err
	}
	// Define a new Secret object
	desired, err := r.newSecretForCR(instance)
	if err != nil {
	return reconcile.Result{}, errs.Wrap(err, "failed to compute secret for cr")
	}
	// Set AWSSecret instance as the owner and controller
	if err := controllerutil.SetControllerReference(instance, desired, r.Scheme); err != nil {
	return reconcile.Result{}, err
	}
	// Check if this Secret already exists
	current := &corev1.Secret{}
	err = r.Client.Get(ctx, types.NamespacedName{Name: desired.Name, Namespace: desired.Namespace}, current)
	if err != nil && errors.IsNotFound(err) {
	reqLogger.Info("Secret does not exist, Creating a new Secret", "desired.Namespace", desired.Namespace, "desired.Name", desired.Name)
	err = r.Client.Create(ctx, desired)
	if err != nil {
	return reconcile.Result{}, err
	}
	// Secret created successfully - requeue after 5 minutes
	reqLogger.Info("Secret Created successfully, RequeueAfter 5 minutes")
	return reconcile.Result{RequeueAfter: time.Second * 300}, nil
	} else if err != nil {
	return reconcile.Result{}, err
	}
	// if Secret exists, only update if versionId has changed
	if string(current.Data["AWSVersionId"]) != desired.StringData["AWSVersionId"] {
	reqLogger.Info("versionId changed, Updating the Secret", "desired.Namespace", desired.Namespace, "desired.Name", desired.Name)
	err = r.Client.Update(ctx, desired)
	if err != nil {
	return reconcile.Result{}, err
	}
	// Secret updated successfully - requeue after 5 minutes
	reqLogger.Info("Secret Updated successfully, RequeueAfter 5 minutes")
	return reconcile.Result{RequeueAfter: time.Second * 300}, nil
	}
	return reconcile.Result{RequeueAfter: time.Second * 300}, nil
	}

Also, this issue was about trying to create a single K8s secret from multiple secretsmanager secrets using the key hierarchy, which isn't relevant with your goal as I think,

[sh240293]

Hey @mumoshu thanks for the reply, I have configured an operator that works with parameter store on basis of path and it will sync all variables defined at that common path, however i gone through deep into troubleshoot i got this error message

"E0414 06:29:07.599684 1 reflector.go:134] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: Failed to list *v1alpha1.ParameterStore: v1alpha1.ParameterStoreList.Items: []v1alpha1.ParameterStore: v1alpha1.ParameterStore.Spec: v1alpha1.ParameterStoreSpec.ValueFrom: v1alpha1.ValueFrom.ParameterStoreRef: v1alpha1.ParameterStoreRef.Path: ReadString: expects " or n, but found [, error found in #10 byte of ...|:{"path":["/farmstoc|..., bigger context ...|"spec":{"valueFrom":{"parameterStoreRef":{"path":["/farmstock-backend/dev/"]}}}}],"kind":"ParameterS|..."

I have no idea on the coding part, but if you can suggest how i can fix that would be really helpful to me. That https://github.com/toVersus/aws-ssm-operator unfortunately this github owner is not responding.

[mumoshu]

@sh240293 Hey. Your error seems to be coming from https://github.com/toVersus/aws-ssm-operator, not aws-secret-operator.