Transformation Example for dockerconfigjson type secret, from KV-V2's flat keys and values

[imranzunzani]

Is there any support in Transformation for setting up a dockerconfigjson type secret for eg. imagePullSecrets, where keys for server, username and password from kv-v2 be mapped to template values?
Like this:
https://blog.kelbert.fr/posts/acces-private-docker-registry-with-vault-secret-operator-s-help/

Else, could you please share, how to do this with the go text template way, or point me to a documentation for that?

Please tag this as question.

[benashz]

Is there any support in Transformation for setting up a dockerconfigjson type secret for eg. imagePullSecrets, where keys for server, username and password from kv-v2 be mapped to template values? Like this: https://blog.kelbert.fr/posts/acces-private-docker-registry-with-vault-secret-operator-s-help/

Yes, it is definitely possible with VSO v0.5.0+. You can configure a secret CR's spec.destination.type=kubernetes.io/dockerconfigjson and provide the appropriate template text to render the dockerconfigjson data. The template definition will be highly dependent on the contents/format of your secret data input.

Here's an example:

Given the following kv-v2 Vault data:

$ vault kv get -format json docker/config | jq .data.data
{
  "auths": {
    "host1": {
      "password": "pass1",
      "username": "bob"
    },
    "host2": {
      "password": "pass2",
      "username": "alice"
    }
  }
}

This VSS resource should produce a valid K8s secret of type kubernetes.io/dockerconfigjson:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: dockerconfig
  namespace: demo-ns
spec:
  destination:
    type: kubernetes.io/dockerconfigjson
    transformation:
      excludeRaw: true
      excludes:
        - .*
      templates:
        ".dockerconfigjson":
          text: |
            {{- $config := dict -}}
            {{- range $k, $v := (get .Secrets "auths") -}}
              {{- $username := get $v "username" -}}
              {{- $password := get $v "password" -}}
              {{- $_ := set $config $k (
                dict 
                  "username" $username
                  "password" $password
                  "auth" (list $username ":" $password | join "" | b64enc)
                )
              -}}
            {{- end -}}
            {{- dict "auths" $config | mustToJson -}}
    create: true
    name: pull-secrets-multi
  hmacSecretData: true
  mount: docker
  path: config
  type: kv-v2

Resulting K8s secret data value:

{
  "auths": {
    "host1": {
      "auth": "Ym9iOnBhc3Mx",
      "password": "pass1",
      "username": "bob"
    },
    "host2": {
      "auth": "YWxpY2U6cGFzczI=",
      "password": "pass2",
      "username": "alice"
    }
  }
}

Given the following kv-v2 Vault data (single auth entry):

$ vault kv get -format json docker/config | jq .data.data
{
  "single": {
    "host": "host1",
    "password": "pass1",
    "username": "bob"
  }
[...]
}

This VSS resource should produce a valid K8s secret of type kubernetes.io/dockerconfigjson (single auth entry):

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: docker-config-single
  namespace: demo-ns
spec:
  destination:
    transformation:
      excludeRaw: true
      excludes:
        - .*
      templates:
        ".dockerconfigjson":
          text: |
            {{- $config := get .Secrets "single" | mustDeepCopy -}}
            {{- $host := get $config "host" -}}
            {{- $_ := set $config "auth" (list (get $config "username") ":" (get $config "password") | join "" | b64enc) -}}
            {{- $_ := unset $config "host" -}}
            {{- dict "auths" (dict $host $config) | mustToJson -}}
    create: true
    name: pull-secrets-single
    overwrite: false
    type: kubernetes.io/dockerconfigjson
  hmacSecretData: true
  mount: docker
  path: config
  type: kv-v2

Resulting K8s secret data value (single auth entry):

{
  "auths": {
    "host1": {
      "auth": "Ym9iOnBhc3Mx",
      "password": "pass1",
      "username": "bob"
    }
  }
}

[imranzunzani]

Hi @benashz ,
The data that I have in Vault is not nested.

vault kv get -format json secret/DockerRegistry | jq .data.data
{
  "hostname": "registry.docker-private.com",
  "password": "registry-password",
  "username": "registry-user"
}

There are no 'auths' and 'hosts' in that data to loop through. Those are required to be constructed using the template, similar to the example in that blog.

[benashz]

Hi @benashz , The data that I have in Vault is not nested.

vault kv get -format json secret/DockerRegistry | jq .data.data
{
  "hostname": "registry.docker-private.com",
  "password": "registry-password",
  "username": "registry-user"
}

There are no 'auths' and 'hosts' in that data to loop through. Those are required to be constructed using the template, similar to the example in that blog.

I think you can probably adapt the example for the single entry that I provided above. The transformation would look something like the following, given this kv-v2 Vault data:

{
  "hostname": "host1",
  "password": "pass1",
  "username": "bob"
}

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: docker-config-single-not-nested
  namespace: demo-ns
spec:
  destination:
    transformation:
      excludeRaw: true
      excludes:
        - .*
      templates:
        ".dockerconfigjson":
          text: |
            {{- $config := .Secrets | mustDeepCopy -}}
            {{- $hostname := get $config "hostname" -}}
            {{- $_ := unset $config "hostname" -}}
            {{- $_ := set $config "auth" (list (get .Secrets "username") ":" (get .Secrets "password") | join "" | b64enc) -}}
            {{- dict "auths" (dict $hostname $config) | mustToJson -}}
    labels:
      vss: "true"
    create: true
    name: image-pull-secrets-single-not-nested
    overwrite: false
    type: kubernetes.io/dockerconfigjson
  hmacSecretData: true
  mount: tenant-kv
  path: x-ns-not-nested
  type: kv-v2

Yields the following k8s secret data:

{
  "auths": {
    "host1": {
      "auth": "Ym9iOnBhc3Mx",
      "password": "pass1",
      "username": "bob"
    }
  }
}

[TJM]

Could you maybe provide an example for using dynamic secrets, such as Google Secrets Engine? or Artifactory Secrets Engine?

I am assuming it supports dynamic (leased) secrets, as I can't figure why someone would use vault if they are OK with having static secrets :)

Thanks :)

[benashz]

Could you maybe provide an example for using dynamic secrets, such as Google Secrets Engine? or Artifactory Secrets Engine?

I am assuming it supports dynamic (leased) secrets, as I can't figure why someone would use vault if they are OK with having static secrets :)

Thanks :)

@TJM Indeed, the secrets transformation feature is supported on all secret type custom resources: VaultDynamicSecret, VaultPKISecret, VaultStaticSecret, HCPVaultSecretsApp.

The template needed to render a docker config JSON would need to match the secret data input. The sample transformations provided here could probably be adapted without too much effort.

[imranzunzani]

Hi @benashz ,

I tried your example, and am getting the following error:

Failed to update k8s secret: Secret "imagePullSecrets" is invalid: metadata.name: Invalid value: "imagePullSecrets": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')	{"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"app","name":"vault-kv-app","uid":"6bcff28e-5017-4e0f-ae51-45e26de83da1","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"376561231"}, "reason": "SecretSyncError"}

I used the following:

    transformation:
      excludeRaw: true
      excludes:
        - .*
      templates:
        ".dockerconfigjson":
          text: |
            {{- $config := .Secrets | mustDeepCopy -}}
            {{- $hostname := get $config "hostname" -}}
            {{- $_ := unset $config "hostname" -}}
            {{- $_ := set $config "auth" (list (get .Secrets "username") ":" (get .Secrets "password") | join "" | b64enc) -}}
            {{- dict "auths" (dict $hostname $config) | mustToJson -}}

[benashz]

Hi @imranzunzani - it looks like the your destination secret name does not conform to RFC1123. Can you provide the complete spec.destination here?

[imranzunzani]

Hi @benashz - Changing the name of the secret resolved it. Thank you.

[makas45]

@benashz Thanks for your support, now below vaultstaticsecret working fine but when we create this operator as Argocd we got the below error.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: image-pull-secret
  namespace: management-system
spec:
  destination:
    transformation:
      excludeRaw: true
      excludes:
        - .*
      templates:
        ".dockerconfigjson":
          text: |
            {{- $config := .Secrets | mustDeepCopy -}}
            {{- $hostname := get $config "hostname" -}}
            {{- $_ := unset $config "hostname" -}}
            {{- $_ := set $config "auth" (list (get .Secrets "username") ":" (get .Secrets "password") | join "" | b64enc) -}}
            {{- dict "auths" (dict $hostname $config) | mustToJson -}}
    labels:
      vss: "true"
    create: true
    name: image-pull-secret
    overwrite: false
    type: kubernetes.io/dockerconfigjson
  hmacSecretData: true
  mount: secret
  path: image-pull-secret
  type: kv-v2
  namespace: openshift
  vaultAuthRef: vault-auth

error message which we got
ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc =helm template . --name-template vault-secrets-operator --namespace managed-operators --kube-version 1.28 --include-crds failed exit status 1: Error: template: vault-secrets-operator/templates/vault-conn-auth.yaml:70:38: executing "vault-secrets-operator/templates/vault-conn-auth.yaml" at <mustDeepCopy>: error calling mustDeepCopy: reflect: call of reflect.Value.Type on zero Value Use --debug flag to render out invalid YAML

[benashz]

@benashz Thanks for your support, now below vaultstaticsecret working fine but when we create this operator as Argocd we got the below error.

error message which we got ComparisonError: Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc =helm template . --name-template vault-secrets-operator --namespace managed-operators --kube-version 1.28 --include-crds failed exit status 1: Error: template: vault-secrets-operator/templates/vault-conn-auth.yaml:70:38: executing "vault-secrets-operator/templates/vault-conn-auth.yaml" at <mustDeepCopy>: error calling mustDeepCopy: reflect: call of reflect.Value.Type on zero Value Use --debug flag to render out invalid YAML

@makas45 - I believe that Helm is attempting to render the templates under spec.destination.transformation. I think you will need to escape template text by surrounding it in {{` `}} so that Helm does not attempt to render it:

Your example updated to prevent Helm from rendering the template content.

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: image-pull-secret
  namespace: management-system
spec:
  destination:
    transformation:
      excludeRaw: true
      excludes:
        - .*
      templates:
        ".dockerconfigjson":
          text: |
            {{`{{- $config := .Secrets | mustDeepCopy -}}
            {{- $hostname := get $config "hostname" -}}
            {{- $_ := unset $config "hostname" -}}
            {{- $_ := set $config "auth" (list (get .Secrets "username") ":" (get .Secrets "password") | join "" | b64enc) -}}
            {{- dict "auths" (dict $hostname $config) | mustToJson -}}`}}
    labels:
      vss: "true"
    create: true
    name: image-pull-secret
    overwrite: false
    type: kubernetes.io/dockerconfigjson
  hmacSecretData: true
  mount: secret
  path: image-pull-secret
  type: kv-v2
  namespace: openshift
  vaultAuthRef: vault-auth

[TJM]

(I know I am responding to a closed issue... but this seemed like the best place for the example:

Artifactory Secrets engine to dockerconfigjson

• Artifactory secrets engine does not send the "hostname" as a configuration option, so it needs to be "hard-coded"
• The template also needs slightly adjusted to use "access_token" as the password.
• The "join" was updated to use a ":" and remove the ":" as one of the fields (that could be edited above too)

              {{- $config := .Secrets | mustDeepCopy -}}
              {{- $hostname := "https://artifactory.company.com" -}}
              {{- $_ := set $config "auth" (list (get .Secrets "username") (get .Secrets "access_token") | join ":" | b64enc) -}}
              {{- dict "auths" (dict $hostname $config) | mustToJson -}}

... one thing to note, it will set several "unnecessary" fields like role, scope, token_id, username and access_token. You could probably remove them, as I am fairly certain the only field that is actually used is "auth".

[makas45]

we have resolved the issue now.