Add IBM provider metadata to secret labels and/or annotations

[vsroy]

Problem Statement

We would like the ESO to add a certain metadata while creating or updating a Kubernetes secret to be reflected in its labels or annotations. The metadata could be any of the fields above that are supported and returned in the response by the IBM Cloud provider.

For a secret created in IBM Cloud Secrets Manager, we have the metadata that can be associated with a secret, like secret-group-id, secret-type, secret-id etc for cross-referencing. This will help in below:

• Make the secret self describing enabling it to be cross-referenced easier for debugging in case something goes wrong in the cluster
• Update timestamp in the secret for easier debugging and knowing if a secret update caused pod errors
• And other details which could help in the long run.
• We also have integrations that are based on the metadata available in the Kubernetes secret labels.

Related Issue

Fixes (#2369)

Proposed Changes

The existing templating concept provided by ESO can be leveraged to supplement the labels, where the metadata can be added, along with the data fields. Refer below yaml file.

Below are the changes from different perspectives:
i. User:
In order for the user to opt-in to adding metadata to secret, an existing optional field can be leveraged under spec.dataFrom.extract.metadataPolicy. This is an optional, whose value defaults to None. The required metadata can then be specified under template.metadata.labels or template.metadata.annotations, same as existing. The required secret data can be specified under template.data. While the secret is being reconciled, it will have the secret data along with the required labels.

In the below example, secret_id and updated_at are the metadata of a secret in the Secrets Manager.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sample-external-secret
  namespace: external-secrets
spec:
  dataFrom:
  - extract:
      key: username_password/test-secret
      metadataPolicy: Fetch           # leveraging optional parameter, defaults to None
    secretKey: secret
  secretStoreRef:
    kind: SecretStore
    name: sm-secret-store
  target:
    name: sample-secret
    template:
      engineVersion: v2
      data:
        secret: "{{ .password }}" 
      metadata:
        labels:
          secret_id: "{{ .id }}"
          updated_at: "{{ .updated_at }}"

ii. Provider:
The GetSecretMap() in provider.go under ibm package returns a map[string][]byte holding the value of map of secret values fetched.
If metadataPolicy is set to Fetch, metadata is also included along with the secret value as map in its return value.
If metadataPolicy is set to None, metadata is not included along with the secret value as map in its return value.
If metadataPolicy is not set, it defaults to None and metadata is not included along with the secret value as map in its return value.

iii. Controller applying template:
While applying the template, the controller will de-serialize the content of this dataMap and extract the required label/annotation keys to build the label/annotation key-values in the secret along with the secret data.

Below is the example of the secret after reconciliation:

apiVersion: v1
# ESO adds data
data:
  secret: OHE0MFV5MGhQb2FmRjZTOGVva3dPQjRMeVZXeXpWSDlrSWgyR1BiVDZTMyc=
# End: ESO adds data

immutable: false
kind: Secret
metadata:
  annotations:
    reconcile.external-secrets.io/data-hash: 02217008d13ed228e75cf6d26fe74324
  creationTimestamp: "2023-05-04T08:41:24Z"

# ESO adds labels
  labels:
    secret_id: 1234
    update_at: 2023-05-04T08:57:19Z
# End: ESO adds labels

  name: sample-secret
  namespace: external-secrets
  ownerReferences:
  - apiVersion: external-secrets.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: sample-external-secret
    uid: c2a018e7-1ac3-421b-bd3b-d9497204f843
  resourceVersion: "1803567"
  uid: f5dff604-611b-4d41-9d65-b860c61a0b8d
type: Opaque

Though above example refers to containing the metadata in labels, the same can be extended to annotations as per user requirement.

Note:
The code changes are limited to ibm provider package

• I have read the contribution guidelines
• All commits are signed with git commit --signoff
• My changes have reasonable test coverage
• All tests pass with make test
• I ensured my PR is ready for review with make reviewable

[paul-the-alien]

Greetings!
Thank you for contributing to this project!
If this is your first time contributing, please make
sure to read the Developer and Contributing Process guides.
Please also mind and follow our Code of Conduct.

Useful commands:

• make fmt: Formats the code
• make check-diff: Ensures the branch is clean
• make reviewable: Ensures a PR is ready for review

[gusfcarvalho]

Hi @vsroy ! How does the proposed flag differs from MetadataPolicy? IMO they are the same , and I rather not have an additional field that does slightly the same thing :/

[vsroy]

@gusfcarvalho Thanks for looking into this. I have removed the dependency from the initially proposed flag and set the functionality to be dependent on the existing MetadataPolicy flag being set to Fetch or None. Please review.

[vsroy]

@moolen Can I have an approval on this please.

[moolen]

code-wise LGTM. Though linting fails, could you please fix that? 🙏
I hope you did an integration test against a live API. Because we don't have that covered yet and i don't have an IBM account.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[vsroy]

@moolen Thank you for reviewing the code. The linting issues have been fixed.
Also the changes have been tested against a live API in a k8s cluster. Let me know if you need anything else.

[moolen]

LGTM