Single property support for JSON based values of GCP Secret Manager integration

[thecanadianroot]

Enhancement feature request

This is an enhancement feature request that I've previously opened for GoogleCloudPlatform/spring-cloud-gcp and it was suggested to me that I open this feature request here instead.

Describe the solution you'd like
Not sure if I am the only one trying to inject a single key from a JSON based secret value with spring-cloud-gcp-secret-manager, but it would be great!

Let's say for example a new form that looks like this is added:

sm://<secret-id>/json/<JSONPath>

This could allow someone to pick a single value from let's say this JSON:

Content of my-database's secret within Secret Manager:

{
   "username": "user",
   "password": "very-secret"
}

Usage within Spring's application.yaml:

spring:
  datasource:
    username: ${sm://my-database/json/username}
    password: ${sm://my-database/json/password}

Describe alternatives you've considered
The example I gave above is really simplified and can be avoided by creating two secrets within the Secret Manager, but when you are dealing with many services you end up with a ton of secrets holding single values that, in the end, relates to only one service. Plus, if multiple values must be changed at the same time, new versions must be added to a lot of secrets instead of just one. Here's an example with a Kafka service:

• a single JSON based secret (my-kafka):

{
  "bootstrap-servers": "server-a:1234,server-b:1234",
  "trust-store-certificates": "MY CERT 1",
  "key-store-certificate-chain": "MY CERT 2",
  "key-store-key": "MY KEY",
  "schema-registry-url": "http://schema:1234",
  "schema-registry-username": "user",
  "schema-registry-password": "very-secret"
}

• a bunch of secrets:

my-kafka-bootstrap-servers: server-a:1234,server-b:1234
my-kafka-trust-store-certificates: MY CERT 1
my-kafka-key-store-certificate-chain: MY CERT 2
my-kafka-key-store-key: MY KEY
my-kafka-schema-registry-url: http://schema:1234
my-kafka-schema-registry-username: user
my-kafka-schema-registry-password: very-secret

Additional context
This feature request came to me since I am using the ExternalSecrets operator (https://external-secrets.io/latest/) for Kubernetes clusters, and it supports that kind of stuff with the Secret Manager:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-kafka
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: my-secret-store
  target:
    name: my-kafka-secret
    creationPolicy: Owner
  data:
    - secretKey: MY_KAFKA_SCHEMA_REGISTRY_USERNAME
      remoteRef:
        key: my-kafka
        property: schema-registry-username
        version: latest

Is this a feature that could be beneficial for the project? I'd like to think so. Feel free to ask me other questions if needed!

[wilkinsona]

Thanks for the proposal, @thecanadianroot.

I'm afraid I disagree with @meltsufin here as I think this should be implemented alongside the support for sm: in Spring Cloud GCP. To Spring Boot, a value like ${sm://my-database/json/username} is almost completely opaque. All it knows is that the sm: prefix means that the remainder of the string (my-database/json/username) should be handled by Spring Cloud GCP.

We do have CloudFoundryVcapEnvironmentPostProcessor in Boot that takes a JSON payload from Cloud Foundry's VCAP_APPLICATION and VCAP_SERVICE environment variables and maps it to several properties in the environment. This logic is specific to CloudFoundry and the structure of its JSON. I would expect similar functionality that's specific to GCP's Secret Manager and how it uses JSON to be part of Spring Cloud GCP rather than Spring Boot itself.

Perhaps there's an opportunity for some lower-level helper classes in Spring Boot that can establish some conventions for extracting a single value from some JSON and assigning it to a property or mapping a whole JSON document to configuration properties but I think we'd need to see multiple implementations doing similar things before we'd be in a position where it made sense to add such functionality to Boot.

[meltsufin]

@wilkinsona Thanks for the feedback. Cloud Secret Manager doesn't have any JSON format. It simply allows storage and retrieval of secrets of type byte[]. So, support for interpreting the byte[] as JSON and extracting properties from it doesn't seem to be specific to Cloud Secret Manager. That being said, as this FR seems to be just in spring-cloud-gcp right now, we can keep it scoped to our repo.

[wilkinsona]

Thanks, @meltsufin. I'll close this one for now then. If this becomes something of a theme and we see the same requirement from other areas we can look at adding some common infrastructure to Spring Boot at that point.