Connector secrets

[snkas]

Overview:

• Secret reference: add enumeration type SimpleStringOrSecretReference with constructor that takes in a string. If the string follows the ${secret:example} pattern, it returns a secret reference to example. Else, it returns a simple
  string with the direct content. Its pattern is similar to that of Kafka Connect / Debezium.

• Secret resolver: converts a secret reference to the secret string it refers to.

  • Resolution options: currently, only secret references named "^[a-z0-9]+(-[a-z0-9]+)*$" are allowed, and are resolved by checking if a file "/etc/secret-volume/.example" exists (e.g., for the secret named example). This file needs to be made available by mounting the secret volume in advance.

  • Resolution result is a SimpleOrSecretString instead of just a String, such that if it is formatted for logs, it does not show the content for the secret variant. Of course, this only lasts until it must be passed as a plain String to the underlying connector library (e.g., librdkafka).

• Both conversion to potential secret reference (to SimpleStringOrSecretReference) and resolution (to SimpleOrSecretString) are currently done at the latest possible point (latest resolution). They will be decoupled if secret references are directly integrated into the API (see future directions).

• Current implementation: secrets through pattern are enabled only for the additional config properties of the Kafka input and Kafka output connector.

Possible future directions:

• Integration of secret references directly into the API (i.e., using a dedicated type to which is deserialized, like SimpleStringOrSecretReference (already added deserialization for it)). This would require changes at the core API, the Python API wrapper and the web console, as they would need to pass an object other than a basic string. It would remove the need for a secret pattern (although this could still be provided as a UI input method).

• Allow secrets to be specified in Docker demo via adding mounted volume at /etc/secret-volume of the pipeline manager.

• Additional resolution sources: external providers, maybe environment variables

• Only at the pipeline the secret reference is resolved. There could already be earlier checks in place to warn if the secret reference does not exist (e.g., in the UI).

Pattern matching considerations:

• A string must follow the ${secret:example} pattern to be parsed as a secret reference. As such, it is impossible to have strings that follow the pattern.

• Because of this, there is one scenario in which the secret pattern matching would have an undesirable effect:

  1. The user has a secret named example, intended for an input field for connector A
  2. The user wants to enter the literal string ${secret:example} for an input field for connector B
  3. Rather than the literal string ${secret:example} being sent, instead the secret string
     behind the example secret is sent to connector B

• This is highly unlikely as (1) it is exceedingly unlikely a user wants to enter a string following the pattern literally, and (2) the user must have already created a secret with the exact internal name and must match it exactly in the literal string. Because of (2), the user must already be aware of secrets and their pattern syntax (having created a secret), and as such the user wanting to enter a literal string like in (1) is improbable.

Motivation why it is not done with Yaml transformation:

• Technically, we could also before deserializing the config YamlValue replace any Yaml/Json-specific construct (e.g., !secret_ref: "example) or any pattern-following Yaml-String with the Yaml-String containing the resolved secret. This would make the connector oblivious to whether it is a secret or not. However, this would make it format in any potential future logs.

• The Yaml replacement could also be a SimpleStringOrSecretRef, but if the replacement is scoped to the entire YamlValue it would require changing the type of all strings. Scoping it only to a subset of the YamlValue (e.g., only the ones under the kafka_options key of the top mapping) would be possible, but would yield no difference in the current pattern-matching implementation and would need the Yaml conversion to become config-aware rather than just directly deserializing.

• It would also just be an inbetween solution, as an even better way is the integration of secret references directly into the API (see above), which would make them directly deserialize from Yaml without config-awareness.

Is this a user-visible change (yes/no): yes

[snkas]

( Note: there is not yet an entry in the CHANGELOG.md )

[snkas]

Rebased with main, added CHANGELOG note, and incorporated feedback (see feedback comments for more details).

[snkas]

( Rebased again with main )