[feat][fn] Add option to merge secrets into config for sink/source

[michaeljmarshall]

Fixes #20862

Motivation

See the issue for detailed motivation. Briefly, the current solution for configuring sinks and sources requires each sink and source to implement calls to materialize secrets using the SecretsProvider. Instead of relying on each connector to implement this feature, this PR introduces an option to pass the configuration into the sink and source open method call, so that the secrets are available without needing them to be passed in as plain text.

One major benefit of this solution is that it integrates with the existing functionality to pass in secrets.

In order to be backwards compatible, I followed the approach used by #20116 to add a configuration to the function worker that will apply to all sinks/sources created by the function worker.

Example for old configuration and new configuration:

 bin/pulsar-admin sinks create --input-specs "spec with secret"

bin/pulsar-admin sinks create --input-specs "spec without secret" --secrets "secret reference"

Modifications

• Add the option to merge a connector's secrets map into the config map that is subsequently passed to the sink or source when calling open.
• Add a new configuration to the function worker (mergeSecretsIntoConfigMap) and an associated new parameter to the JavaInstanceStarter (--merge_secrets_into_config_map) to enable this feature. The default value keeps the current behavior.

Verifying this change

This change added tests.

Does this pull request potentially affect one of the following parts:

Adds a new configuration option. It is backwards compatible and defaults to use the old behavior.

Documentation

• doc-required

I'll need a PR to add docs for this feature.

Matching PR in forked repository

PR in forked repository: michaeljmarshall#53

[eolivelli]

I have one question: how does this work with structured configurations ? A value in the configuration may be a Map, is it possible to update entries inside a nested map?

[eolivelli]

Why this doesn't apply to Functions?

[michaeljmarshall]

It could apply to functions. I targeted Sources and Sinks because those are often cases where you are running a third party connector and in those cases, it is easier to merge the configs than updating the code. In the function runtime, it seems to be a well defined flow to get a secret from the function's context. That being said, we could also just fix each implementation.

I provide some direct examples of issues with connectors in this comment #20862 (comment).

[michaeljmarshall]

how does this work with structured configurations ? A value in the configuration may be a Map, is it possible to update entries inside a nested map?

The current logic is a naive putIfAbsent. Is it common to have nested maps? Would it always be valid to insert a sub-map?

[michaeljmarshall]

@eolivelli - looks like recursive secret interpolation would be valuable here:

pulsar/pulsar-io/kafka-connect-adaptor/src/main/java/org/apache/pulsar/io/kafka/connect/PulsarKafkaConnectSinkConfig.java

Lines 59 to 62 in 55523ac

	@FieldDoc(
	defaultValue = "",
	help = "Config properties to pass to the kafka connector.")
	private Map<String, String> kafkaConnectorConfigProperties;

[michaeljmarshall]

This will be superseded by the outcome of PIP 289. My preference is to use #20901.