Skip to content

Don't use multiple = characters to encode extra configuration #54

@kevinwallimann

Description

@kevinwallimann

Currently, extra configuration (with keys only known at runtime) is provided as follows:
writer.parquet.extra.conf.1=key1=value1

This format is not intuitive. Also the .1 in the key does not convey any information. Therefore, the extra configuration should be changed to
writer.parquet.option.key=value

To that end, Configuration.subset and Configuration.keys might be used

Unfortunately, this is inconsistent with reader.option.key=value, but it's consistent with all other configuration properties which include an identifier for the component. Arguably, reader.option.key=value should be changed to reader.kafka.option.key=value even though this results in properties like reader.kafka.option.kafka.security.protocol

Breaking changes
Configuration property naming pattern for extra configuration changes from writer.parquet.extra.conf.1=key1=value1 to writer.parquet.options.key1=value1

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions