Currently, extra configuration (with keys only known at runtime) is provided as follows:
writer.parquet.extra.conf.1=key1=value1
This format is not intuitive. Also the .1 in the key does not convey any information. Therefore, the extra configuration should be changed to
writer.parquet.option.key=value
To that end, Configuration.subset and Configuration.keys might be used
Unfortunately, this is inconsistent with reader.option.key=value, but it's consistent with all other configuration properties which include an identifier for the component. Arguably, reader.option.key=value should be changed to reader.kafka.option.key=value even though this results in properties like reader.kafka.option.kafka.security.protocol
Breaking changes
Configuration property naming pattern for extra configuration changes from writer.parquet.extra.conf.1=key1=value1 to writer.parquet.options.key1=value1
Currently, extra configuration (with keys only known at runtime) is provided as follows:
writer.parquet.extra.conf.1=key1=value1This format is not intuitive. Also the
.1in the key does not convey any information. Therefore, the extra configuration should be changed towriter.parquet.option.key=valueTo that end,
Configuration.subsetandConfiguration.keysmight be usedUnfortunately, this is inconsistent with
reader.option.key=value, but it's consistent with all other configuration properties which include an identifier for the component. Arguably,reader.option.key=valueshould be changed toreader.kafka.option.key=valueeven though this results in properties likereader.kafka.option.kafka.security.protocolBreaking changes
Configuration property naming pattern for extra configuration changes from
writer.parquet.extra.conf.1=key1=value1towriter.parquet.options.key1=value1