Summary
The ClickHouse sink's query_settings block currently only exposes 6 hardcoded async insert settings, all passed as HTTP URL query parameters:
[sinks.my_sink.query_settings.async_insert_settings]
enabled = true
wait_for_processing = true
wait_for_processing_timeout = 10
deduplicate = false
max_data_size = 10000
max_query_number = 100
ClickHouse supports hundreds of query-level settings (e.g. deduplicate_blocks_in_dependent_materialized_views, insert_quorum, max_insert_threads, input_format_skip_unknown_fields, max_insert_block_size, etc.) that are passed as HTTP query parameters. There is currently no way to set any of these from the Vector ClickHouse sink.
Proposed solution
Add an extra_query_params map to the ClickHouse sink config that forwards arbitrary key-value pairs as HTTP query parameters on every insert request:
[sinks.my_sink.extra_query_params]
deduplicate_blocks_in_dependent_materialized_views = "0"
insert_quorum = "2"
max_insert_block_size = "1048576"
These would be appended to the request URL alongside the existing async insert params.
Use case
deduplicate_blocks_in_dependent_materialized_views = 0 — disable deduplication for dependent materialized views when Vector handles delivery guarantees itself (at-least-once with idempotent writes)insert_quorum — enforce write quorum on replicated ClickHouse clustersmax_insert_threads — control parallelism of insert processing on the server sideinput_format_skip_unknown_fields — tolerate schema evolution without insert failures
Without this, the only alternative is setting these in the ClickHouse user profile for the Vector user, which is less flexible and couples ClickHouse server config to Vector behavior.
Implementation note
The existing set_uri_query in src/sinks/clickhouse/service.rs already appends params individually. An extra_query_params: HashMap<String, String> in QuerySettingsConfig (or as a top-level sink field) would iterate and append each entry the same way.
Summary
The ClickHouse sink's
query_settingsblock currently only exposes 6 hardcoded async insert settings, all passed as HTTP URL query parameters:ClickHouse supports hundreds of query-level settings (e.g.
deduplicate_blocks_in_dependent_materialized_views,insert_quorum,max_insert_threads,input_format_skip_unknown_fields,max_insert_block_size, etc.) that are passed as HTTP query parameters. There is currently no way to set any of these from the Vector ClickHouse sink.Proposed solution
Add an
extra_query_paramsmap to the ClickHouse sink config that forwards arbitrary key-value pairs as HTTP query parameters on every insert request:These would be appended to the request URL alongside the existing async insert params.
Use case
deduplicate_blocks_in_dependent_materialized_views = 0— disable deduplication for dependent materialized views when Vector handles delivery guarantees itself (at-least-once with idempotent writes)insert_quorum— enforce write quorum on replicated ClickHouse clustersmax_insert_threads— control parallelism of insert processing on the server sideinput_format_skip_unknown_fields— tolerate schema evolution without insert failuresWithout this, the only alternative is setting these in the ClickHouse user profile for the Vector user, which is less flexible and couples ClickHouse server config to Vector behavior.
Implementation note
The existing
set_uri_queryinsrc/sinks/clickhouse/service.rsalready appends params individually. Anextra_query_params: HashMap<String, String>inQuerySettingsConfig(or as a top-level sink field) would iterate and append each entry the same way.