feat(clickhouse sink): add query_settings to clickhouse sink

[pm5]

Summary

It PR adds a query_settings field in the configuration of clickhouse sink, with a few configurations for asynchronous inserts.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

I use the following Vector config to test it:

sources:
  demo_logs:
    type: demo_logs
    format: syslog
    count: 10
    interval: 0.0

sinks:
  clickhouse:
    inputs: ["demo_logs"]
    type: clickhouse
    endpoint: http://localhost:8123/
    database: default
    table: demo_logs
    skip_unknown_fields: true
    auth:
      user: default
      password: ''
      strategy: basic
    query_settings:
      async_insert: true
      wait_for_async_insert: true
      async_insert_max_data_size: 1000000
      async_insert_max_query_number: 1000000

Start ClickHouse, enable asynchronous logs, create a demo_logs table:

CREATE TABLE default.demo_logs
(
    `host` String,
    `message` String,
    `service` String,
    `source_type` String,
    `timestamp` String
)
ENGINE = MergeTree
ORDER BY timestamp

Run Vector. And then check ClickHouse logs for asynchronous inserts:

select now(), * from system.asynchronous_insert_log order by event_time format Vertical

You should see the queries done by Vector.

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

• Please read our Vector contributor resources.
  • make check-all is a good command to run locally. This check is
    defined here. Some of these
    checks might not be relevant to your PR. For Rust changes, at the very least you should run:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run dd-rust-license-tool write to regenerate the license inventory and commit the changes (if any). More details here.

References

• Closes: Feature request: ClickHouse query settings #22373
• About asynchronous inserts: https://clickhouse.com/docs/cloud/bestpractices/asynchronous-inserts
• If asynchronous insert is not used even when it is enabled in the settings, follow the steps in Insert query not using async_insert ClickHouse/ClickHouse#49308 to enable debug logs to see the reason.

[bits-bot]

All committers have signed the CLA.

[pm5]

Hi @pront ! Thanks for updating the info about this PR. Is there anything I can add to help with the reviewing?

[pront]

Thank you @pm5, this looks good overall. Left a small comment.

[pront]

All the fields are related to https://clickhouse.com/docs/cloud/bestpractices/asynchronous-inserts. Maybe we can rename to async_insert_setttings and add a link to the docs.

[pm5]

There is #22373 which suggests moving the three existing query settings into a query_settings option, so this was implemented with that in mind. Maybe we can instead move the existing query settings into query_settings as proposed in the future.

Also, not all of ClickHouse settings make sense in the context of Vector, but maybe things like insert_deduplicate do make sense. We can also add these into the query_settings option when there is need for them.

[pm5]

We can for sure add a link to the doc.

[pm5]

I added the links above on the query_settings option.

[pront]

Basically all these settings have the same async_insert prefix (or suffix), that's why I was thinking we might be able to avoid repeating it. But it's a minor UX concern.

[pm5]

Personally I prefer keeping the original ClickHouse option names for now. There are just a lot of them and a lot of work if we want to bring additional organisation into them. So if you don't mind, I'm closing this thread.

[pront]

Personally I prefer keeping the original ClickHouse option names for now.

This is necessary for backwards compatibility. I was referring only to new fields. For example we now have:

query_settings:
  async_insert: true
  wait_for_async_insert: true
  async_insert_deduplicate: true

An alternative is:

query_settings:
  async_insert:
    wait_for_async_insert: true
    async_insert_deduplicate: true

I also realized there's a nuance with async_insert being omitted and other fields like wait_for_async_insert being present in the config.

[pm5]

I'm not sure how this is better :) Isn't the alternative duplicating the async keyword in the subcategory name and the config names?

[pm5]

Thanks @pront . Let me know what you think of the current version.

[pm5]

I found a problem with bool type options and fixed it. Also asked clippy to allow more arguments on set_uri_query.

[pront]

I found a problem with bool type options and fixed it. Also asked clippy to allow more arguments on set_uri_query.

Please do not force push because I have to review the whole PR every time 😅