Add enable_add_distinct_to_in_subqueries setting to optimize distributed IN subqueries.

[fhw12345]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

A new setting, enable_add_distinct_to_in_subqueries, has been introduced. When enabled, ClickHouse will automatically add DISTINCT to subqueries in IN clauses for distributed queries. This can significantly reduce the size of temporary tables transferred between shards and improve network efficiency.
Note: This is a trade-off—while network transfer is reduced, additional merging (deduplication) work is required on each node. Enable this setting when network transfer is a bottleneck and the merging cost is acceptable.

[clickhouse-gh]

Workflow [PR], commit [b68aebb]

[clickhouse-gh]

Workflow [PR], commit [a999f29]

Summary: ❌

job_name	test_name	status	info	comment
Stress test (amd_tsan)		failure
	Server died	FAIL
	Hung check failed, possible deadlock found (see hung_check.log)	FAIL
	Killed by signal (in clickhouse-server.log)	FAIL
	Fatal message in clickhouse-server.log (see fatal_messages.txt)	FAIL
	Killed by signal (output files)	FAIL
	Found signal in gdb.log	FAIL

[fhw12345]

Hi @novikd could u help have a look at this PR, thanks

[novikd]

Overall the idea of the optimization is good, but it should be implemented in another place. Please, also provide some performance test or a performance improvement mesures.

[fhw12345]

Hi @novikd, just following up to kindly request a review when you have a moment, thanks.

[novikd]

LGTM

[fhw12345]

LGTM

@novikd I've updated the code following your comments, please have a check when you have a moment. Thanks!

[novikd]

It's strange that the perf tests don't run the new test.

[fhw12345]

It's strange that the perf tests don't run the new test.

Do I need to make any other changes to trigger the perf test for the new test?

[fhw12345]

It's strange that the perf tests don't run the new test.

And this is the perf test result in local:

[novikd]

LGTM

[fhw12345]

@novikd could you help merge the PR thanks!

[novikd]

PR / Stress test (amd_tsan) (pull_request)Failing after 19m

#81144