Skip creating and renaming the temp inner table of non-append Refreshable MV DDLs in Replicated DBs

[tuanpach]

…

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Skip creating and renaming the old temp table of non-append RMV DDLs in Replicated DBs.

Refreshing in non-append Refreshable Materialized Views:

1. Create a temporary table: .tmp_inner.
2. Fill in the data from the SELECT clause
3. Rename (with exchange) the temporary table to the target table name. The target table name can be the inner table of the view, or the table specified in the view create query.
4. Drop the temporary table after exchanging. Note that the temporary table after exchanging is not the table created in step 1. It is the one exchanged in step 3.
5. Update the coordination info in Keeper

Queries created in steps 1, 3, and 4 are replicated through the DDL logs.
If a replica is woken up after a long sleep, there are lots of creating, renaming and dropping DDLs generated by the RMVs.
To speed up the DDL processing, these old create DDLs are skipped.
Once the creating DDL is skipped, there is no table to rename in the renaming DDL, so the renaming DDL is skipped if the table name does not exist.
For the dropping DDLs, as the query contains IF EXISTS, we don't need to skip.

Enable this logic with DB replicated setting allow_skipping_old_temporary_tables_ddls_of_refreshable_materialized_views.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

[clickhouse-gh]

Workflow [PR], commit [8f331e0]

Summary: ❌

job_name	test_name	status	info	comment
Stateless tests (amd_tsan, sequential, 2/2)		failure
	03394_pr_insert_select_threads	FAIL
Stateless tests (amd_ubsan, parallel)		failure
	02177_issue_31009	FAIL
AST fuzzer (amd_tsan)		failure
	SUMMARY: ThreadSanitizer: data race ci/tmp/build/./src/Common/PODArray.h:155:17 in void DB::PODArrayBase<1ul, 4096ul, Allocator<false, false>, 63ul, 64ul>::realloc<>(unsigned long)	FAIL

[tuanpach]

00002_log_and_exception_messages_formatting	#84876
Logical error thrown (see clickhouse-server.log or logical_errors.txt)	#84771
Server died	#84669
00816_long_concurrent_alter_column	#84984

[tuanpach]

In the new push, I added allow_skipping_old_temporary_tables_ddls_of_refreshable_materialized_views to control the behavior. The default value is false.

[tuanpach]

• test_restore_replica/test.py::test_restore_replica_sequential
  • test_restore_replica is flaky #84847, Integration tests: OOM (ASan) #84881
• test_storage_rabbitmq
  • Flaky integration test test_storage_rabbitmq #71049
• Server died
  • Failed assert assertHasValidVersionMetadata() #83988

[antaljanosbenjamin]

Sorry for the late review.

[antaljanosbenjamin]

Let's make sure I understood these conditions correctly.
I think the first one means we should always create the table that is the current table, because that is necessary.

The second one is more cryptic IMO. Skip creating table if the DDL log was created before the last update of the coordination info? I guess this means there will be some new tables.

[tuanpach]

I added comments in the code file and updated the PR description. Please let me know if it needs to be clarified further.

[tuanpach]

• 03394_pr_insert_select_threads
  • 03394_pr_insert_select_threads is flaky #85012
• 02177_issue_31009
  • 02177_issue_31009 fails: Timeout! Killing process #85190

[antaljanosbenjamin]

But this can be fixed in another PR, you don't have to run a whole CI again.

Suggested change

	// It is possible the the temporary table is created and replciated before the the coordiation info is updated.
	// It is possible the the temporary table is created and replciated before the coordiation info is updated.

[azat]

@tuanpach new test is flaky. Should we revert?

>           assert count_skip_ddls(node2, db_name, last_log_ts) > 0
E           AssertionError: assert 0 > 0
E            +  where 0 = count_skip_ddls(<helpers.cluster.ClickHouseInstance object at 0x7fccc1f4b070>, 'test_mbd6xuJe', '2025-08-15 14:07:32.744195')

https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=85698&sha=528856cc8dba02ab2b6ecb7024ac6e23b0e4fc8a&name_0=PR&name_1=Integration%20tests%20%28amd_tsan%2C%204%2F6%29

[tuanpach]

@tuanpach new test is flaky. Should we revert?

>           assert count_skip_ddls(node2, db_name, last_log_ts) > 0
E           AssertionError: assert 0 > 0
E            +  where 0 = count_skip_ddls(<helpers.cluster.ClickHouseInstance object at 0x7fccc1f4b070>, 'test_mbd6xuJe', '2025-08-15 14:07:32.744195')

https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=85698&sha=528856cc8dba02ab2b6ecb7024ac6e23b0e4fc8a&name_0=PR&name_1=Integration%20tests%20%28amd_tsan%2C%204%2F6%29

Let me check it first.

[tuanpach]

In the test, the DB is recovered, so no DDLs to skip. I will update the test to make it stable.