send query events from the SELECT part of CREATE OR REPLACE TABLE query

[Diskein]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

You can see a progress bar, logs and performance statistics for CREATE OR REPLACE TABLE queries with SELECT in clickhouse-client. This query will no longer lead to timeout even if the SELECT is quite time consuming. Resolves #38416

Details

The SELECT section might take a lot of time, so let's:

• Send progress, logs, and stats for this query
• Avoid firing TCP session receive timeouts
• Make this query cancellable

Resolves #38416

[Diskein]

The PR resolves #38416

[clickhouse-gh]

Workflow [PR], commit [2b31be1]

Summary: ❌

job_name	test_name	status	info	comment
Stateless tests (amd_ubsan, parallel)		failure
	02177_issue_31009	FAIL	cidb, flaky
Integration tests (amd_asan, old analyzer, 3/6)		failure
	test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]	FAIL	cidb, flaky
Integration tests (amd_binary, 5/5)		failure
	test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]	FAIL	cidb, flaky
Integration tests (arm_binary, distributed plan, 4/4)		failure
	test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]	FAIL	cidb, flaky
Integration tests (amd_tsan, 3/6)		failure
	test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header]	FAIL	cidb, flaky

[tuanpach]

There is an error:

/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/src/Interpreters/executeQuery.h:89:6: error: function 'DB::executeTrivialBlockIO' has a definition with different parameter names [readability-inconsistent-declaration-parameter-name,-warnings-as-errors]
   89 | void executeTrivialBlockIO(BlockIO & streams, ContextPtr context, bool with_interactve_cancel = false);
      |      ^
/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/src/Interpreters/executeQuery.cpp:2162:6: note: the definition seen here
 2162 | void executeTrivialBlockIO(BlockIO & streams, ContextPtr context, bool with_interactive_cancel)
      |      ^
/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/src/Interpreters/executeQuery.h:89:6: note: differing parameters are named here: ('with_interactve_cancel'), in definition: ('with_interactive_cancel')
   89 | void executeTrivialBlockIO(BlockIO & streams, ContextPtr context, bool with_interactve_cancel = false);
      |      ^                                                                 ~~~~~~~~~~~~~~~~~~~~~~
      |                                                                        with_interactive_cancel

[Diskein]

test_storage_delta/test.py::test_replicated_database_and_unavailable_s3 is broken test on master see
test_mask_sensitive_info/test.py::test_table_functions - is known to be flaky - #87544

[Diskein]

Thank you @tuanpach for updating the branch!

The failures aren't related to the changes:

• Logical error: 'Invalid number of columns in chunk pushed to OutputPort. Expected 5, found 12 - probably related to Logical Error: Invalid number of columns in chunk pushed to OutputPort with loop function #83757
• 03602_alter_update_nullable_json is flaky - it happened in an another PR - Add more tests for clickhouse-disks with plain_rewritable #86780
• test_multiple_disks/test.py::test_kill_while_insert is flaky - test_multiple_disks/test.py::test_kill_while_insert is flaky #87141

[tuanpach]

test_s3_access_headers

• Failure in test_s3_access_headers/test.py::test_custom_access_header[test_access_over_custom_header] #89758

02177_issue_31009

• Test 02177_issue_31009 is flaky #86335

[tuanpach]

@Diskein I merged master into your branch as there were some conflicts.

Could you help double-check?

[Diskein]

@Diskein I merged master into your branch as there were some conflicts.

Could you help double-check?

Yep, it looks good to me.

I've seen your changes in which you replaced an optional to a shared_ptr and used weak_ptrs for checking if callbacks use a valid query state. I'm a little bit worried about other callbacks in TCPHandler too. it's maybe just a matter of time when somebody reuses them or/and their query context.

[tuanpach]

t's maybe just a matter of time when somebody reuses them or/and their query context.

You are right. I will have a separate PR to fix other callbacks.

[tavplubix]

@Diskein the new test is flaky: https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=7196ad8c74ef26ac3eb611138141a75481127d34&name_0=MasterCI&name_1=Stateless%20tests%20%28amd_tsan%2C%20s3%20storage%2C%20parallel%29&name_1=Stateless%20tests%20%28amd_tsan%2C%20s3%20storage%2C%20parallel%29