Scheme shard support for cdc stream per index #4650

dcherednik · 2024-05-17T12:58:15Z

Changelog category

Improvement

Additional information

...

github-actions · 2024-05-17T13:01:33Z

⚪ 2024-05-17 13:01:33 UTC Pre-commit check for a8afd05 has started.
⚪ 2024-05-17 13:01:35 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-17 13:40:36 UTC Build successful.
⚪ 2024-05-17 13:42:24 UTC Tests are running...
🔴 2024-05-17 15:19:33 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38057	33224	0	17	4807	9

github-actions · 2024-05-17T13:01:37Z

⚪ 2024-05-17 13:01:37 UTC Pre-commit check for a8afd05 has started.
⚪ 2024-05-17 13:01:39 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-17 13:42:05 UTC Build successful.
⚪ 2024-05-17 13:43:54 UTC Tests are running...
🔴 2024-05-17 15:41:22 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10673	10498	0	39	114	22

github-actions · 2024-05-17T13:01:41Z

⚪ 2024-05-17 13:01:41 UTC Pre-commit check for a8afd05 has started.
⚪ 2024-05-17 13:01:44 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-17 13:39:12 UTC Build successful.

github-actions · 2024-05-17T16:49:53Z

⚪ 2024-05-17 16:49:53 UTC Pre-commit check for 5edbfcd has started.
⚪ 2024-05-17 16:49:56 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-17 16:51:21 UTC Build successful.
⚪ 2024-05-17 16:53:06 UTC Tests are running...
🔴 2024-05-17 18:27:32 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38058	33236	0	16	4794	12

github-actions · 2024-05-17T16:49:55Z

⚪ 2024-05-17 16:49:54 UTC Pre-commit check for 5edbfcd has started.
⚪ 2024-05-17 16:49:57 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-17 16:51:21 UTC Build successful.

github-actions · 2024-05-17T17:01:31Z

⚪ 2024-05-17 17:01:31 UTC Pre-commit check for 5edbfcd has started.
⚪ 2024-05-17 17:01:33 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-17 17:03:03 UTC Build successful.
⚪ 2024-05-17 17:04:51 UTC Tests are running...
🔴 2024-05-17 19:17:54 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10671	10551	0	35	71	14

ydb/core/tx/schemeshard/schemeshard__operation_create_cdc_stream.cpp

CyberROFL · 2024-05-17T17:36:07Z

ydb/core/protos/flat_scheme_op.proto

@@ -827,6 +827,10 @@ message TCreateCdcStream {
    optional TCdcStreamDescription StreamDescription = 2;
    optional uint64 RetentionPeriodSeconds = 3 [default = 86400]; // 1d by default
    optional uint32 TopicPartitions = 4;
+    oneof IndexMode {
+        bool AllIndexes = 5; // Create topic per each index


Есть смысл передавать AllIndexes = false? Чем это отличается от IndexMode == NOT_SET?

Ничем, но просто поле enum нельзя положить в oneof, или как ты предлагаешь?

Предлагаю google.protobuf.empty AllIndexes, если достаточно (и всегда будет) самого факта. Или хотя бы TAllIndexes AllIndexes, где TAllIndexes — это пустое сообщение (если его предполагается когда-нибудь расширять).

CyberROFL · 2024-05-17T17:36:45Z

ydb/core/protos/flat_scheme_op.proto

@@ -827,6 +827,10 @@ message TCreateCdcStream {
    optional TCdcStreamDescription StreamDescription = 2;
    optional uint64 RetentionPeriodSeconds = 3 [default = 86400]; // 1d by default
    optional uint32 TopicPartitions = 4;
+    oneof IndexMode {
+        bool AllIndexes = 5; // Create topic per each index
+        string IndexName = 6;


Не очень понял, для чего нужен этот режим?

Ну мы можем или создать стримы для всех индексов, или только для определенных. Кроме того нам надо передать TCreateCdcStream в сабоперацию и удобно иметь тут же информацию об индексе для которой создана сабоперация. Есть идеи как это выразить лучше?

Какой сценарий создания стрима только для определенного (даже не списка) индекса?

Ну к примеру у тебя часть индексов используется для местной аналитики и не нужна на реплике, а часть для OLTP запросов. Или же наоборот. Список наверно лучше тут.

ydb/core/tx/schemeshard/schemeshard__operation_create_cdc_stream.cpp

dcherednik

Обновил pr и оставил комментарии.

github-actions · 2024-05-21T12:17:34Z

⚪ 2024-05-21 12:17:34 UTC Pre-commit check for 535cd94 has started.
⚪ 2024-05-21 12:17:35 UTC Build linux-x86_64-relwithdebinfo is running...
⚫ 2024-05-21 12:54:06 UTC Check cancelled

github-actions · 2024-05-21T12:18:16Z

⚪ 2024-05-21 12:18:16 UTC Pre-commit check for 535cd94 has started.
⚪ 2024-05-21 12:18:17 UTC Build linux-x86_64-release-clang14 is running...
⚫ 2024-05-21 12:54:07 UTC Check cancelled

github-actions · 2024-05-21T12:18:34Z

⚪ 2024-05-21 12:18:34 UTC Pre-commit check for 535cd94 has started.
⚪ 2024-05-21 12:18:37 UTC Build linux-x86_64-release-asan is running...
⚫ 2024-05-21 12:54:22 UTC Check cancelled

github-actions · 2024-05-21T12:55:04Z

⚪ 2024-05-21 12:55:04 UTC Pre-commit check for d236d96 has started.
⚪ 2024-05-21 12:55:05 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-21 13:04:02 UTC Build successful.
⚪ 2024-05-21 13:05:43 UTC Tests are running...
🔴 2024-05-21 15:06:06 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10674	10595	0	26	37	16

github-actions · 2024-05-21T12:57:01Z

⚪ 2024-05-21 12:57:00 UTC Pre-commit check for d236d96 has started.
⚪ 2024-05-21 12:57:02 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-21 13:04:21 UTC Build successful.

github-actions · 2024-05-21T13:03:54Z

⚪ 2024-05-21 13:03:54 UTC Pre-commit check for d236d96 has started.
⚪ 2024-05-21 13:03:57 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-21 13:11:42 UTC Build successful.
⚪ 2024-05-21 13:13:31 UTC Tests are running...
🔴 2024-05-21 14:46:56 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38123	33331	0	1	4785	6

github-actions · 2024-05-22T13:57:01Z

⚪ 2024-05-22 13:57:01 UTC Pre-commit check for ab225b5 has started.
⚪ 2024-05-22 13:57:02 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-22 14:37:57 UTC Build successful.
⚪ 2024-05-22 14:39:30 UTC Tests are running...
🔴 2024-05-22 16:32:38 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
10668	10613	0	20	24	11

github-actions · 2024-05-22T13:58:29Z

⚪ 2024-05-22 13:58:29 UTC Pre-commit check for ab225b5 has started.
⚪ 2024-05-22 13:58:32 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-22 14:33:20 UTC Build successful.

github-actions · 2024-05-22T13:59:29Z

⚪ 2024-05-22 13:59:29 UTC Pre-commit check for ab225b5 has started.
⚪ 2024-05-22 13:59:30 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-22 14:41:45 UTC Build successful.
⚪ 2024-05-22 14:43:22 UTC Tests are running...
🔴 2024-05-22 16:24:37 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38132	33335	0	1	4785	11

github-actions bot added the improvement label May 17, 2024

dcherednik added the rebase-and-check Rebase PR with the current main and check label May 17, 2024

github-actions bot removed the rebase-and-check Rebase PR with the current main and check label May 17, 2024

CyberROFL reviewed May 17, 2024

View reviewed changes

dcherednik force-pushed the cdc_stream_table_with_index_2_with_cancelable_pq_3 branch from 7e6b712 to 924a023 Compare May 21, 2024 11:28

dcherednik added 3 commits May 21, 2024 11:44

Scheme shard support for cdc stream per index

82781e8

Clean the code

3dbdd98

fix

4af1121

dcherednik force-pushed the cdc_stream_table_with_index_2_with_cancelable_pq_3 branch from 924a023 to 4af1121 Compare May 21, 2024 11:51

dcherednik commented May 21, 2024

View reviewed changes

fix

2d63f85

Use empty message instead of bool

3b1f52c

CyberROFL approved these changes May 22, 2024

View reviewed changes

dcherednik merged commit 636451e into ydb-platform:main May 22, 2024
3 of 5 checks passed

serbel324 mentioned this pull request May 24, 2024

Add description of new ICB controls for CMS #4818

Merged

This was referenced May 27, 2024

Cli changes for autoscaling #4871

Closed

Fix getting mid of range for autoscaling #4884

Closed

Fix getting mid range #4885

Merged

MrLolthe1st pushed a commit to MrLolthe1st/ydb that referenced this pull request May 28, 2024

Scheme shard support for cdc stream per index (ydb-platform#4650)

b8ac6ee

niksaveliev mentioned this pull request May 29, 2024

Autoscaling logs #4976

Merged

This was referenced May 29, 2024

YQ-3292 fixed kqprun ResourceBrokerConfig passing #4994

Merged

Added ExecutionId for script execution logs #5014

Merged

This was referenced May 30, 2024

Forbid autoscaling with size retention #5054

Merged

Mute split by load ut #5264

Closed

Mute split by load ut #5267

Closed

Mute split by load ut #5272

Merged

This was referenced Jun 7, 2024

Fix build #5310

Merged

Fix split by load ut #5383

Merged

niksaveliev mentioned this pull request Jun 17, 2024

Fix win build #5622

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scheme shard support for cdc stream per index #4650

Scheme shard support for cdc stream per index #4650

dcherednik commented May 17, 2024

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

CyberROFL May 17, 2024

dcherednik May 17, 2024

CyberROFL May 21, 2024 •

edited

Loading

CyberROFL May 17, 2024

dcherednik May 17, 2024

CyberROFL May 22, 2024

dcherednik May 22, 2024

dcherednik left a comment

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 22, 2024 •

edited

Loading

github-actions bot commented May 22, 2024 •

edited

Loading

github-actions bot commented May 22, 2024 •

edited

Loading

Scheme shard support for cdc stream per index #4650

Scheme shard support for cdc stream per index #4650

Conversation

dcherednik commented May 17, 2024

Changelog category

Additional information

github-actions bot commented May 17, 2024 • edited Loading

github-actions bot commented May 17, 2024 • edited Loading

github-actions bot commented May 17, 2024 • edited Loading

github-actions bot commented May 17, 2024 • edited Loading

github-actions bot commented May 17, 2024 • edited Loading

github-actions bot commented May 17, 2024 • edited Loading

CyberROFL May 17, 2024

Choose a reason for hiding this comment

dcherednik May 17, 2024

Choose a reason for hiding this comment

CyberROFL May 21, 2024 • edited Loading

Choose a reason for hiding this comment

CyberROFL May 17, 2024

Choose a reason for hiding this comment

dcherednik May 17, 2024

Choose a reason for hiding this comment

CyberROFL May 22, 2024

Choose a reason for hiding this comment

dcherednik May 22, 2024

Choose a reason for hiding this comment

dcherednik left a comment

Choose a reason for hiding this comment

github-actions bot commented May 21, 2024 • edited Loading

github-actions bot commented May 21, 2024 • edited Loading

github-actions bot commented May 21, 2024 • edited Loading

github-actions bot commented May 21, 2024 • edited Loading

github-actions bot commented May 21, 2024 • edited Loading

github-actions bot commented May 21, 2024 • edited Loading

github-actions bot commented May 22, 2024 • edited Loading

github-actions bot commented May 22, 2024 • edited Loading

github-actions bot commented May 22, 2024 • edited Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

github-actions bot commented May 17, 2024 •

edited

Loading

CyberROFL May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 21, 2024 •

edited

Loading

github-actions bot commented May 22, 2024 •

edited

Loading

github-actions bot commented May 22, 2024 •

edited

Loading

github-actions bot commented May 22, 2024 •

edited

Loading