YQ-4750 supported solomon writing in KQP #26213

GrigoriyPA · 2025-10-02T08:57:23Z

Changelog entry

Supported solomon insert into solomon in query service and streaming queries

Changelog category

New feature

Description for reviewers

github-actions · 2025-10-02T09:00:33Z

🟢 2025-10-02 09:00:33 UTC The validation of the Pull Request description is successful.

github-actions · 2025-10-02T09:00:38Z

⚪ 2025-10-02 09:00:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for bff1bb2 has started.
⚪ 2025-10-02 09:00:52 UTC Artifacts will be uploaded here
⚫ 2025-10-02 09:02:54 UTC Check cancelled

github-actions · 2025-10-02T09:01:05Z

⚪ 2025-10-02 09:01:04 UTC Pre-commit check linux-x86_64-release-asan for bff1bb2 has started.
⚪ 2025-10-02 09:01:18 UTC Artifacts will be uploaded here
⚫ 2025-10-02 09:02:54 UTC Check cancelled

github-actions · 2025-10-02T09:04:31Z

⚪ 2025-10-02 09:04:31 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7150dd3 has started.
⚪ 2025-10-02 09:04:44 UTC Artifacts will be uploaded here
⚫ 2025-10-02 09:05:29 UTC Check cancelled

Copilot

Pull Request Overview

Enables writing metrics to Solomon from KQP (both query service and streaming queries), introducing a new SoInsert path and wiring it through DQ integration and compute actor.

Add TSoInsert node and associated type annotation and physical optimization to build Solomon sink stages.
Gate legacy SoWriteToShard path vs. new DQ-integrated write via TSolomonState::WriteThroughDqIntegration; wire up Solomon write actor in KQP; extend tests and test utilities.
Improve error messages and add Solomon emulator helpers for tests; add a streaming query UT that writes into Solomon.

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
ydb/tests/tools/kqprun/configuration/app_config.conf	Enables schema secrets (feature flag), supporting credentials use for external sources/sinks.
ydb/library/yql/providers/solomon/solomon_accessor/client/solomon_accessor_client.cpp	Improves error diagnostics with curl/http status details.
ydb/library/yql/providers/solomon/provider/yql_solomon_provider.h	Adds WriteThroughDqIntegration flag to toggle write path.
ydb/library/yql/providers/solomon/provider/yql_solomon_physical_optimize.cpp	Introduces BuildSinkStage, adds SoInsert handler, and conditions legacy path.
ydb/library/yql/providers/solomon/provider/yql_solomon_dq_integration.cpp	Wraps legacy writes into SoInsert when DQ integration write is enabled.
ydb/library/yql/providers/solomon/provider/yql_solomon_datasink_type_ann.cpp	Adds type annotation for SoInsert, factors out validation, adjusts SoShard handling.
ydb/library/yql/providers/solomon/expr_nodes/yql_solomon_expr_nodes.json	Declares new TSoInsert callable.
ydb/library/yql/providers/solomon/actors/ut/ya.make	Adds test library dependencies and helpers.
ydb/library/yql/providers/solomon/actors/ut/ut_helpers.[h,cpp]	Removes local Solomon emulator helpers (moved to shared testlib).
ydb/library/yql/providers/solomon/actors/ut/dq_solomon_write_actor_ut.cpp	Switches to shared helpers and strengthens assertions; enables big batch test.
ydb/library/testlib/ya.make, solomon_helpers/*	Adds shared Solomon emulator helpers for tests.
ydb/core/kqp/ut/federated_query/datastreams/*	Adds recipe include and a streaming query UT that writes into Solomon.
ydb/core/kqp/ut/federated_query/common/common.cpp	Wires CreateSolomonGateway to the federated query setup.
ydb/core/kqp/opt/kqp_opt_effects.cpp	Tightens invariant checks for external effects.
ydb/core/kqp/host/kqp_host.cpp	Enables DQ write path for Solomon in KQP host.
ydb/core/kqp/compute_actor/kqp_compute_actor.cpp	Registers Solomon write async IO factory for compute actor.
ydb/core/kqp/gateway/behaviour/streaming_query/optimization.cpp	Recognizes Solomon as a streaming write sink without increasing PQ write count.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

ydb/library/yql/providers/solomon/provider/yql_solomon_physical_optimize.cpp

ydb/library/yql/providers/solomon/provider/yql_solomon_datasink_type_ann.cpp

github-actions · 2025-10-02T09:05:09Z

⚪ 2025-10-02 09:05:09 UTC Pre-commit check linux-x86_64-release-asan for 7150dd3 has started.
⚫ 2025-10-02 09:05:25 UTC Check cancelled

github-actions · 2025-10-02T09:06:57Z

⚪ 2025-10-02 09:06:56 UTC Pre-commit check linux-x86_64-release-asan for b40aae3 has started.
⚪ 2025-10-02 09:07:21 UTC Artifacts will be uploaded here
⚫ 2025-10-02 09:10:53 UTC Check cancelled

github-actions · 2025-10-02T09:07:05Z

⚪ 2025-10-02 09:07:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for b40aae3 has started.
⚪ 2025-10-02 09:07:19 UTC Artifacts will be uploaded here
⚫ 2025-10-02 09:10:53 UTC Check cancelled

github-actions · 2025-10-02T09:12:22Z

⚪ 2025-10-02 09:12:22 UTC Pre-commit check linux-x86_64-release-asan for abdcea4 has started.
⚪ 2025-10-02 09:12:36 UTC Artifacts will be uploaded here
⚪ 2025-10-02 09:16:08 UTC ya make is running...
🟡 2025-10-02 11:14:54 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15678	15237	0	153	264	24

🟢 2025-10-02 11:14:59 UTC Build successful.
🟡 2025-10-02 11:15:20 UTC ydbd size 3.7 GiB changed* by +227.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `b1e3f5a`	merge: `abdcea4`	diff	diff %
ydbd size	3 991 216 344 Bytes	3 991 449 320 Bytes	+227.5 KiB	+0.006%
ydbd stripped size	1 487 784 800 Bytes	1 487 864 352 Bytes	+77.7 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-10-02T09:12:24Z

⚪ 2025-10-02 09:12:23 UTC Pre-commit check linux-x86_64-relwithdebinfo for abdcea4 has started.
⚪ 2025-10-02 09:12:39 UTC Artifacts will be uploaded here
⚪ 2025-10-02 09:16:21 UTC ya make is running...
🟡 2025-10-02 10:45:50 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38616	35712	0	2	2862	40

⚪ 2025-10-02 10:45:59 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-10-02 11:05:16 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
394 (only retried tests)	261	0	0	108	25

🟢 2025-10-02 11:05:19 UTC Build successful.
🟢 2025-10-02 11:05:36 UTC ydbd size 2.2 GiB changed* by +82.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `b1e3f5a`	merge: `abdcea4`	diff	diff %
ydbd size	2 399 031 392 Bytes	2 399 116 152 Bytes	+82.8 KiB	+0.004%
ydbd stripped size	512 872 808 Bytes	512 888 680 Bytes	+15.5 KiB	+0.003%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

ydb/core/kqp/gateway/behaviour/streaming_query/optimization.cpp

gridnevvvit · 2025-10-02T12:32:01Z

ydb/core/kqp/host/kqp_host.cpp


        auto solomonState = MakeIntrusive<TSolomonState>();

+        solomonState->SupportRtmrMode = false;


что такое rtmr mode?

Это legacy режим записи в solomon в случае RTMR over YQL, в этом случае отключается DQ integration для записи. Он отключается в KQP, чтобы была возможность писать в solomon через DQ integration

github-actions · 2025-10-06T09:16:20Z

⚪ 2025-10-06 09:16:19 UTC Pre-commit check linux-x86_64-release-asan for 4524125 has started.
⚪ 2025-10-06 09:16:28 UTC Artifacts will be uploaded here
⚪ 2025-10-06 09:20:03 UTC ya make is running...
🟡 2025-10-06 11:55:14 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15710	15294	0	157	233	26

🟢 2025-10-06 11:55:22 UTC Build successful.
🟡 2025-10-06 11:55:46 UTC ydbd size 3.7 GiB changed* by +867.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `f41318a`	merge: `4524125`	diff	diff %
ydbd size	4 018 509 160 Bytes	4 019 397 008 Bytes	+867.0 KiB	+0.022%
ydbd stripped size	1 493 368 096 Bytes	1 493 566 208 Bytes	+193.5 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-10-06T09:18:44Z

⚪ 2025-10-06 09:18:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4524125 has started.
⚪ 2025-10-06 09:18:58 UTC Artifacts will be uploaded here
⚪ 2025-10-06 09:22:29 UTC ya make is running...
🟡 2025-10-06 11:18:29 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38657	35888	0	2	2740	27

⚪ 2025-10-06 11:18:41 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-10-06 11:34:31 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
252 (only retried tests)	228	0	1	1	22

⚪ 2025-10-06 11:34:33 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-10-06 11:43:40 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
238 (only retried tests)	217	0	1	0	20

🟢 2025-10-06 11:43:43 UTC Build successful.
🟢 2025-10-06 11:44:01 UTC ydbd size 2.3 GiB changed* by +71.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `6806a43`	merge: `4524125`	diff	diff %
ydbd size	2 416 481 832 Bytes	2 416 555 376 Bytes	+71.8 KiB	+0.003%
ydbd stripped size	514 852 840 Bytes	514 863 528 Bytes	+10.4 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-10-06T12:20:46Z

⚪ 2025-10-06 12:20:45 UTC Pre-commit check linux-x86_64-release-asan for 890e5ec has started.
⚪ 2025-10-06 12:20:59 UTC Artifacts will be uploaded here
⚪ 2025-10-06 12:24:33 UTC ya make is running...
🟡 2025-10-06 14:29:15 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15691	15260	0	130	280	21

🟢 2025-10-06 14:29:20 UTC Build successful.
🟡 2025-10-06 14:29:42 UTC ydbd size 3.7 GiB changed* by +221.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `280fc84`	merge: `890e5ec`	diff	diff %
ydbd size	4 019 649 800 Bytes	4 019 876 728 Bytes	+221.6 KiB	+0.006%
ydbd stripped size	1 493 543 392 Bytes	1 493 624 032 Bytes	+78.8 KiB	+0.005%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-10-06T12:21:02Z

⚪ 2025-10-06 12:21:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for 890e5ec has started.
⚪ 2025-10-06 12:21:34 UTC Artifacts will be uploaded here
⚪ 2025-10-06 12:26:10 UTC ya make is running...
🟡 2025-10-06 13:53:27 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
38639	35870	0	4	2740	25

⚪ 2025-10-06 13:53:39 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-10-06 14:06:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
231 (only retried tests)	208	0	1	1	21

⚪ 2025-10-06 14:06:29 UTC ya make is running... (failed tests rerun, try 3)
🟢 2025-10-06 14:28:32 UTC Tests successful.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
218 (only retried tests)	196	0	0	1	21

🟢 2025-10-06 14:28:35 UTC Build successful.
🟢 2025-10-06 14:28:52 UTC ydbd size 2.3 GiB changed* by +79.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `280fc84`	merge: `890e5ec`	diff	diff %
ydbd size	2 416 635 320 Bytes	2 416 717 056 Bytes	+79.8 KiB	+0.003%
ydbd stripped size	514 879 080 Bytes	514 897 960 Bytes	+18.4 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

GrigoriyPA requested review from evanevanevanevannnn and uzhastik October 2, 2025 08:57

GrigoriyPA requested review from a team as code owners October 2, 2025 08:57

github-actions bot added the new-feature label Oct 2, 2025

GrigoriyPA requested a review from Copilot October 2, 2025 08:59

Copilot AI reviewed Oct 2, 2025

View reviewed changes

GrigoriyPA requested a review from gridnevvvit October 2, 2025 09:11

uzhastik reviewed Oct 2, 2025

View reviewed changes

ydb/core/kqp/gateway/behaviour/streaming_query/optimization.cpp Outdated Show resolved Hide resolved

gridnevvvit reviewed Oct 2, 2025

View reviewed changes

GrigoriyPA added 5 commits October 6, 2025 12:13

Supported solomon writing in KQP

88af003

Fixed settings validation

e23557c

Cleanup

0ae7d64

Fixed typo

53e4dee

Simplified providers check

134d385

GrigoriyPA force-pushed the YQ-4750-support-solomon-writing branch from a856e98 to 134d385 Compare October 6, 2025 09:13

GrigoriyPA added 2 commits October 6, 2025 12:14

Cleanup

b7fd106

Cleanup 2

82d2884

GrigoriyPA requested review from gridnevvvit and uzhastik October 6, 2025 09:17

GrigoriyPA added the rebase-and-check Rebase PR with the current base branch and check label Oct 6, 2025

github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Oct 6, 2025

gridnevvvit approved these changes Oct 6, 2025

View reviewed changes

evanevanevanevannnn approved these changes Oct 7, 2025

View reviewed changes

GrigoriyPA merged commit 7d731ec into ydb-platform:main Oct 7, 2025
19 of 20 checks passed

GrigoriyPA deleted the YQ-4750-support-solomon-writing branch October 7, 2025 08:33

neyrox pushed a commit to neyrox/ydb that referenced this pull request Oct 7, 2025

YQ-4750 supported solomon writing in KQP (ydb-platform#26213)

a95cb52

serbel324 mentioned this pull request Oct 9, 2025

Mute broken test TBsLocalRecovery::WriteRestartReadHugeDecreased #26610

Merged

neyrox pushed a commit to neyrox/ydb that referenced this pull request Oct 10, 2025

YQ-4750 supported solomon writing in KQP (ydb-platform#26213)

67a0ce7

serbel324 mentioned this pull request Oct 10, 2025

Make VERIFY in ApplyGroupInfo() more verbose #26692

Open

GrigoriyPA added a commit to GrigoriyPA/ydb that referenced this pull request Oct 14, 2025

YQ-4750 supported solomon writing in KQP (ydb-platform#26213)

1692330

GrigoriyPA mentioned this pull request Oct 14, 2025

merge to stable-25-3 YQ-4312 features and fixes for streaming 3 #26809

Merged

GrigoriyPA added a commit to GrigoriyPA/ydb that referenced this pull request Oct 14, 2025

YQ-4750 supported solomon writing in KQP (ydb-platform#26213)

b032065


		auto solomonState = MakeIntrusive<TSolomonState>();

		solomonState->SupportRtmrMode = false;

YQ-4750 supported solomon writing in KQP #26213

YQ-4750 supported solomon writing in KQP #26213

Uh oh!

Conversation

GrigoriyPA commented Oct 2, 2025

Changelog entry

Changelog category

Description for reviewers

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

gridnevvvit Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

GrigoriyPA Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading

github-actions bot commented Oct 2, 2025 •

edited

Loading

GrigoriyPA Oct 6, 2025 •

edited

Loading

github-actions bot commented Oct 6, 2025 •

edited

Loading

github-actions bot commented Oct 6, 2025 •

edited

Loading

github-actions bot commented Oct 6, 2025 •

edited

Loading

github-actions bot commented Oct 6, 2025 •

edited

Loading