YQ-4919 enable_streaming_queries_counters feature flag #29604

kardymonds · 2025-11-26T13:38:07Z

Changelog entry

...

Changelog category

Not for changelog (changelog entry is not required)

Description for reviewers

...

github-actions · 2025-11-26T13:39:30Z

⚪ 2025-11-26 13:39:29 UTC Pre-commit check linux-x86_64-relwithdebinfo for 731869b has started.
⚪ 2025-11-26 13:39:46 UTC Artifacts will be uploaded here
⚪ 2025-11-26 13:41:55 UTC ya make is running...
🟡 2025-11-26 16:03:49 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41629	38757	0	4	2839	29

⚪ 2025-11-26 16:04:05 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-11-26 16:14:39 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
62 (only retried tests)	46	0	2	0	14

⚪ 2025-11-26 16:14:46 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-11-26 16:23:01 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
32 (only retried tests)	16	0	2	0	14

🟢 2025-11-26 16:23:32 UTC Build successful.
🟢 2025-11-26 16:23:57 UTC ydbd size 2.3 GiB changed* by +6.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `c875917`	merge: `731869b`	diff	diff %
ydbd size	2 456 472 400 Bytes	2 456 479 376 Bytes	+6.8 KiB	+0.000%
ydbd stripped size	523 448 384 Bytes	523 449 984 Bytes	+1.6 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-11-26T13:39:30Z

⚪ 2025-11-26 13:39:30 UTC Pre-commit check linux-x86_64-release-asan for 731869b has started.
⚪ 2025-11-26 13:39:47 UTC Artifacts will be uploaded here
⚪ 2025-11-26 13:41:52 UTC ya make is running...
🟡 2025-11-26 15:34:19 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13473	13358	0	101	7	7

🟢 2025-11-26 15:34:28 UTC Build successful.
🟢 2025-11-26 15:34:57 UTC ydbd size 3.8 GiB changed* by +11.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `c875917`	merge: `731869b`	diff	diff %
ydbd size	4 111 923 912 Bytes	4 111 935 352 Bytes	+11.2 KiB	+0.000%
ydbd stripped size	1 528 251 056 Bytes	1 528 256 624 Bytes	+5.4 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-11-26T13:41:21Z

🟢 2025-11-26 13:41:20 UTC The validation of the Pull Request description is successful.

Copilot

Pull request overview

This PR introduces a new feature flag EnableStreamingQueriesCounters to conditionally enable performance counters for streaming queries. When disabled, the system creates empty counter groups to avoid collecting metrics overhead.

Added EnableStreamingQueriesCounters feature flag in protobuf definitions (defaulting to false)
Implemented conditional counter creation logic across multiple components (checkpoint coordinator, row dispatcher, topic session)
Updated checkpoint storage service counter subgroup naming for clarity

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ydb/core/protos/feature_flags.proto	Adds the `EnableStreamingQueriesCounters` feature flag definition (field 218, default false)
ydb/tests/fq/streaming/conftest.py	Enables the feature flag in streaming query tests
ydb/core/kqp/proxy_service/kqp_proxy_service.cpp	Renames counter subgroup from "storage_service" to "checkpoints_storage_service" for clarity
ydb/core/kqp/executer_actor/kqp_data_executer.cpp	Conditionally creates path-specific counter subgroups for checkpoint coordinator based on feature flag
ydb/core/fq/libs/row_dispatcher/topic_session.cpp	Implements conditional counter creation for topic and query-specific metrics based on feature flag
ydb/core/fq/libs/row_dispatcher/row_dispatcher.cpp	Adds feature flag check for query statistics and StartSession counter creation; includes debug code that should be removed

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-26T13:47:56Z

ydb/core/fq/libs/row_dispatcher/row_dispatcher.cpp

+    TAggQueryStat(const TString& queryId, const ::NMonitoring::TDynamicCounterPtr& counters, const NYql::NPq::NProto::TDqPqTopicSource& sourceParams, bool enableStreamingQueriesCounters)
        : QueryId(queryId)
        , SubGroup(counters) {
+            Cerr << "enableStreamingQueriesCounters " << enableStreamingQueriesCounters <<Endl;


Debug logging statement using Cerr should be removed before merging. This appears to be leftover debug code that will produce unwanted console output in production.

Suggested change

Cerr << "enableStreamingQueriesCounters " << enableStreamingQueriesCounters <<Endl;

Copilot · 2025-11-26T13:47:57Z

ydb/core/fq/libs/row_dispatcher/row_dispatcher.cpp

+    TAggQueryStat(const TString& queryId, const ::NMonitoring::TDynamicCounterPtr& counters, const NYql::NPq::NProto::TDqPqTopicSource& sourceParams, bool enableStreamingQueriesCounters)
        : QueryId(queryId)
        , SubGroup(counters) {
+            Cerr << "enableStreamingQueriesCounters " << enableStreamingQueriesCounters <<Endl;


Missing space before <<Endl. Should be << Endl for consistency with the rest of the statement.

Suggested change

Cerr << "enableStreamingQueriesCounters " << enableStreamingQueriesCounters <<Endl;

Cerr << "enableStreamingQueriesCounters " << enableStreamingQueriesCounters << Endl;

github-actions · 2025-11-26T20:06:59Z

⚪ 2025-11-26 20:06:58 UTC Pre-commit check linux-x86_64-release-asan for 3a6281f has started.
⚪ 2025-11-26 20:08:53 UTC Artifacts will be uploaded here
⚪ 2025-11-26 20:11:00 UTC ya make is running...
🟡 2025-11-26 22:02:46 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13510	13426	0	70	7	7

🟢 2025-11-26 22:02:54 UTC Build successful.
🟢 2025-11-26 22:03:28 UTC ydbd size 3.8 GiB changed* by +41.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `fe49740`	merge: `3a6281f`	diff	diff %
ydbd size	4 111 714 392 Bytes	4 111 757 048 Bytes	+41.7 KiB	+0.001%
ydbd stripped size	1 528 117 520 Bytes	1 528 133 008 Bytes	+15.1 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-11-26T20:08:54Z

⚪ 2025-11-26 20:08:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3a6281f has started.
⚪ 2025-11-26 20:09:10 UTC Artifacts will be uploaded here
⚪ 2025-11-26 20:11:19 UTC ya make is running...
🟡 2025-11-26 22:33:08 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
41666	38803	0	5	2838	20

⚪ 2025-11-26 22:33:20 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-11-26 22:43:19 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
46 (only retried tests)	32	0	1	0	13

⚪ 2025-11-26 22:43:26 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-11-26 22:51:10 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
28 (only retried tests)	14	0	1	0	13

🟢 2025-11-26 22:51:16 UTC Build successful.
🟢 2025-11-26 22:51:40 UTC ydbd size 2.3 GiB changed* by +26.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `fe49740`	merge: `3a6281f`	diff	diff %
ydbd size	2 456 428 592 Bytes	2 456 455 304 Bytes	+26.1 KiB	+0.001%
ydbd stripped size	523 437 504 Bytes	523 441 984 Bytes	+4.4 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

GrigoriyPA · 2025-11-27T07:56:32Z

ydb/library/yql/providers/pq/async_io/dq_pq_write_actor.cpp

+#include "ydb/core/base/appdata_fwd.h"
+#include "ydb/core/base/feature_flags.h"


Suggested change

#include "ydb/core/base/appdata_fwd.h"

#include "ydb/core/base/feature_flags.h"

#include <ydb/core/base/appdata_fwd.h>

#include <ydb/core/base/feature_flags.h>

GrigoriyPA · 2025-11-27T07:57:30Z

ydb/library/yql/providers/pq/async_io/dq_pq_write_actor.cpp

            : TxId(std::visit([](auto arg) { return ToString(arg); }, txId))
            , Counters(counters) {
-            if (Counters) {
+            if (Counters && NKikimr::AppData()->FeatureFlags.GetEnableStreamingQueriesCounters()) {


Тут не возникнет проблем при запуске в какой-нибудь RTMR / yql инсталяции, где возможно нету AppData?

GrigoriyPA · 2025-11-27T07:59:00Z

ydb/library/yql/providers/pq/async_io/dq_pq_read_actor.cpp

+#include "ydb/core/base/appdata_fwd.h"
+#include "ydb/core/base/feature_flags.h"


Suggested change

#include "ydb/core/base/appdata_fwd.h"

#include "ydb/core/base/feature_flags.h"

#include <ydb/core/base/appdata_fwd.h>

#include <ydb/core/base/feature_flags.h>

GrigoriyPA · 2025-11-27T07:59:23Z

ydb/library/yql/providers/pq/async_io/dq_pq_read_actor.cpp

            : TxId(std::visit([](auto arg) { return ToString(arg); }, txId))
            , Counters(counters) {
-            if (counters) {
+            if (counters && NKikimr::AppData()->FeatureFlags.GetEnableStreamingQueriesCounters()) {


Тоже, не уверен, что тут можно пользоваться AppData

GrigoriyPA · 2025-11-27T07:59:39Z

ydb/library/yql/providers/pq/async_io/dq_pq_rd_read_actor.cpp

        : TxId(std::visit([](auto arg) { return ToString(arg); }, txId))
        , Counters(counters) {
-        if (Counters) {
+        if (Counters && NKikimr::AppData()->FeatureFlags.GetEnableStreamingQueriesCounters()) {


Тоже, не верен, что тут можно пользоваться AppData

GrigoriyPA · 2025-11-27T08:02:02Z

ydb/library/yql/providers/pq/async_io/dq_pq_read_actor.cpp

-            if (counters) {
+            if (counters && NKikimr::AppData()->FeatureFlags.GetEnableStreamingQueriesCounters()) {
                SubGroup = Counters->GetSubgroup("source", "PqRead");
            } else {


Может в else ветке, если counters != nullptr, просто не добавлять лейбл и репортить агрегированные значения? Или там ещё лейблы потом добавляются?

GrigoriyPA · 2025-11-27T08:02:31Z

ydb/core/fq/libs/row_dispatcher/topic_session.cpp

+#include "ydb/core/base/appdata_fwd.h"
+#include "ydb/core/base/feature_flags.h"


Suggested change

#include "ydb/core/base/appdata_fwd.h"

#include "ydb/core/base/feature_flags.h"

#include <ydb/core/base/appdata_fwd.h>

#include <ydb/core/base/feature_flags.h>

GrigoriyPA · 2025-11-27T08:04:45Z

ydb/core/fq/libs/row_dispatcher/topic_session.cpp

-    void Init(const ::NMonitoring::TDynamicCounterPtr& counters, const TString& topicPath, const TString& readGroup, ui32 partitionId) {
-        TopicGroup = counters->GetSubgroup("topic", SanitizeLabel(topicPath));
+    void Init(const ::NMonitoring::TDynamicCounterPtr& counters, const TString& topicPath, const TString& readGroup, ui32 partitionId, bool enableStreamingQueriesCounters) {
+        TopicGroup = enableStreamingQueriesCounters


Может если enableStreamingQueriesCounters = false, то можно всё же агрегированные каунтеры репортить?

Кажется все лейблы создаются тут же (можно просто для ReadGroup и PartitionGroup репортить в counters)

GrigoriyPA · 2025-11-27T08:04:58Z

ydb/core/fq/libs/row_dispatcher/row_dispatcher.cpp

+#include "ydb/core/base/appdata_fwd.h"
+#include "ydb/core/base/feature_flags.h"


Suggested change

#include "ydb/core/base/appdata_fwd.h"

#include "ydb/core/base/feature_flags.h"

#include <ydb/core/base/appdata_fwd.h>

#include <ydb/core/base/feature_flags.h>

GrigoriyPA · 2025-11-27T08:05:54Z

ydb/core/fq/libs/row_dispatcher/row_dispatcher.cpp

        , SubGroup(counters) {
+        if (!enableStreamingQueriesCounters) {
+            SubGroup = MakeIntrusive<::NMonitoring::TDynamicCounters>();
+        }


Может быть тут всё же можно зарепортить агрегированные каунтеры? (если не сложно, вроде все лейблы в этой же функции прописываются)

enable_streaming_queries_counters feature flag added

3b1e682

Copilot AI review requested due to automatic review settings November 26, 2025 13:38

kardymonds requested review from a team as code owners November 26, 2025 13:38

github-actions bot added the not-for-changelog label Nov 26, 2025

Copilot started reviewing on behalf of kardymonds November 26, 2025 13:39 View session

Copilot finished reviewing on behalf of kardymonds November 26, 2025 13:40

Copilot AI reviewed Nov 26, 2025

View reviewed changes

add sensors to source/sink

16b5589

GrigoriyPA reviewed Nov 27, 2025

View reviewed changes

	Cerr << "enableStreamingQueriesCounters " << enableStreamingQueriesCounters <<Endl;
	Cerr << "enableStreamingQueriesCounters " << enableStreamingQueriesCounters << Endl;

		#include "ydb/core/base/appdata_fwd.h"
		#include "ydb/core/base/feature_flags.h"

YQ-4919 enable_streaming_queries_counters feature flag #29604

Are you sure you want to change the base?

YQ-4919 enable_streaming_queries_counters feature flag #29604

Uh oh!

Conversation

kardymonds commented Nov 26, 2025

Changelog entry

Changelog category

Description for reviewers

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 26, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Nov 26, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading

github-actions bot commented Nov 26, 2025 •

edited

Loading