YQ-4911 disabled transparent sys columns in PQ source

[GrigoriyPA]

Changelog entry

Disabled transparent sys columns in PQ source

Changelog category

• Bugfix

Description for reviewers

Bugfix ticket: https://st.yandex-team.ru/YQ-4911

[github-actions]

⚪ 2025-11-21 15:30:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for 14a08d6 has started.
⚪ 2025-11-21 15:31:11 UTC Artifacts will be uploaded here
⚪ 2025-11-21 15:33:23 UTC ya make is running...
⚫ 2025-11-21 15:37:14 UTC Check cancelled

[github-actions]

⚪ 2025-11-21 15:32:03 UTC Pre-commit check linux-x86_64-release-asan for 14a08d6 has started.
⚪ 2025-11-21 15:32:20 UTC Artifacts will be uploaded here
⚪ 2025-11-21 15:34:29 UTC ya make is running...
⚫ 2025-11-21 15:37:16 UTC Check cancelled

[github-actions]

🟢 2025-11-21 15:32:22 UTC The validation of the Pull Request description is successful.

[Copilot]

Pull request overview

This PR disables transparent system columns in the PQ (Persistent Queue) source to fix a bug. The change introduces a new configuration flag AllowTransperentSystemColumns (defaulting to true for backward compatibility but set to false in KQP) that controls whether system columns can use the transparent prefix (_yql_sys_tsp_).

Key changes:

• Refactored metadata field handling to support conditional transparent system columns
• Changed from static vector to dynamic map-based system for metadata fields
• Added configuration flag to control transparent column behavior

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 24 comments.

Show a summary per file

File	Description
ydb/library/yql/providers/pq/provider/yql_pq_provider.h	Added AllowTransperentSystemColumns flag to TPqState
ydb/library/yql/providers/pq/common/pq_meta_fields.h	Updated function signatures to support transparent column control
ydb/library/yql/providers/pq/common/pq_meta_fields.cpp	Refactored metadata field handling with new TPqMetadataField class
ydb/library/yql/providers/pq/provider/yql_pq_dq_integration.cpp	Updated to pass AllowTransperentSystemColumns flag
ydb/library/yql/providers/pq/provider/yql_pq_datasource_type_ann.cpp	Updated to pass AllowTransperentSystemColumns flag
ydb/library/yql/providers/pq/provider/yql_pq_datasource.cpp	Updated to pass AllowTransperentSystemColumns flag
ydb/library/yql/providers/pq/async_io/dq_pq_meta_extractor.cpp	Refactored to use key-based extractors instead of system column names
ydb/core/kqp/host/kqp_host.cpp	Set AllowTransperentSystemColumns to false in KQP host
ydb/core/kqp/ut/federated_query/datastreams/datastreams_ut.cpp	Added regression test for projection queries

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

⚪ 2025-11-21 15:39:00 UTC Pre-commit check linux-x86_64-release-asan for c05e044 has started.
⚪ 2025-11-21 15:39:18 UTC Artifacts will be uploaded here
⚪ 2025-11-21 15:41:24 UTC ya make is running...
🟡 2025-11-21 16:53:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
12384	12268	0	79	21	16

🟢 2025-11-21 16:53:43 UTC Build successful.
🟢 2025-11-21 16:54:16 UTC ydbd size 3.8 GiB changed* by +15.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: ba86a15	merge: c05e044	diff	diff %
ydbd size	4 109 044 696 Bytes	4 109 060 568 Bytes	+15.5 KiB	+0.000%
ydbd stripped size	1 527 307 536 Bytes	1 527 311 408 Bytes	+3.8 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions]

⚪ 2025-11-21 15:39:52 UTC Pre-commit check linux-x86_64-relwithdebinfo for c05e044 has started.
⚪ 2025-11-21 15:40:27 UTC Artifacts will be uploaded here
⚪ 2025-11-21 15:42:31 UTC ya make is running...
🟡 2025-11-21 17:12:37 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
39342	36457	0	4	2844	37

⚪ 2025-11-21 17:12:50 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-11-21 17:23:43 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
84 (only retried tests)	69	0	1	0	14

⚪ 2025-11-21 17:23:49 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-11-21 17:31:59 UTC Some tests failed, follow the links below.

Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
30 (only retried tests)	15	0	1	0	14

🟢 2025-11-21 17:32:05 UTC Build successful.
🟢 2025-11-21 17:32:27 UTC ydbd size 2.3 GiB changed* by +14.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: ba86a15	merge: c05e044	diff	diff %
ydbd size	2 454 536 952 Bytes	2 454 551 872 Bytes	+14.6 KiB	+0.001%
ydbd stripped size	523 091 288 Bytes	523 094 616 Bytes	+3.2 KiB	+0.001%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions]

⚪ 2025-11-24 13:44:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for b0d20d9 has started.
⚪ 2025-11-24 13:44:48 UTC Artifacts will be uploaded here
⚪ 2025-11-24 13:46:53 UTC ya make is running...
🟡 2025-11-24 15:12:04 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
39357	36479	0	3	2844	31

⚪ 2025-11-24 15:12:17 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-11-24 15:23:40 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
74 (only retried tests)	59	0	0	0	15

🟢 2025-11-24 15:23:47 UTC Build successful.
🟢 2025-11-24 15:24:06 UTC ydbd size 2.3 GiB changed* by +72.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: 72f7256	merge: b0d20d9	diff	diff %
ydbd size	2 455 293 832 Bytes	2 455 368 024 Bytes	+72.5 KiB	+0.003%
ydbd stripped size	523 292 888 Bytes	523 311 640 Bytes	+18.3 KiB	+0.004%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions]

⚪ 2025-11-24 13:44:35 UTC Pre-commit check linux-x86_64-release-asan for b0d20d9 has started.
⚪ 2025-11-24 13:44:53 UTC Artifacts will be uploaded here
⚪ 2025-11-24 13:46:59 UTC ya make is running...
🟡 2025-11-24 14:54:21 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
11839	11768	0	55	7	9

🟢 2025-11-24 14:54:28 UTC Build successful.
🟢 2025-11-24 14:55:00 UTC ydbd size 3.8 GiB changed* by +15.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: 9c507f9	merge: b0d20d9	diff	diff %
ydbd size	4 110 241 408 Bytes	4 110 257 272 Bytes	+15.5 KiB	+0.000%
ydbd stripped size	1 527 785 808 Bytes	1 527 789 680 Bytes	+3.8 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}