Skip to content

Support convertations PG types from YQL to Apache Arrow #28647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Nov 12, 2025
Merged

Support convertations PG types from YQL to Apache Arrow #28647

merged 4 commits into from
Nov 12, 2025

Conversation

@dahbka-lis
Copy link
Member

@dahbka-lis dahbka-lis commented Nov 11, 2025

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

...

Copilot AI review requested due to automatic review settings November 11, 2025 14:44
@dahbka-lis dahbka-lis requested a review from a team as a code owner November 11, 2025 14:44
@dahbka-lis dahbka-lis linked an issue Nov 11, 2025 that may be closed by this pull request
Support PG types from public mapping YQL to Arrow #28616
Closed
@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 14:45:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for eaf4763 has started.
⚪ 2025-11-11 14:46:18 UTC Artifacts will be uploaded here
🔴 2025-11-11 14:48:15 UTC Graph compare failed, see the logs.

Copilot finished reviewing on behalf of dahbka-lis November 11, 2025 14:46
@github-actions
Copy link

github-actions bot commented Nov 11, 2025

🟢 2025-11-11 14:47:10 UTC The validation of the Pull Request description is successful.

Copilot AI reviewed Nov 11, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for converting PostgreSQL types between YQL and Apache Arrow format. The implementation enables PG types to be serialized to Arrow's string representation using PostgreSQL's native text format.

  • Implements bidirectional conversion between PG types and Arrow format using string representation
  • Adds comprehensive test coverage for PG type conversions (Bool, Int8, Text)
  • Removes pg_dummy dependencies and adds proper pg_wrapper dependencies

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
ydb/core/kqp/ut/arrow/ya.make Removes pg_dummy dependency that is no longer needed
ydb/core/kqp/common/result_set_format/ya.make Adds pg_wrapper dependency for PG type conversion support
ydb/core/kqp/common/result_set_format/ut/ya.make Removes pg_dummy dependency from unit tests
ydb/core/kqp/common/result_set_format/ut/kqp_formats_ut_helpers.cpp Implements ExtractUnboxedValue for PG types using PgValueFromNativeText
ydb/core/kqp/common/result_set_format/ut/kqp_formats_arrow_ut.cpp Adds comprehensive test coverage for PG type conversions with test helper functions
ydb/core/kqp/common/result_set_format/kqp_formats_arrow.h Updates documentation to reflect PG type support in Arrow conversion
ydb/core/kqp/common/result_set_format/kqp_formats_arrow.cpp Implements PG type conversion logic using PgValueToNativeText for Arrow serialization

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 14:48:12 UTC Pre-commit check linux-x86_64-release-asan for eaf4763 has started.
⚪ 2025-11-11 14:48:33 UTC Artifacts will be uploaded here
⚫ 2025-11-11 14:49:30 UTC Check cancelled

@dahbka-lis
Update ydb/core/kqp/common/result_set_format/ut/kqp_formats_arrow_ut.cpp
871cdb1 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 14:50:54 UTC Pre-commit check linux-x86_64-release-asan for 95e67e6 has started.
⚪ 2025-11-11 14:51:00 UTC Artifacts will be uploaded here
🔴 2025-11-11 14:52:17 UTC Graph compare failed, see the logs.

@dahbka-lis dahbka-lis added the rebase-and-check Rebase PR with the current base branch and check label Nov 11, 2025
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Nov 11, 2025
@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 14:54:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for 95e67e6 has started.
⚪ 2025-11-11 14:54:29 UTC Artifacts will be uploaded here
🔴 2025-11-11 14:55:46 UTC Graph compare failed, see the logs.

@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 14:54:28 UTC Pre-commit check linux-x86_64-release-asan for 95e67e6 has started.
⚪ 2025-11-11 14:54:32 UTC Artifacts will be uploaded here
🔴 2025-11-11 14:55:51 UTC Graph compare failed, see the logs.

@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 16:23:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for 020ee15 has started.
⚪ 2025-11-11 16:24:11 UTC Artifacts will be uploaded here
🔴 2025-11-11 16:25:32 UTC Graph compare failed, see the logs.

@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 16:24:19 UTC Pre-commit check linux-x86_64-release-asan for 020ee15 has started.
⚪ 2025-11-11 16:24:36 UTC Artifacts will be uploaded here
🔴 2025-11-11 16:25:51 UTC Graph compare failed, see the logs.

@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 19:23:37 UTC Pre-commit check linux-x86_64-relwithdebinfo for 73568de has started.
⚪ 2025-11-11 19:23:54 UTC Artifacts will be uploaded here
⚪ 2025-11-11 19:26:14 UTC ya make is running...
🟡 2025-11-11 21:01:59 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
39867 37040 0 3 2793 31

⚪ 2025-11-11 21:02:12 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-11-11 21:11:23 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
66 (only retried tests) 50 0 0 0 16

🟢 2025-11-11 21:11:29 UTC Build successful.
🟢 2025-11-11 21:11:51 UTC ydbd size 2.3 GiB changed* by +5.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e6893df merge: 73568de diff diff %
ydbd size 2 440 020 280 Bytes 2 440 025 656 Bytes +5.2 KiB +0.000%
ydbd stripped size 519 052 752 Bytes 519 054 096 Bytes +1.3 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Nov 11, 2025 •
edited
Loading

⚪ 2025-11-11 19:25:49 UTC Pre-commit check linux-x86_64-release-asan for 73568de has started.
⚪ 2025-11-11 19:26:38 UTC Artifacts will be uploaded here
⚪ 2025-11-11 19:29:35 UTC ya make is running...
🟡 2025-11-11 21:26:42 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16247 15702 0 195 321 29

🟢 2025-11-11 21:26:51 UTC Build successful.
🟢 2025-11-11 21:27:20 UTC ydbd size 3.8 GiB changed* by +10.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 23bb07f merge: 73568de diff diff %
ydbd size 4 085 278 672 Bytes 4 085 289 424 Bytes +10.5 KiB +0.000%
ydbd stripped size 1 516 686 440 Bytes 1 516 690 024 Bytes +3.5 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@dahbka-lis dahbka-lis enabled auto-merge (squash) November 12, 2025 09:37
@dahbka-lis dahbka-lis merged commit 5494a1b into ydb-platform:main Nov 12, 2025
10 checks passed
@dahbka-lis dahbka-lis mentioned this pull request Nov 18, 2025
Open
dahbka-lis added a commit to dahbka-lis/ydb that referenced this pull request Nov 18, 2025
@dahbka-lis
Support convertations PG types from YQL to Apache Arrow (ydb-platform…
745386c 
…#28647)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
qyryq pushed a commit to qyryq/ydb that referenced this pull request Nov 24, 2025
@dahbka-lis @qyryq
Support convertations PG types from YQL to Apache Arrow (ydb-platform…
11ef7ce 
…#28647)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@gridnevvvit gridnevvvit gridnevvvit approved these changes

Assignees

No one assigned

Labels

not-for-changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support PG types from public mapping YQL to Arrow

2 participants

@dahbka-lis @gridnevvvit