[fix](arrow) correct the Flight SQL GetTables schema and the TIMESTAMPTZ arrow reader - #66344
Merged
morningman merged 2 commits intoAug 2, 2026
Merged
Conversation
The schema GetTables reports and the data that follows it disagreed in two places, and getArrowType's own comment says they must not: it is documented as mirroring convert_to_arrow_type in the backend. A DATEV2 column was described as Date(MILLISECOND) -- date64 -- while BE writes arrow::Date32Type. A client that types its columns from this schema sees a datetime where a date arrives, so it renders the column as one, compares it as one, and its read fails with "not support convert to datetimev2 from arrow type: 16" on the first batch. A complex column was described with a placeholder child: an array's element type was the Null type, a map's pair was a bare list, a struct had no fields at all. An Arrow ARRAY/MAP/STRUCT carries its element types in its children and nowhere else, so this said "array of nothing" about every array in the catalog while BE emitted ListType(item), MapType(key, value) and StructType(fields) in the batches. describeTables already reports the whole tree -- Column's children even name an array's element "item" and a map's pair "key"/"value", which is what Arrow calls them -- so the children are built from it. A descriptor that reports none keeps the old placeholders: a source that cannot describe its nested types is no worse off than before. The test pins each mapping against what BE emits rather than against itself: the date unit, an array's element type, a map's key/value pair under the entries struct, a struct's fields, a nested tree walked to its leaves, and the placeholders a descriptor without children still gets. The last case asserts after a round trip through the serialized schema, because that byte string -- not the Field objects -- is what reaches the client in the table_schema column. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
DataTypeTimeStampTzSerDe never overrode read_column_from_arrow, so the one it inherited from DataTypeNumberSerDe<TYPE_TIMESTAMPTZ> ran: its fixed-width path memcpy's the array's buffer straight into the column. Arrow hands it int64 EPOCH values; the column stores PACKED date/time values. Both are eight bytes wide, so the width check passes, the scan succeeds, and every row is silently wrong -- a Doris datetime read back over Arrow Flight SQL rendered as "+08:05" with no date in it. The reader converts by unit into the microseconds the column stores. NANO is divided rather than refused, since no Doris datetime type keeps sub-microsecond digits, and the division floors so a pre-1970 instant does not move forward by one microsecond. The value is read as an instant on the UTC line, the inverse of what write_column_to_arrow emits, so the round trip is exact. Two cases are deliberate. A null slot is not converted: the bytes under it are whatever the source left there, and running a garbage epoch through the range check would fail a batch whose rows are all well-formed. An arrow type this serde cannot decode is an error rather than a fallback -- accepting anything eight bytes wide is how the corruption above stayed invisible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1z6M8Gj9F9pErafGdfdjx
morningman
force-pushed
the
fix-flightsql-arrow-type-fidelity
branch
from
August 1, 2026 04:06
038d93c to
6ec4202
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28953 ms |
Contributor
TPC-DS: Total hot run time: 169641 ms |
Contributor
ClickBench: Total hot run time: 23.79 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Gabriel39
approved these changes
Aug 1, 2026
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
Bot
added
approved
Contributor
|
PR approved by anyone and no changes requested. |
github-actions Bot
pushed a commit
that referenced
this pull request
Aug 2, 2026
…PTZ arrow reader (#66344) ### What problem does this PR solve? Related Issue: #65615 Problem Summary: Two places where Doris says one thing about an Arrow value and does another. Both were found while reading Doris over Arrow Flight SQL, and both reproduce on master on their own. **1. `CommandGetTables` describes types the batches do not carry** `FlightSqlSchemaHelper.getArrowType` is documented as mirroring `convert_to_arrow_type` in the backend -- the schema a client reads from `GetTables` is what it types its columns by. Two of the mappings had drifted from what BE actually writes: - A `DATEV2` column was described as `Date(MILLISECOND)` -- Arrow date64 -- while BE writes `arrow::Date32Type`, a day number. A client that trusts the schema renders and compares the column as a datetime and then fails on the first batch with `not support convert to datetimev2 from arrow type: 16`. - A complex column was described with a placeholder child: an array's element type was the Null type, a map's pair was a bare list, a struct had no fields at all. An Arrow ARRAY/MAP/STRUCT carries its element types in its children and nowhere else, so this said "array of nothing" about every array in the catalog while BE emitted `ListType(item)`, `MapType(key, value)` and `StructType(fields)` in the data. `describeTables` already reports the whole tree -- `Column.createChildrenColumn` even names an array's element `item` and a map's pair `key`/`value`, which is what Arrow calls them -- so the children are now built from it, recursively. A descriptor that reports no children keeps the old placeholders: a source that cannot describe its nested types is no worse off than before. **2. A TIMESTAMPTZ column read from Arrow was decoded by copying its bits** `DataTypeTimeStampTzSerDe` never overrode `read_column_from_arrow`, so the one it inherited from `DataTypeNumberSerDe<TYPE_TIMESTAMPTZ>` ran: its fixed-width path `memcpy`s the array's buffer straight into the column. Arrow hands it int64 EPOCH values; the column stores PACKED date/time values. Both are eight bytes wide, so the width check passes, the scan succeeds, and every row is silently wrong -- a Doris datetime read back over Arrow Flight SQL rendered as `+08:05` with no date in it. This is reachable today through the `remote_doris` catalog. Its schema is the remote table's `Column` objects deserialized verbatim (`RemoteDorisRestClient.parseColumns`), so a remote TIMESTAMPTZ column stays TIMESTAMPTZ locally; `RemoteDorisScanNode` plans the scan as `FORMAT_ARROW`; and `remote_doris_reader` materializes it through this serde. The remote side writes the epochs with the serde's own `write_column_to_arrow`, so both halves of that round trip are Doris code and the mismatch is entirely internal. The other Arrow readers -- `arrow_stream_reader` for the `arrow` file format, `python_udtf_function`, `paimon_cpp_reader` -- share the same serde and are covered by the same change; whether each of them can present a TIMESTAMPTZ column today is not something this PR claims either way. The reader converts by unit into the microseconds the column stores. NANO is divided rather than refused, since no Doris datetime type keeps sub-microsecond digits, and the division floors so a pre-1970 instant does not move forward by one microsecond. The value is read as an instant on the UTC line, the inverse of what `write_column_to_arrow` emits, so the round trip is exact. Two cases are deliberate. A null slot is not converted: the bytes under it are whatever the source left there, and running a garbage epoch through the range check would fail a batch whose rows are all well-formed. An Arrow type this serde cannot decode is an error rather than a fallback -- accepting anything eight bytes wide is how the corruption above stayed invisible.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related Issue: #65615
Problem Summary:
Two places where Doris says one thing about an Arrow value and does another. Both were found
while reading Doris over Arrow Flight SQL, and both reproduce on master on their own.
1.
CommandGetTablesdescribes types the batches do not carryFlightSqlSchemaHelper.getArrowTypeis documented as mirroringconvert_to_arrow_typein thebackend -- the schema a client reads from
GetTablesis what it types its columns by. Two of themappings had drifted from what BE actually writes:
DATEV2column was described asDate(MILLISECOND)-- Arrow date64 -- while BE writesarrow::Date32Type, a day number. A client that trusts the schema renders and compares thecolumn as a datetime and then fails on the first batch with
not support convert to datetimev2 from arrow type: 16.type, a map's pair was a bare list, a struct had no fields at all. An Arrow ARRAY/MAP/STRUCT
carries its element types in its children and nowhere else, so this said "array of nothing"
about every array in the catalog while BE emitted
ListType(item),MapType(key, value)andStructType(fields)in the data.describeTablesalready reports the whole tree --Column.createChildrenColumneven names anarray's element
itemand a map's pairkey/value, which is what Arrow calls them -- so thechildren are now built from it, recursively. A descriptor that reports no children keeps the old
placeholders: a source that cannot describe its nested types is no worse off than before.
2. A TIMESTAMPTZ column read from Arrow was decoded by copying its bits
DataTypeTimeStampTzSerDenever overroderead_column_from_arrow, so the one it inherited fromDataTypeNumberSerDe<TYPE_TIMESTAMPTZ>ran: its fixed-width pathmemcpys the array's bufferstraight into the column. Arrow hands it int64 EPOCH values; the column stores PACKED date/time
values. Both are eight bytes wide, so the width check passes, the scan succeeds, and every row is
silently wrong -- a Doris datetime read back over Arrow Flight SQL rendered as
+08:05with nodate in it.
This is reachable today through the
remote_doriscatalog. Its schema is the remote table'sColumnobjects deserialized verbatim (RemoteDorisRestClient.parseColumns), so a remoteTIMESTAMPTZ column stays TIMESTAMPTZ locally;
RemoteDorisScanNodeplans the scan asFORMAT_ARROW; andremote_doris_readermaterializes it through this serde. The remote sidewrites the epochs with the serde's own
write_column_to_arrow, so both halves of that round tripare Doris code and the mismatch is entirely internal.
The other Arrow readers --
arrow_stream_readerfor thearrowfile format,python_udtf_function,paimon_cpp_reader-- share the same serde and are covered by the samechange; whether each of them can present a TIMESTAMPTZ column today is not something this PR
claims either way.
The reader converts by unit into the microseconds the column stores. NANO is divided rather than
refused, since no Doris datetime type keeps sub-microsecond digits, and the division floors so a
pre-1970 instant does not move forward by one microsecond. The value is read as an instant on the
UTC line, the inverse of what
write_column_to_arrowemits, so the round trip is exact.Two cases are deliberate. A null slot is not converted: the bytes under it are whatever the source
left there, and running a garbage epoch through the range check would fail a batch whose rows are
all well-formed. An Arrow type this serde cannot decode is an error rather than a fallback --
accepting anything eight bytes wide is how the corruption above stayed invisible.
Release note
Fix two Arrow type-fidelity bugs.
GetTablesover Arrow Flight SQL reportedDATEV2as date64 while the data is date32, anddescribed
ARRAY/MAP/STRUCTcolumns with placeholder children instead of their real elementtypes; a client that types its columns from that schema failed on the first batch.
A
TIMESTAMPTZcolumn materialized from Arrow was decoded by copying the epoch integers over thecolumn's packed values -- same width, no error, every row wrong. This affects any reader that
builds a TIMESTAMPTZ column from Arrow, including the native Paimon reader.
Check List (For Author)
Test
Behavior changed:
DATEV2and for nested types, the Arrowtypes the batches actually carry. A client that had worked around the old schema by
ignoring it is unaffected; one that trusted it stops failing.
TIMESTAMPTZcolumn read from Arrow now holds the instant the source sent instead ofa reinterpreted epoch. Values previously returned on those paths were wrong, so results
change -- to the correct ones.
Does this need documentation?
Check List (For Reviewer who merge this PR)