[Bug] Arrow Flight SQL: first VARIANT read on a pre-existing connection returns an empty string instead of {}

[morningman]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Apache Doris 4.1.3-rc02, commit 31263df4dc1d4d3a27517d264802cd4d6b92c874

Client: Python + ADBC Flight SQL driver (adbc_driver_flightsql), FE arrow_flight_sql_port = 41070.
The MySQL/JDBC protocol is used as the control path for comparison.

What's Wrong?

If an ADBC Flight SQL connection is opened before a VARIANT table is created and written, the first read of a persisted empty JSON object on that connection can return an empty string instead of {}.

Running exactly the same query a second time on the same connection returns {}, and a freshly opened ADBC connection also returns {} on its first try.

The raw Arrow schema is int32, string, string:

• pre-existing connection, first read: ["1", "", ""]
• same connection, second read: ["1", "{}", "{}"]
• new connection after the write: ["1", "{}", "{}"]

What You Expected?

The first read and all subsequent reads should be identical. An empty JSON object should come back as {} for both nullable and NOT NULL VARIANT columns.

How to Reproduce?

1. Open the ADBC Flight SQL connection first, without querying the target table yet.
2. Over a separate MySQL/JDBC connection, run the DDL and INSERT below.
3. On the pre-existing ADBC connection, run the same SELECT twice in a row.
4. Compare the two Arrow results, then open a new ADBC connection and run it once more as a control.

DROP TABLE IF EXISTS adbc_variant_first_read;
CREATE TABLE adbc_variant_first_read (
  id INT,
  v_nullable VARIANT NULL,
  v_not_null VARIANT NOT NULL
) DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES("replication_num"="1");

INSERT INTO adbc_variant_first_read
VALUES (1, PARSE_JSON('{}'), PARSE_JSON('{}'));

SELECT id, CAST(v_nullable AS STRING), CAST(v_not_null AS STRING)
FROM adbc_variant_first_read ORDER BY id;

Client side:

import adbc_driver_flightsql.dbapi as flight_sql

# Open this connection BEFORE the JDBC session runs CREATE TABLE and INSERT.
conn = flight_sql.connect(uri="grpc://127.0.0.1:41070",
                          db_kwargs={"username": "root", "password": ""})
cur = conn.cursor()
sql = ("SELECT id, CAST(v_nullable AS STRING), CAST(v_not_null AS STRING) "
       "FROM adbc_variant_first_read ORDER BY id")
cur.execute(sql)
print(cur.fetch_arrow_table().to_pylist())  # first read may contain empty strings
cur.execute(sql)
print(cur.fetch_arrow_table().to_pylist())  # returns {}

Anything Else?

This looks like stale per-connection schema/metadata state on the Flight SQL session: the connection was established before the table existed, and the first read resolves the variant subcolumns against that stale state.

Workaround: retry the identical read, or re-establish the ADBC connection after the write completes. A silent retry was deliberately not added to the test path, because it would hide the consistency problem.

Also reproducible through the variant_p0/variant_hirachinal regression case, in addition to the isolated table above.

Tracking issue: #65615

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_0480a2cd30b5

Initial triage

This should be treated as a correctness bug, not only a client-display issue: a persisted JSON object is reported as a different, valid string value on the first read. The identical Arrow schema on the bad and good executions (int32, string, string), together with both nullable and non-nullable columns being affected, makes an Arrow type mismatch or outer-nullability handling unlikely.

The proposed root cause (stale table/schema metadata held by the pre-existing Flight SQL connection) is not yet established. In the supplied sequence, the new-connection control runs only after the old connection has executed the query twice. It therefore changes both connection age and query order: a first-reader/storage initialization or timing effect could produce the same observation.

What the 4.1.3-rc02 code supports

I inspected commit 31263df4dc1d4d3a27517d264802cd4d6b92c874 read-only.

• The Flight SQL path resets its result channel for each statement and constructs a new processor/executor. I did not find a per-connection table-schema snapshot in FlightSqlConnectContext, DorisFlightSqlProducer, or FlightSqlConnectProcessor that would directly explain a table created after connection establishment.
• CAST(VARIANT AS STRING) is evaluated before the Arrow result conversion. The Flight result writer receives an ordinary Doris string column, so Arrow schema metadata alone cannot turn {} into an empty string.
• The V1 ColumnVariant serializer explicitly emits {} when a row has no visible paths, and the hierarchical reader comments/code preserve an empty root placeholder as a present empty object rather than SQL NULL.
• The V2 assembler has the same explicit rule: an empty whole-Variant row is {}, while an absent subtree is SQL NULL.
• Unit coverage exists for the isolated empty-object assembly semantics, but I did not find an end-to-end regression covering persisted scan -> Variant-to-string cast -> Flight result on the first query.

These are verified code-path facts, but they do not identify the component that first produces the zero-length value. The remaining hypotheses include a first-read storage/Variant initialization path, a session-state difference, and the ADBC/Arrow conversion boundary. There is currently insufficient evidence to rank one as the root cause.

Information needed to isolate the fault

Please attach:

1. Exact versions of adbc_driver_flightsql, PyArrow, and the Arrow C++ libraries used by the driver.
2. Whether the failure is deterministic, its frequency over clean-table repetitions, and whether delaying the first Flight query changes it.
3. SHOW CREATE TABLE, FE/BE topology and deployment mode, plus the effective Variant settings—especially enable_variant_v2 and the table's document/hierarchical-storage settings. Please do not infer these from release defaults.
4. Query IDs for the first bad and second good executions, with the corresponding FE/BE logs and query profiles. A profile pair is useful even when runtimes are similar because it identifies the scan instance and result-sink path.
5. The result of a diagnostic query that returns the cast value together with LENGTH(...) and HEX(...) on the first execution. If the server-computed length is already 0, the fault precedes Arrow string-buffer materialization; if it is 2 while the fetched value is empty, focus on the Flight/ADBC boundary.

The most important reproduction control is to remove the query-order confounder after each fresh table creation:

• Make a connection opened after the insert the very first reader, before the pre-existing connection runs anything.
• In a separate repetition, make the pre-existing connection the first reader.
• Before any Flight query, run the exact CAST(... AS STRING) query over MySQL/JDBC and record LENGTH/HEX as a storage/control-path check.
• Repeat with a non-empty object such as {"a":1} and an empty array to determine whether this is specific to the empty-object root placeholder.

Suggested maintainer next steps

Reproduce on the reported commit with the controlled ordering above, then inspect the value at three boundaries for the bad query ID: after Variant row assembly, after VARIANT -> STRING serialization, and immediately before the Doris block is converted to an Arrow batch. The first boundary at which the bytes change from 7b 7d to zero length will determine ownership without assuming a connection cache.

Once isolated, add an end-to-end regression that opens the Flight connection before DDL, persists {}, and asserts the first read, repeated read, and first read from a second connection. A lower-level persisted-segment scan/cast test should accompany it if the corruption is already present before the result sink.

Retrying or reconnecting is a reasonable temporary workaround, but a silent retry would mask a correctness failure and should not be considered the fix. This issue can remain linked to #65615, but it should not be closed as a duplicate without identifying the failing boundary. No labels are currently attached; kind/bug, area/rpc, and the repository's existing VARIANT/storage ownership label (if available) would make routing clearer.