[Bug](High) array_agg_foreach fails on APPEND_ONLY table streams with "array_agg not support write"

[MoanasDaddyXu]

Tracking

• Tracking issue: [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265
• Internal reference: DORIS-27442

Search before asking

• I searched the existing issues and found no issue with the same ROW-binlog table-stream failure.
• [Bug] array_agg_foreach funtion is broken #52252 concerns missing or unstable results when querying a regular table; this issue instead fails during an APPEND_ONLY table-stream query with NOT_IMPLEMENTED_ERROR.

Version

Reproduced on Doris commit a8b1fd95e2a.

What's Wrong?

Calling array_agg_foreach on an APPEND_ONLY table stream backed by ROW binlog fails during query execution:

[NOT_IMPLEMENTED_ERROR]array_agg not support write

The equivalent aggregation over the base table produces a result. The failure is specific to querying the table stream and also affects nested array input.

What You Expected?

array_agg_foreach should execute on the ROW-binlog table stream and return results with the same semantics as the corresponding base-table query.

How to Reproduce?

DROP DATABASE IF EXISTS array_agg_stream_repro;
CREATE DATABASE array_agg_stream_repro;
USE array_agg_stream_repro;

CREATE TABLE foreach_table (
    id INT,
    a ARRAY<INT>,
    b ARRAY<ARRAY<INT>>
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
    "replication_num" = "1",
    "binlog.enable" = "true",
    "binlog.format" = "ROW"
);

CREATE STREAM foreach_table_stream ON TABLE foreach_table
PROPERTIES ("type" = "append_only");

INSERT INTO foreach_table VALUES
    (1, [1, 2, 3], [[1], [1, 2, 3], [2]]),
    (2, [20], [[2]]),
    (3, [100], [[1]]),
    (4, NULL, [NULL]),
    (5, [NULL, 2], [[2], NULL]);

-- The equivalent query on foreach_table succeeds.
SELECT array_agg_foreach(a) FROM foreach_table_stream;

SELECT /*+ SET_VAR(parallel_pipeline_task_num=1) */
    size(array_agg_foreach(b)),
    size(element_at(array_agg_foreach(b), 1)),
    size(element_at(array_agg_foreach(b), 2)),
    size(element_at(array_agg_foreach(b), 3))
FROM foreach_table_stream;

Both table-stream queries fail with the error above.

be.out

The following excerpt contains only the relevant, sanitized BE output:

Pipeline task failed. reason: [NOT_IMPLEMENTED_ERROR]array_agg not support write
report error status: array_agg not support write to coordinator

Impact

Queries that use array_agg_foreach cannot run against APPEND_ONLY table streams, blocking incremental-processing workloads that aggregate array columns.

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_c762c3070a5c

Initial triage

This is a valid BE aggregate-state serialization bug. It is high impact for affected table-stream/incremental queries, but the current evidence does not indicate data corruption or a BE crash. The issue is open with no labels, assignee, or milestone; adding the project’s bug plus BE/function and table-stream/binlog area labels would help routing.

The reproduction, commit, schema, data, and error are sufficient to make the nested-array failure actionable.

Code-level finding

At the reported commit, the error text has a single relevant source: the complex-type specialization of AggregateFunctionArrayAggData stores values in column_data, but its binary write, read, and merge methods are unconditional NOT_IMPLEMENTED_ERROR throws.

array_agg_foreach/array_agg_foreachv2 keeps one nested aggregate state per array position. Its serialize() loops over those positions and calls nested_function->serialize(). Therefore, for b ARRAY<ARRAY<INT>>, the nested function is array_agg(ARRAY<INT>); it selects the complex-type state above and deterministically throws as soon as a partial aggregate state must be serialized. A one-stage base-table plan can avoid that method, which explains why an equivalent base-table query may appear healthy while a plan with an aggregate-state boundary fails.

This is a latent array_agg complex-state contract gap exposed by the table-stream plan, not evidence of malformed ROW-binlog array decoding.

There is one important discrepancy to resolve: for a ARRAY<INT>, the foreach nested function should be array_agg(INT), and the primitive specialization does implement write/read/merge. The supplied combined log excerpt does not show why that query reportedly reaches the complex-type throw. Since the error literal comes from the complex specialization, the first query either has a different runtime aggregate parameter type/plan than expected or needs a separate per-query confirmation.

Targeted missing evidence

Please attach the following for each query separately:

1. EXPLAIN VERBOSE (or the most detailed available plan) for both the base-table and table-stream forms, showing local/global aggregate phases and exchanges.
2. The query ID and full BE stack trace for the first (a) query, or the serialized TAggregateExpr.param_types for its array_agg_foreachv2 evaluator.
3. Separate success/error output for the two statements rather than one combined sanitized excerpt.

No broader logs are needed initially. The nested b failure is already code-confirmed; these artifacts are only needed to pin down the scalar-array a path.

Recommended next steps

1. Implement a complete complex-type state contract for AggregateFunctionArrayAggData: merge by appending column_data, and provide matching serialization/deserialization. The existing complex-type AggregateFunctionCollectListData SerDe-backed implementation is a useful reference, though maintainers should decide whether to reuse that format or introduce a column-native intermediate format with explicit compatibility considerations.
2. Add a BE unit test that exercises serialize -> deserialize -> merge -> result for array_agg_foreach with both ARRAY<INT> and ARRAY<ARRAY<INT>>, including NULLs and variable-length rows. Existing tests mostly exercise add/result and do not cover this failing contract.
3. Add a table_stream_p0 regression using the supplied ROW-binlog/APPEND_ONLY setup, with base-table controls and both single- and multi-pipeline execution.
4. Before finalizing the fix, verify that the first query’s runtime nested argument is INT; if it is unexpectedly ARRAY<INT>, fix that planner/translator type propagation in addition to the complex-state serializer.