DataFusionError(Internal("The size of the sorted batch is larger than the size of the input batch: 2120 > 2312"))

[andygrove]

Describe the bug

Found this running some SQL benchmarks

DataFusionError(Internal("The size of the sorted batch is larger than the size of the input batch: 2120 > 2312"))

The error is also confusing .. 2120 is not larger than 2312

To Reproduce
tbd

Expected behavior
This should not happen

Additional context
Found this running some SQL benchmarks

[isidentical]

I think I am able to trigger it using the query below:

CREATE EXTERNAL TABLE decimal_simple (
            c1  DECIMAL(10,6) NOT NULL,
            c2  DOUBLE NOT NULL,
            c3  BIGINT NOT NULL,
            c4  BOOLEAN NOT NULL,
            c5  DECIMAL(12,7) NOT NULL
            )
            STORED AS CSV
            WITH HEADER ROW
            LOCATION '../datafusion/core/tests/decimal_data.csv';

select * from decimal_simple where c1 >= 0.00004 order by c1 limit 100;

But still not sure about the underlying issue.

The size of the input array changes even though the contents stay the same (so probably buffers too). Maybe that assumption doesn't hold when we are recreating the arrays (and/or maybe the ordering actually changes how buffers are built), so instead of triggering an assertion, we might need to factor this case in and just run into this branch when we are sure that the new_size is less than size.

[Dandandan]

I think the code should update the size

I think I am able to trigger it using the query below:

CREATE EXTERNAL TABLE decimal_simple (
            c1  DECIMAL(10,6) NOT NULL,
            c2  DOUBLE NOT NULL,
            c3  BIGINT NOT NULL,
            c4  BOOLEAN NOT NULL,
            c5  DECIMAL(12,7) NOT NULL
            )
            STORED AS CSV
            WITH HEADER ROW
            LOCATION '../datafusion/core/tests/decimal_data.csv';

select * from decimal_simple where c1 >= 0.00004 order by c1 limit 100;

But still not sure about the underlying issue.

The size of the input array changes even though the contents stay the same (so probably buffers too). Maybe that assumption doesn't hold when we are recreating the arrays (and/or maybe the ordering actually changes how buffers are built), so instead of triggering an assertion, we might need to factor this case in and just run into this branch when we are sure that the new_size is less than size.

Seems indeed we should handle the case of a sorted batch being somehow bigger than the input batch, and add an issue in arrow-rs to tackle this there.

[jychen7]

add an issue in arrow-rs to tackle this there.

is there an issue in arrow-rs to track? I didn't find it, so create apache/arrow-rs#4087. Feel free to close if it is duplicated

[Dandandan]

Thanks @jychen7 I don't think I did create it.