[Bug] agg table and unique table get different result when enable_vectorized_engine=false

[nextdreamblue]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master

What's Wrong?

相同的数据，使用agg table replace和unique table进行查询返回的结果不同

MySQL [test]> select l_orderkey,l_partkey,l_suppkey,l_linenumber,l_comment from lineitem_agg where l_orderkey=5970000001 and l_partkey=75814607 and l_suppkey=5814608 and l_linenumber=1;
+------------+-----------+-----------+--------------+-----------+
| l_orderkey | l_partkey | l_suppkey | l_linenumber | l_comment |
+------------+-----------+-----------+--------------+-----------+
| 5970000001 |  75814607 |   5814608 |            1 |  00_63    |
+------------+-----------+-----------+--------------+-----------+
1 row in set (0.02 sec)

MySQL [test]> select l_orderkey,l_partkey,l_suppkey,l_linenumber,l_comment from lineitem_unique where l_orderkey=5970000001 and l_partkey=75814607 and l_suppkey=5814608 and l_linenumber=1;
+------------+-----------+-----------+--------------+-----------+
| l_orderkey | l_partkey | l_suppkey | l_linenumber | l_comment |
+------------+-----------+-----------+--------------+-----------+
| 5970000001 |  75814607 |   5814608 |            1 |  00_100   |
+------------+-----------+-----------+--------------+-----------+
1 row in set (0.02 sec)

What You Expected?

返回相同值

How to Reproduce?

创建两个表

CREATE TABLE `lineitem_agg` (
  `l_orderkey` bigint(20) NULL,
  `l_partkey` int(11) NULL,
  `l_suppkey` int(11) NULL,
  `l_linenumber` int(11) NULL,
  `l_quantity` decimal(15, 2) REPLACE NULL,
  `l_extendedprice` decimal(15, 2) REPLACE NULL,
  `l_discount` decimal(15, 2) REPLACE NULL,
  `l_tax` decimal(15, 2) REPLACE NULL,
  `l_returnflag` char(1) REPLACE NULL,
  `l_linestatus` char(1) REPLACE NULL,
  `l_shipdate` date REPLACE NULL,
  `l_commitdate` date REPLACE NULL,
  `l_receiptdate` date REPLACE NULL,
  `l_shipinstruct` char(25) REPLACE NULL,
  `l_shipmode` char(10) REPLACE NULL,
  `l_comment` varchar(44) REPLACE NULL
) ENGINE=OLAP
AGGREGATE KEY(`l_orderkey`, `l_partkey`, `l_suppkey`, `l_linenumber`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`l_orderkey`, `l_partkey`, `l_suppkey`, `l_linenumber`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2",
"disable_auto_compaction" = "false"
);

CREATE TABLE `lineitem_unique` (
  `l_orderkey` bigint(20) NULL,
  `l_partkey` int(11) NULL,
  `l_suppkey` int(11) NULL,
  `l_linenumber` int(11) NULL,
  `l_quantity` decimal(15, 2) NULL,
  `l_extendedprice` decimal(15, 2) NULL,
  `l_discount` decimal(15, 2) NULL,
  `l_tax` decimal(15, 2) NULL,
  `l_returnflag` char(1) NULL,
  `l_linestatus` char(1) NULL,
  `l_shipdate` date NULL,
  `l_commitdate` date NULL,
  `l_receiptdate` date NULL,
  `l_shipinstruct` char(25) NULL,
  `l_shipmode` char(10) NULL,
  `l_comment` varchar(44) NULL
) ENGINE=OLAP
UNIQUE KEY(`l_orderkey`, `l_partkey`, `l_suppkey`, `l_linenumber`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`l_orderkey`, `l_partkey`, `l_suppkey`, `l_linenumber`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2",
"disable_auto_compaction" = "false"
);

导入数据1：
数据文件：
https://doris-agg-table-bug-1301087413.cos.ap-beijing.myqcloud.com/agg_table_bug/test_agg_table_data_1

这个数据中包含100w数据，其中每隔10000条表，有1条主键为 " 5970999942|130195832|5195859|1 " 的数据，并且l_comment值按序号递增的数据，一种有100条主键相同l_comment不同的行，以及另一组主键为" 5979996005|197922326|5422384|4 "的多行数据。

同一份数据文件导入两个表。

先关闭向量化：

set enable_vectorized_engine=false;

然后执行如下sql

MySQL [test]> select l_orderkey,l_partkey,l_suppkey,l_linenumber,l_comment from lineitem_agg where l_orderkey=5970000001 and l_partkey=75814607 and l_suppkey=5814608 and l_linenumber=1;
+------------+-----------+-----------+--------------+-----------+
| l_orderkey | l_partkey | l_suppkey | l_linenumber | l_comment |
+------------+-----------+-----------+--------------+-----------+
| 5970000001 |  75814607 |   5814608 |            1 |  00_63    |
+------------+-----------+-----------+--------------+-----------+
1 row in set (0.02 sec)

MySQL [test]> select l_orderkey,l_partkey,l_suppkey,l_linenumber,l_comment from lineitem_unique where l_orderkey=5970000001 and l_partkey=75814607 and l_suppkey=5814608 and l_linenumber=1;
+------------+-----------+-----------+--------------+-----------+
| l_orderkey | l_partkey | l_suppkey | l_linenumber | l_comment |
+------------+-----------+-----------+--------------+-----------+
| 5970000001 |  75814607 |   5814608 |            1 |  00_100   |
+------------+-----------+-----------+--------------+-----------+
1 row in set (0.02 sec)

得到不一样的结果

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[nextdreamblue]

初步分析：

这个地方和行存处理replace聚合的逻辑有关

向量化_agg_key_next_block处理replace是先处理的第一行就是最终行，后续有相同key的值，replace的逻辑是
if (this->has_set_value()) {
return;
}
有值了就直接丢弃后续的行

非向量化_agg_key_next_row是先获取理论上丢弃的行，然后依次用后续行的值覆盖之前的值，最终处理完所有相同key的行就会得到最终replace的值
这种情况在处理一个rowset有多个segment，每个segment都有相同key的时候，会把相同key的segment_id小的放到后边处理，导致segment小的行的值会覆盖segment_id大的行，导致repalce的最终结果是返回了一个segment_id小的value

而unique表，把同一个rowset下segment_id小的行置为set_need_skip(true)，后续就直接把这类行淘汰了，最终返回的就是segment_id更大的value