HIVE-24827. Hive aggregation query returns incorrect results for non text files.

[ayushtkn]

No description provided.

[kgyrtkirk]

will this also work for in cases when the input data is compressed?
I think the existing one did worked for that as well...tests passed so it was either not covered or still working fine :)

[kgyrtkirk]

seems to be a duplicate block ; you could move it into a method

[ayushtkn]

Thanx @kgyrtkirk for the review. I tried the scenario in HIVE-24224. I think it worked as expected:

+-------------------+--------------------+----------------+
| bz2tst2.sequence  |     bz2tst2.id     | bz2tst2.other  |
+-------------------+--------------------+----------------+
| 9                 | 20200315 X00 1356  | 123            |
| 17                | 20200315 X00 1357  | 123            |
+-------------------+--------------------+----------------+

The file in was :

 printf "offset,id,other\n9,\"20200315 X00 1356\",123\n17,\"20200315 X00 1357\",123\nrst,rst,rst" > data.csv

bzip2 -f data.csv 

hdfs dfs -put data.csv.bz2 hdfs://hostname:8020/warehouse/tablespace/external/hive/bz2tst2

Table is

+----------------------------------------------------+
|                   createtab_stmt                   |
+----------------------------------------------------+
| CREATE EXTERNAL TABLE `bz2tst2`(                   |
|   `sequence` int,                                  |
|   `id` string,                                     |
|   `other` string)                                  |
| ROW FORMAT SERDE                                   |
|   'org.apache.hadoop.hive.serde2.OpenCSVSerde'     |
| STORED AS INPUTFORMAT                              |
|   'org.apache.hadoop.mapred.TextInputFormat'       |
| OUTPUTFORMAT                                       |
|   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' |
| LOCATION                                           |
|   'hdfs://ayushsaxena-3.ayushsaxena.root.hwx.site:8020/warehouse/tablespace/external/hive/bz2tst2' |
| TBLPROPERTIES (                                    |
|   'bucketing_version'='2',                         |
|   'skip.footer.line.count'='1',                    |
|   'skip.header.line.count'='1',                    |
|   'transient_lastDdlTime'='1614334965')            |
+----------------------------------------------------+

Seems working, the UT was working. :-)