HIVE-26980: Creation of materialized view stored by Iceberg fails if …

[kasakrisz]

…source table has tinyint column

What changes were proposed in this pull request?

Some Hive column datatypes are currently not supported by Iceberg. In case of CTAS statements and materialized views Hive converts some of the source table column types to a compatible Iceberg column type.
For the conversion a select operator is generated. The number of input and output columns has to be the same. The number of output columns also depends on dynamic partitioning but in case of Iceberg target table partitioning is handled by the storage handler so it should be ignored.

Why are the changes needed?

To support partitioned materialized view stored by iceberg and to support ctas statements which create tables stored by Iceberg but the source table/query has a column datatype which is not supported by Iceberg.

Does this PR introduce any user-facing change?

No. But such statements runs successfuly.

How was this patch tested?

mvn test -Dtest.output.overwrite -Dtest=TestIcebergCliDriver -Dqfile=ctas_iceberg_orc.q,mv_iceberg_partitioned_orc.q,mv_iceberg_partitioned_orc2.q -pl itests/qtest-iceberg -Piceberg -Pitests

[deniskuzZ]

@kasakrisz, could you please elaborate on if condition? what if destType=DEST_LOCAL_FILE?
IsQuery() would be always false for CTAS and MV

[kasakrisz]

• IIUC when destType=DEST_LOCAL_FILE we are inserting into a local directory and no conversion is required since the target is not an actual table. Example.

  hive/ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q

  Line 2 in 0a5558d

  insert overwrite local directory '../../data/files/local_src_table_1'

  
  I extended the if condition with checking whether the destinationTable is null and does it has a storage handler.

• yes, isQuery should be false for ctas and mv
  

  hive/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

  Lines 2443 to 2444 in 0a5558d

  	if (qb.isCTAS() || qb.isMaterializedView()) {
  	qb.setIsQuery(false);

[deniskuzZ]

👍

[InvisibleProgrammer]

Hi, @kasakrisz !

Thank you for fixing that issue. I have faced with the exact same in https://issues.apache.org/jira/browse/HIVE-26977?

Could you please add that qtest as well to validate your fix fixes the other issue as well. I already tested it on my computer and it worked.

set iceberg.mr.schema.auto.conversion=true;
set hive.vectorized.execution.enabled=true;
set hive.explain.user=false;

drop table if exists source_table;
create external table source_table
(
    id char(16)
);

insert into source_table values ('ID_1');
insert into source_table values ('ID_2');

drop table if exists target_table;

explain
create table target_table
stored by iceberg
stored as orc
as select * from source_table;

create table target_table
stored by iceberg
stored as orc
as select * from source_table;

select count(*) from target_table;

Thank you,
Zsolt

[kasakrisz]

@InvisibleProgrammer
Thanks for validating this patch. I checked your test case and found that the test iceberg/iceberg-handler/src/test/queries/positive/ctas_iceberg_orc.q updated by this patch covers your scenario. The only difference is that you explicitly enabled vectorization but that is enabled by default in PTests.

[InvisibleProgrammer]

The only difference is that you explicitly enabled vectorization but that is enabled by default in PTests.

Sad to hear. That means, if we run the test on the build server, we get a different result than running it on our machine. Could you please add that vectorization test to the setting to get the exact same behavior?

[deniskuzZ]

LGTM +1, pending tests

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information