Skip to content

Remove unsupported dataType cases in VeloxBackend#11767

Closed
VvanFalleaves wants to merge 2 commits into
apache:mainfrom
VvanFalleaves:main
Closed

Remove unsupported dataType cases in VeloxBackend#11767
VvanFalleaves wants to merge 2 commits into
apache:mainfrom
VvanFalleaves:main

Conversation

@VvanFalleaves
Copy link
Copy Markdown

Removed handling for StructType, ArrayType, and MapType in dataType matching.

DROP TABLE IF EXISTS hive_support;
CREATE TABLE IF NOT EXISTS hive_support (
  int_col INT,
  array_col ARRAY<INT>,
  map_col MAP<STRING, INT>,
  struct_col STRUCT<field1: INT, field2: STRING, field3: BOOLEAN>
)
PARTITIONED BY (part_col INT)
STORED AS PARQUET
TBLPROPERTIES (
  'parquet.compression' = 'ZSTD',
  'parquet.compression.zstd.level' = '5',
  'parquet.enable.dictionary' = 'false'
);
WITH number_seq AS (
  SELECT CAST(id AS INT) AS id_int
  FROM range(1, 50000000, 1, 100)
)
INSERT OVERWRITE TABLE hive_support
PARTITION (part_col) 
SELECT
  id_int AS int_col,
  ARRAY(
    id_int % 10,
    id_int % 100,
    id_int % 1000
  ) AS array_col,
  MAP(
    CONCAT('key_', CAST(id_int % 5 AS STRING)), id_int % 100,
    CONCAT('key_', CAST((id_int + 1) % 5 AS STRING)), (id_int + 1) % 100
  ) AS map_col,
  named_struct(
    'field1', id_int,
    'field2', CONCAT('str_', CAST(id_int AS STRING)),
    'field3', id_int % 2 = 0
  ) AS struct_col,
  id_int % 10 AS part_col
FROM number_seq
SORT BY id_int;

The following error is reported during the execution:

26/03/10 20:50:46 WARN GlutenFallbackReporter: Validation failed for plan: WriteFiles[QueryId=4], due to: Unsupported native write: Found unsupported type:ArrayType,MapType,StructType.

The Velox version downloaded by gluten in the branch-1.3 is https://github.com/oap-project/velox/tree/gluten-1.3.0. In this case, HiveDataSink::appendData does not support PARTITIONED BY and complex types. However, in newer Velox, HiveDataSink::appendData method has been updated, but gluten does not fix it.

Removed handling for StructType, ArrayType, and MapType in dataType matching.
@github-actions github-actions Bot added the VELOX label Mar 16, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 1, 2026

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions Bot added the stale stale label May 1, 2026
@github-actions
Copy link
Copy Markdown

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

@github-actions github-actions Bot closed this May 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant