HIVE-29749: TypeInfoUtils fails to parse struct type strings with special characters in field names

[architjainjain]

HIVE-29749: Fix special characters in Iceberg partition column names with quoted identifiers

What changes were proposed in this pull request?

This PR fixes support for special characters (e.g. !@#%^&*()) in Iceberg partition column names
when using backtick-quoted identifiers (hive.support.quoted.identifiers=column).

One code change was made:

TypeInfoUtils.java — parseType() struct branch: When parsing a struct type string, the
struct field name reader previously accepted only a single token (which must have isType=true).
This caused an IllegalArgumentException when the field name contained special characters such as
!, @, #, %, ^, &, *, (, ) because the tokenizer splits those into individual
isType=false tokens. The fix changes the struct name reader to consume all consecutive tokens
until it encounters a structural delimiter (:, >, or ,), concatenating them into the full
field name. Characters that ARE structural delimiters (:, >, ,) remain unsupported in bare
field names, as they are used to delimit the struct type grammar itself.

---

Why are the changes needed?

When an Iceberg table is created with a backtick-quoted partition column containing special
characters (e.g. `gpa_!@#%^&*()`), the partition field name is stored in the Iceberg
metadata as a raw unquoted string (gpa_!@#%^&*()).

During an INSERT into such a table, Hive's ColumnStatsAutoGatherContext triggers automatic
column statistics gathering. This calls TypeInfoUtils.getTypeInfoFromTypeString() which calls
tokenize() and then parseType().

The tokenizer splits the struct type string (e.g. struct<gpa_!@#%^&*():double>) into tokens.
Since !, @, #, %, ^, &, *, (, ) are not isTypeChar(), each is emitted as its
own isType=false token. Then parseType() called expect("name") which only accepted a single
valid-type token for the field name — so encountering ! after gpa_ caused:

IllegalArgumentException: Error: ':' expected at position 137 of
'...struct<gpa_!@#%^&*():double>' but '!' is found.

This error surfaced as a SemanticException during the column stats rewrite, failing the INSERT.

---

Does this PR introduce any user-facing change?

Yes. Previously, executing an INSERT into an Iceberg table with special characters in partition
column names would fail with a SemanticException when hive.stats.autogather=true (the
default). After this fix, such operations succeed correctly.

Before:

set hive.support.quoted.identifiers=column;
create table t(`col` string) partitioned by(`gpa_!@#%^&*()` double) stored by iceberg;
insert into t select name, gpa from src;
-- ERROR: SemanticException: Line 1:X Argument type mismatch '...':
-- Error: ':' expected at position 137 of 'struct<gpa_!@#%^&*():double>' but '!' is found.

After:

-- Same DDL and INSERT succeeds without error.

---

How was this patch tested?

A new Q test QuotedIdentifier_1.q was added under iceberg/iceberg-handler/src/test/queries/positive/
and registered in testconfiguration.properties under iceberg.llap.query.files. It covers:

1. CREATE TABLE with special characters (!@#%^&*()) in both regular column names and the
   partition column name
2. INSERT into the table — directly exercises the
   ColumnStatsAutoGatherContext → TypeInfoUtils.getTypeInfoFromTypeString() → parseType()
   path that was the root cause of the bug
3. CREATE VIEW selecting from the special-character-named table and grouping by a
   special-character column

Test run command:

mvn test -pl itests/qtest-iceberg -Pitests \
  -Dtest=TestIcebergLlapLocalCliDriver \
  -Dqfile=QuotedIdentifier_1.q

To regenerate the expected output file:

mvn test -pl itests/qtest-iceberg -Pitests \
  -Dtest=TestIcebergLlapLocalCliDriver \
  -Dqfile=QuotedIdentifier_1.q \
  -Dtest.output.overwrite=true

[deniskuzZ]

please create a unit test, i think q test is an overkill

Subject: [PATCH] patch
---
Index: serde/src/test/org/apache/hadoop/hive/serde2/typeinfo/TestTypeInfoUtils.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/serde/src/test/org/apache/hadoop/hive/serde2/typeinfo/TestTypeInfoUtils.java b/serde/src/test/org/apache/hadoop/hive/serde2/typeinfo/TestTypeInfoUtils.java
--- a/serde/src/test/org/apache/hadoop/hive/serde2/typeinfo/TestTypeInfoUtils.java	(revision ef821c2a22ffdddc56e0280a33356878abbce04c)
+++ b/serde/src/test/org/apache/hadoop/hive/serde2/typeinfo/TestTypeInfoUtils.java	(date 1784639442922)
@@ -20,6 +20,7 @@
 
 
 
+import java.util.Arrays;
 import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
 import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
 import static org.junit.Assert.assertEquals;
@@ -125,4 +126,43 @@
       assertEquals("Failed for " + testCase.typeString, testCase.expectedScale, decimalType.getScale());
     }
   }
+
+  @Test
+  public void testStructFieldNameWithSpecialChars() {
+    TypeInfo typeInfo = TypeInfoUtils.getTypeInfoFromTypeString("struct<gpa_!@#$%^&*():double>");
+    StructTypeInfo structTypeInfo = (StructTypeInfo) typeInfo;
+    assertEquals(1, structTypeInfo.getAllStructFieldNames().size());
+    assertEquals("gpa_!@#$%^&*()", structTypeInfo.getAllStructFieldNames().get(0));
+    assertEquals(TypeInfoFactory.doubleTypeInfo, structTypeInfo.getAllStructFieldTypeInfos().get(0));
+  }
+
+  @Test
+  public void testStructFieldNameStartingWithSpecialChar() {
+    TypeInfo typeInfo = TypeInfoUtils.getTypeInfoFromTypeString("struct<!@#$%^&*()_age:int>");
+    StructTypeInfo structTypeInfo = (StructTypeInfo) typeInfo;
+    assertEquals(Arrays.asList("!@#$%^&*()_age"), structTypeInfo.getAllStructFieldNames());
+  }
+
+  @Test
+  public void testStructMultipleFieldsWithSpecialChars() {
+    TypeInfo typeInfo = TypeInfoUtils.getTypeInfoFromTypeString(
+        "struct<name:string,!@#$%^&*()_age:int,gpa_!@#$%^&*():double>");
+    StructTypeInfo structTypeInfo = (StructTypeInfo) typeInfo;
+    assertEquals(Arrays.asList("name", "!@#$%^&*()_age", "gpa_!@#$%^&*()"),
+        structTypeInfo.getAllStructFieldNames());
+  }
+
+  @Test
+  public void testEmptyStruct() {
+    TypeInfo typeInfo = TypeInfoUtils.getTypeInfoFromTypeString("struct<>");
+    StructTypeInfo structTypeInfo = (StructTypeInfo) typeInfo;
+    assertEquals(0, structTypeInfo.getAllStructFieldNames().size());
+  }
+
+  @Test
+  public void testStructTrailingComma() {
+    TypeInfo typeInfo = TypeInfoUtils.getTypeInfoFromTypeString("struct<a:int,>");
+    StructTypeInfo structTypeInfo = (StructTypeInfo) typeInfo;
+    assertEquals(Arrays.asList("a"), structTypeInfo.getAllStructFieldNames());
+  }
 }

```

[deniskuzZ]

+1, pending tests

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[architjainjain]

@deniskuzZ Test passed !!!