[SUPPORT] Flink cant read metafield '_hoodie_commit_time'

[LinMingQiang]

Tips before filing an issue
Flink ： 1.15.1
Hudi ： master

CREATE TABLE ITTestMetaField(
_hoodie_commit_time STRING,
id STRING PRIMARY KEY NOT ENFORCED
)
WITH (
'index.type'='BUCKET',
'payload.class'='org.apache.hudi.common.model.PartialUpdateAvroPayload',
'precombine.field'='ts',
'table.type' = 'MERGE_ON_READ'
)

Stacktrace

Caused by: java.lang.NullPointerException
	at org.apache.flink.formats.parquet.vector.reader.AbstractColumnReader.readToVector(AbstractColumnReader.java:160)
	at org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader.nextBatch(ParquetColumnarRowSplitReader.java:312)
	at org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader.ensureBatch(ParquetColumnarRowSplitReader.java:288)
	at org.apache.hudi.table.format.cow.vector.reader.ParquetColumnarRowSplitReader.reachedEnd(ParquetColumnarRowSplitReader.java:267)
	at org.apache.hudi.table.format.ParquetSplitRecordIterator.hasNext(ParquetSplitRecordIterator.java:42)
	at org.apache.hudi.table.format.mor.MergeOnReadInputFormat$BaseFileOnlyFilteringIterator.hasNext(MergeOnReadInputFormat.java:563)
	at org.apache.hudi.table.format.mor.MergeOnReadInputFormat.reachedEnd(MergeOnReadInputFormat.java:264)
	at org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:89)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:332)

There's no problem querying other metafields

[LinMingQiang]

[LinMingQiang]

[danny0405]

Seems a bug, could you fire a PR and fix it?

[voonhous]

We've encountered similar issues around this code recently.

we can't seem to reproduce your issue, is it possible to provide a minimal example of your table so i can trigger this bug locally to see if the issue we are encountering is the same?

i.e. possible to share your hudi table file?

[LinMingQiang]

id STRING PRIMARY KEY NOT ENFORCED,
name STRING,
age bigint,
ts string,
`par` STRING 
) PARTITIONED BY (`par`) 
 WITH (
'index.type'='BUCKET',
'payload.class'='org.apache.hudi.common.model.PartialUpdateAvroPayload',
'precombine.field'='ts',
'changelog.enabled'='false',
'compaction.delta_commits'='1',
'compaction.async.enabled'='true',
'write.tasks'='1',
'hoodie.bucket.index.num.buckets'='1',
'compaction.schedule.enable'='true',
'table.type' = 'MERGE_ON_READE',
'hoodie.datasource.write.hive_style_partitioning'='true',
'hive_sync.partition_extractor_class'='org.apache.hudi.hive.HiveStylePartitionValueExtractor',
'path' = 'file:///Users/hunter/workspace/hudipr/HUDI-6032/hudi-debug/hudi-debug-flink/target/HUDI_6032',
'connector' = 'hudi'
)


insert into HUDI_6032(id, name,age, ts, par)  values('id1','name1',1, 'ts1','par1'),('id1','name2',2, 'ts2','par1')


CREATE TABLE HUDI_6032(
_hoodie_commit_time STRING,
id STRING PRIMARY KEY NOT ENFORCED 
) PARTITIONED BY (id) 
 WITH (
...
)

[Coco0201]

现在可以读取_hoodie_commit_time了么?flink1.13读取hudi0.13.1还是报错Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: SQL parse failed. Encountered "_hoodie_commit_time" @LinMingQiang

[danny0405]

Did you declare the _hoodie_commit_time as a schema field in your table?

[Coco0201]

Did you declare the _hoodie_commit_time as a schema field in your table?

Yes.My java codes as follows:
tabEnv.executeSql(“create table cdc_hudi(_hoodie_commit_time string) with (...)”)

[Coco0201]

Did you declare the _hoodie_commit_time as a schema field in your table?

I found the comma which is in the DDL of my flink table was forgotten.So there is no problem while reading metafields.