[SUPPORT] Col-stats MDT mis-records nullCount=valueCount when Parquet column stats are absent (e.g. due to long string values), causing silent zero-row query results

[linliu-code]

Describe the problem you faced

When Hudi's column-stats MDT writes a file's stats and the underlying Parquet file omits column-chunk statistics (which parquet-mr does when at least one value exceeds the stats truncation threshold, e.g. a string > ~1 KB), Hudi mis-records the file's stats as nullCount = valueCount — semantically "all N values in this file are NULL."

On query, Hudi's data-skipping reader uses this MDT record and concludes "no row in this file can match any non-null predicate," so the file is silently skipped. Equality predicates against any column of that file (including short, non-null values in the same file) return zero rows.

This is silent data loss for any Hudi 1.1.1 table where:

• col-stats MDT is enabled (the 1.1.1 default), and
• at least one row in the file has a value larger than parquet-mr's stats truncation threshold for its column type (~1 KB for binary/string).

The bug is unrelated to RLI, bloom filters, partition stats, or NaN handling. It triggers purely on the col-stats writer path against any Parquet file where the writer omitted column-chunk stats.

To Reproduce

Single-file pyspark script. Run via spark-submit with the Hudi 1.1.1 Spark 3.4 bundle:

export HUDI_BUNDLE=/path/to/hudi-spark3.4-bundle_2.12-1.1.1.jar
spark-submit \
  --master 'local[2]' \
  --jars "$HUDI_BUNDLE" \
  --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
  --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar \
  --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \
  --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \
  repro.py

import os, shutil, tempfile, glob
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, IntegerType, LongType
from pyspark.sql import functions as F

PATH = tempfile.mkdtemp(prefix="hudi_colstats_long_string_")
spark = SparkSession.builder.appName("repro").config("spark.sql.shuffle.partitions","1").getOrCreate()
spark.sparkContext.setLogLevel("WARN")

schema = StructType([
    StructField("rk", StringType(), False),
    StructField("val", IntegerType(), True),
    StructField("ts", LongType(), False),
])

# Default Hudi 1.1.1 options: MDT is enabled, col-stats is on by default
opts = {
    "hoodie.table.name": "repro",
    "hoodie.datasource.write.recordkey.field": "rk",
    "hoodie.datasource.write.partitionpath.field": "",
    "hoodie.datasource.write.precombine.field": "ts",
    "hoodie.datasource.write.table.type": "COPY_ON_WRITE",
    "hoodie.metadata.enable": "true",
    "hoodie.parquet.small.file.limit": "0",
}

# Two rows: one short rk, one above the parquet-mr stats truncation threshold
rows = [("Y", 1, 1), ("Y" * 4096, 2, 1)]
spark.createDataFrame(rows, schema).write.format("hudi").options(**opts).mode("overwrite").save(PATH)

# Hudi reader
hudi = spark.read.format("hudi").load(PATH)
print(f"Hudi count: {hudi.count()}")
print(f"Hudi WHERE rk='Y': {hudi.where(F.col('rk') == F.lit('Y')).count()}  (expected 1)")

# Same file via raw Spark Parquet reader
pq = sorted(glob.glob(f"{PATH}/*.parquet"))[0]
print(f"Raw Parquet WHERE rk='Y': {spark.read.parquet(pq).where(F.col('rk') == F.lit('Y')).count()}  (expected 1)")

# Inspect MDT col-stats
mdt = os.path.join(PATH, ".hoodie", "metadata")
spark.read.format("hudi").load(mdt).createOrReplaceTempView("vmdt")
print("\nMDT col-stats for rk:")
spark.sql("""
  SELECT ColumnStatsMetadata.fileName AS f,
         ColumnStatsMetadata.minValue, ColumnStatsMetadata.maxValue,
         ColumnStatsMetadata.nullCount, ColumnStatsMetadata.valueCount
  FROM vmdt WHERE type=3 AND ColumnStatsMetadata.columnName='rk'
""").show(truncate=False)

Expected behavior

WHERE rk = 'Y' should return 1 row from both the Hudi reader and the raw Parquet reader — the short row IS in the file.

The MDT col-stats record for rk should reflect that statistics are unavailable (e.g., min=null, max=null with nullCount left unset) so the data-skipping reader treats the file as "must scan." It should NOT claim nullCount=2, valueCount=2.

Actual behavior

Hudi count: 2
Hudi WHERE rk='Y': 0                              <<< BUG (expected 1)
Raw Parquet WHERE rk='Y': 1                        (expected 1)

MDT col-stats for rk:
+--------------------+--------+--------+---------+-----------+
|f                   |minValue|maxValue|nullCount|valueCount |
+--------------------+--------+--------+---------+-----------+
|<file_id>.parquet   |NULL    |NULL    |2        |2          |
+--------------------+--------+--------+---------+-----------+

The MDT claims both rows are null. On query, Hudi's data-skipping reads this record, concludes "no row in this file is non-null, hence no row matches rk = 'Y'", and skips the file.

The same Parquet file read via spark.read.parquet(...) returns 1 row correctly — the data IS in the file; the bug is purely in Hudi's col-stats writer/reader path.

Root cause

hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java, method readColumnStatsFromMetadata, line 320 (release-1.1.1):

return HoodieColumnRangeMetadata.create(
    filePath,
    columnChunkMetaData.getPath().toDotString(),
    convertToNativeJavaType(..., stats.genericGetMin(), valueMetadata),
    convertToNativeJavaType(..., stats.genericGetMax(), valueMetadata),
    // NOTE: In case when column contains only nulls Parquet won't be creating
    //       stats for it instead returning stubbed (empty) object. In that case
    //       we have to equate number of nulls to the value count ourselves
    stats.isEmpty() ? columnChunkMetaData.getValueCount() : stats.getNumNulls(),   // <-- BUG
    columnChunkMetaData.getValueCount(),
    columnChunkMetaData.getTotalSize(),
    columnChunkMetaData.getTotalUncompressedSize(),
    valueMetadata);

The intent in the comment is correct, but stats.isEmpty() returns true in two semantically-different cases:

Parquet condition	stats.isEmpty()	stats.isNumNullsSet()	Correct nullCount
Column has all-null values	true	true	valueCount
Column has at least one value too large for stats (parquet-mr omits stats entirely)	true	false	unknown — should not be valueCount

Hudi's current code treats both the same way, fabricating nullCount = valueCount in the second case. The data-skipping reader then correctly applies that record and skips the file, producing zero rows.

parquet-mr exposes Statistics.isNumNullsSet() exactly to distinguish these cases.

Suggested patch

// in readColumnStatsFromMetadata, replace line 320 with:
long nullCount;
if (stats.isEmpty()) {
    if (stats.isNumNullsSet()) {
        // Genuine "all values null" case: parquet recorded num_nulls explicitly
        // and chose not to write min/max for an all-null column.
        nullCount = columnChunkMetaData.getValueCount();
    } else {
        // "Statistics absent" case (e.g. a value exceeded the parquet-mr stats
        // truncation threshold). We have no information about nulls. Use the
        // unset sentinel so the data-skipping reader treats this column's
        // stats as unavailable and falls back to scanning.
        nullCount = -1;  // or whatever sentinel HoodieColumnRangeMetadata
                          // recognizes as "unknown"; see also the data-skipping
                          // reader's interpretation of nullCount.
    }
} else {
    nullCount = stats.getNumNulls();
}

The data-skipping reader on the query side may also need a small change to recognize the "unknown nullCount" sentinel and skip stats-based pruning for that file × column (rather than treating nullCount of -1 as "all values matching the predicate are absent").

Cross-reference

This is the same code-surface bug class as #18754 (NaN in numeric columns silently records min=0.0, max=0.0). Both arise from ParquetUtils.readColumnStatsFromMetadata mishandling exceptional parquet-mr Statistics states. A single PR could fix both:

• NaN case ([SUPPORT] Single NaN value corrupts col-stats and silently drops query rows across whole table (regression in 1.1.1) #18754): convertToNativeJavaType(..., stats.genericGetMin(), ...) returns 0.0 when stats are absent due to NaN. The fix should produce null min/max and unset null-count.
• Long-value case (this issue): stats.isEmpty() ? valueCount : getNumNulls() returns valueCount when stats are absent due to truncation. The fix is shown above.

Environment Description

• Hudi version: 1.1.1 (current GA from Maven Central; hudi-spark3.4-bundle_2.12-1.1.1.jar)
• Spark version: 3.4.3
• Hadoop version: 3 (bundled with Spark)
• Storage: local FS — bug is in the col-stats writer/reader and is storage-independent
• Running on Docker?: optional

Stacktrace

n/a — no exception; the bug is silent.

[linliu-code]

Update — exact threshold confirmed via parquet-mr source + empirical bisection

Following up on the root cause. I bisected the trigger condition end-to-end. The exact threshold is:

Parquet drops the column-chunk statistics (and therefore Hudi mis-records nullCount=valueCount) when min.length() + max.length() >= 4096 bytes.

This is from parquet-mr 1.12.3:

// org/apache/parquet/format/converter/ParquetMetadataConverter.java:149
public static final long MAX_STATS_SIZE = 4096; // limit stats to 4k

// ParquetMetadataConverter.toParquetStatistics:707
if (!stats.isEmpty() && withinLimit(stats, truncateLength)) {
    formatStats.setNull_count(stats.getNumNulls());  // null_count ONLY set inside this guard
    ...
}
return formatStats;  // empty formatStats returned when guard fails

// org/apache/parquet/column/statistics/BinaryStatistics.java:106
public boolean isSmallerThan(long size) {
    return !hasNonNullValue() || ((min.length() + max.length()) < size);
}

When withinLimit returns false, formatStats stays empty — including null_count. On read, parquet-mr reconstructs a Java Statistics<?> where isEmpty()=true, isNumNullsSet()=false, hasNonNullValue()=false. Hudi's line 320 then ternaries on isEmpty()=true and records nullCount = valueCount.

Empirical confirmation of the exact boundary

Same test rig as the original repro, varying only min and max lengths:

min_len	max_len	sum	Result
1	2000	2001	stats kept
1	4094	4095	stats kept (tightest pass)
1	4095	4096	stats DROPPED (boundary)
1	4100	4101	stats dropped
1	4096	4097	stats dropped (original repro)

The < (strict-less-than) in isSmallerThan is what makes sum=4096 exactly already trip the drop.

Side-finding: configuration overrides don't propagate to Hudi's writer

parquet-mr's default ParquetProperties.DEFAULT_STATISTICS_TRUNCATE_LENGTH = 2147483647 (Integer.MAX_VALUE), effectively disabling truncation, which makes isSmallerThanWithTruncation(4096, MAX_VALUE) also fail for any value over ~4 KB.

I attempted to mitigate from the Spark/user side by overriding spark.sql.parquet.statisticsTruncateLength and parquet.statistics.truncate.length (both via spark.conf.set(...) and via Hadoop conf). Neither override propagated to Hudi's Parquet writer — stats were still dropped at the same threshold. Users have no working knob to avoid this on the write side until the line-320 fix lands. Worth surfacing in the same PR.

Real-world impact characterization

A single row in a file whose target column has a value contributing min+max >= 4 KB is enough to trip the bug. Common real-world cases that hit this:

• A description, notes, comment, payload, error_message, or stack_trace column where any one row exceeds ~4 KB.
• A serialized-JSON column where one row is a larger document.
• A composite primary key (tenant_id + entity_path + uuid + ...) where one tenant's key is long.
• A URL column or similar where any one value exceeds ~4 KB.

Once such a row is written into a file, every equality query against any column of that file silently returns zero rows.

Suggested patch (refined)

Both the line-320 and the corresponding NaN-side fix (convertToNativeJavaType returning fabricated 0.0 — see #18754) come from the same misinterpretation: parquet-mr's empty Statistics is treated as "all-null column." The fix needs to distinguish "stats genuinely absent" via isNumNullsSet():

// hudi-hadoop-common/.../ParquetUtils.java readColumnStatsFromMetadata, around line 320
long nullCount;
Object minVal = null;
Object maxVal = null;
if (stats.isEmpty()) {
    if (stats.isNumNullsSet()) {
        // genuine all-null column: parquet recorded num_nulls and chose
        // not to write min/max for an all-null column
        nullCount = columnChunkMetaData.getValueCount();
        // min/max remain null
    } else {
        // stats unavailable (e.g. min.length + max.length >= 4096 in parquet-mr's
        // ParquetMetadataConverter.toParquetStatistics). Record "stats unknown" so
        // the data-skipping reader treats this column's stats as unavailable.
        nullCount = -1L;   // sentinel; reader must interpret as "no skip info"
        // min/max remain null
    }
} else {
    nullCount = stats.getNumNulls();
    minVal = convertToNativeJavaType(columnChunkMetaData.getPrimitiveType(),
                                     stats.genericGetMin(), valueMetadata);
    maxVal = convertToNativeJavaType(columnChunkMetaData.getPrimitiveType(),
                                     stats.genericGetMax(), valueMetadata);
}
return HoodieColumnRangeMetadata.create(
    filePath,
    columnChunkMetaData.getPath().toDotString(),
    (Comparable) minVal,
    (Comparable) maxVal,
    nullCount,
    columnChunkMetaData.getValueCount(),
    columnChunkMetaData.getTotalSize(),
    columnChunkMetaData.getTotalUncompressedSize(),
    valueMetadata);

Plus reader-side: the data-skipping evaluator must check for the "unknown" sentinel (or equivalent state) on HoodieColumnRangeMetadata and bypass stats-based pruning for that file × column.

Reproduction is unchanged from the original report; the threshold info is just useful for triage and for users to scope their existing tables (SELECT COUNT(*) FROM tbl WHERE LENGTH(target_col) >= 4095 is a rough first-pass check).

[linliu-code]

PR up for review: #18792

Same PR also fixes #18754. The suggested patch in this issue (line 320 of ParquetUtils.readColumnStatsFromMetadata) is implemented here, using Statistics.isNumNullsSet() to distinguish all-null columns from columns where stats are truly absent.