Undetermined behavior when fetching from iceberg table

[liurenjie1024]

Apache Iceberg version

1.6.1

Query engine

Spark

Please describe the bug 🐞

1. Run mkdir warehouse/default to create dir
2. Download attachment for table data
3. Run tar -xzf iceberg.tar.gz && mv tmp/* /tmp/ to put table data into dir
4. Start spark shell

/home/ubuntu/Apps/spark-3.5.4-bin-hadoop3/bin/spark-sql \
    --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.2\
    --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.spark_catalog.type=hadoop \
    --conf spark.sql.catalog.spark_catalog.warehouse=/tmp/spark-warehouse-28428

5. Run count(*) it shows 217 rows.

spark-sql (default)> select count(*) from tmp_table_gw0_754291685_0;
217
Time taken: 2.33 seconds, Fetched 1 row(s)

6. Run select * from tmp_table_gw0_754291685_0 it reports fetching rows not 750 rows.

spark-sql (default)> select * from tmp_table_gw0_754291685_0;
...
Time taken: 0.213 seconds, Fetched 327 row(s)

The oddest part is that the difference only happens when the first time you run select * after select count(*), after that when you run select * everything is back to normal.

I tested againts 1.6.1, 1.7.2, and 1.9.2, buth all failed.

iceberg.tar.gz

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[RussellSpitzer]

This table has Equality, Position and Data parquet files, and lot of them (1447)! Do you have any hints on how the table was generated? I wanted to start walking through it in the debugger but it's got a lot of pieces.

[RussellSpitzer]

So I loaded this table and I got a completely different number of rows - 2048L
It was also stable. I tested this by loading in a integration test within the Iceberg repo

I did register the table metadata.json and didn't load it via HadoopCatalog

[liurenjie1024]

So I loaded this table and I got a completely different number of rows - 2048L It was also stable. I tested this by loading in a integration test within the Iceberg repo

I did register the table metadata.json and didn't load it via HadoopCatalog

Oh, sorry, I made a mistake by changing the root dir. We have more than 1k files since we have many partitions, and the tests files are generated by our own test framework. Let me try to refine the tests.

[liurenjie1024]

I've updated the data file and issue description to reflect latest changes, now it only contains several files and deletion files, but the error still happens.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[liurenjie1024]

It's a bug, don't close it.

[pvary]

@liurenjie1024: You an add not-stale flag

[RussellSpitzer]

It's the SparkExecutorCache, We are caching the delete file with different orderings. In the example table one of the eq deletes has [2, 3] which gets cached as [3, 2] then when it's re-used it's assumed it's [2, 3] which breaks everything. Will post some more details in a bit.