[FixtureCatalog][Dev][Scan Planning] `delete_file_references` bug when 1 puffin file contains blobs for separate data files

[Tishj]

Apache Iceberg version

None

Query engine

None

Please describe the bug 🐞

Spark sql (not actually runnable, just representative)

CREATE or REPLACE TABLE default.simple_v3_table
TBLPROPERTIES (
	'format-version' = '3',
	'write.delete.mode' = 'merge-on-read',
	'write.delete.format' = 'puffin',
	'write.update.mode' = 'merge-on-read'
)
AS SELECT * FROM parquet_file_view;

delete from default.simple_v3_table where c_custkey % 2 = 0;

Produced a puffin file like so:

File Size: 25484 bytes
Number of Blobs: 2

==================================================

Blob 0:
  Type: deletion-vector-v1
  Snapshot ID: -1
  Sequence Number: -1
  Fields: [2147483645]
  Properties: {'referenced-data-file': '/Users/thijs/DuckDBLabs/duckdb_iceberg/data/generated/iceberg/spark-local/default/simple_v3_table/data/00001-2-a1c10f5b-fff5-4f98-9020-207ede834bd8-0-00001.parquet', 'cardinality': '61440'}
  Data Length: 16432 bytes

Blob 1:
  Type: deletion-vector-v1
  Snapshot ID: -1
  Sequence Number: -1
  Fields: [2147483645]
  Properties: {'referenced-data-file': '/Users/thijs/DuckDBLabs/duckdb_iceberg/data/generated/iceberg/spark-local/default/simple_v3_table/data/00002-3-a1c10f5b-fff5-4f98-9020-207ede834bd8-0-00001.parquet', 'cardinality': '13560'}
  Data Length: 8232 bytes

Footer Payload:
{
  "blobs": [
    {
      "type": "deletion-vector-v1",
      "fields": [
        2147483645
      ],
      "snapshot-id": -1,
      "sequence-number": -1,
      "offset": 4,
      "length": 16432,
      "properties": {
        "referenced-data-file": "/Users/thijs/DuckDBLabs/duckdb_iceberg/data/generated/iceberg/spark-local/default/simple_v3_table/data/00001-2-a1c10f5b-fff5-4f98-9020-207ede834bd8-0-00001.parquet",
        "cardinality": "61440"
      }
    },
    {
      "type": "deletion-vector-v1",
      "fields": [
        2147483645
      ],
      "snapshot-id": -1,
      "sequence-number": -1,
      "offset": 16436,
      "length": 8232,
      "properties": {
        "referenced-data-file": "/Users/thijs/DuckDBLabs/duckdb_iceberg/data/generated/iceberg/spark-local/default/simple_v3_table/data/00002-3-a1c10f5b-fff5-4f98-9020-207ede834bd8-0-00001.parquet",
        "cardinality": "13560"
      }
    }
  ],
  "properties": {
    "created-by": "Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"
  }
}

Which has 2 blobs, for separate referenced-data-file references.

But the Scan Planning API returns:

{
	"status": "completed",
	"plan-id": "sync-86b10b50-5c2c-487f-8768-8fceebe30ffa",
	"delete-files": [
		{
			"spec-id": 0,
			"content": "position-deletes",
			"file-path": "s3://warehouse/default/simple_v3_table/data/00000-6-0d2bb4cd-d1bb-462a-ac95-2f85d31b2724-00001-deletes.puffin",
			"file-format": "puffin",
			"partition": [],
			"file-size-in-bytes": 25366,
			"record-count": 61440,
			"referenced-data-file": "s3://warehouse/default/simple_v3_table/data/00001-2-e48509dd-3593-4cff-b3c0-56dada5eb013-0-00001.parquet",
			"content-offset": 4,
			"content-size-in-bytes": 16432
		},
		{
			"spec-id": 0,
			"content": "position-deletes",
			"file-path": "s3://warehouse/default/simple_v3_table/data/00000-6-0d2bb4cd-d1bb-462a-ac95-2f85d31b2724-00001-deletes.puffin",
			"file-format": "puffin",
			"partition": [],
			"file-size-in-bytes": 25366,
			"record-count": 13560,
			"referenced-data-file": "s3://warehouse/default/simple_v3_table/data/00002-3-e48509dd-3593-4cff-b3c0-56dada5eb013-0-00001.parquet",
			"content-offset": 16436,
			"content-size-in-bytes": 8232
		}
	],
	"file-scan-tasks": [
		{
			"data-file": {
				"spec-id": 0,
				"content": "data",
				"file-path": "s3://warehouse/default/simple_v3_table/data/00001-2-e48509dd-3593-4cff-b3c0-56dada5eb013-0-00001.parquet",
				"file-format": "parquet",
				"partition": [],
				"file-size-in-bytes": 6596897,
				"record-count": 122880,
				"split-offsets": [
					4
				],
				"sort-order-id": 0,
				"first-row-id": 0
			},
			"delete-file-references": [
				1
			],
			"residual-filter": true
		},
		{
			"data-file": {
				"spec-id": 0,
				"content": "data",
				"file-path": "s3://warehouse/default/simple_v3_table/data/00002-3-e48509dd-3593-4cff-b3c0-56dada5eb013-0-00001.parquet",
				"file-format": "parquet",
				"partition": [],
				"file-size-in-bytes": 1458262,
				"record-count": 27120,
				"split-offsets": [
					4
				],
				"sort-order-id": 0,
				"first-row-id": 122880
			},
			"delete-file-references": [
				1
			],
			"residual-filter": true
		}
	]
}

(note that the paths are different, I wasn't able to inspect the puffin file from the rest catalog, so I reran the query against a local catalog in order to be able to inspect the puffin file)

Both file-scan-tasks entries reference the same delete-files index, but they should be separate

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Tishj]

Oh huh, printing the JSON body, I see that the PositionalDeleteFile contains a referenced-data-file which has the correct value, but isn't part of the spec.
I was at first going to make an issue, that referenced-data-file should be added to PositionalDeleteFile, but then I dug further into the spec to find that it isn't strictly necessary because of delete-file-references.
We could piece together the referenced-data-file from that, if it were correct.