Make Spark readers function asynchronously for many small files.

[varun-lakhyani]

Feature Request / Improvement

In Spark readers, scan tasks are currently processed sequentially. The iteration logic in BaseReader.next() opens one task at a time, fully consumes it, and only then proceeds to the next task.

iceberg/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/BaseReader.java

Lines 131 to 145 in f49b2fd

	public boolean next() throws IOException {
	try {
	while (true) {
	if (currentIterator.hasNext()) {
	this.current = currentIterator.next();
	return true;
	} else if (tasks.hasNext()) {
	this.currentIterator.close();
	this.currentTask = tasks.next();
	this.currentIterator = open(currentTask);
	} else {
	this.currentIterator.close();
	return false;
	}
	}

With a large number of small files let's say hundreds or thousands of 5–10 KB files, this sequential task processing can lead to significant overhead.
Each task is opened and read independently, which may underutilize available CPU and I/O parallelism, especially on object stores with non-trivial per request latency.

Possible Improvement:
It may be beneficial to optionally allow Spark readers to process multiple small-file tasks concurrently, buffering rows into a shared iterator for downstream processing, while preserving the existing sequential behavior by default.

Query engine

None

Willingness to contribute

• I would be willing to contribute this improvement/feature with guidance from the Iceberg community

[RussellSpitzer]

I think this is a great idea, and we should probably see some POC's or design documents detailing possible solutions. A good test case to start with would be the compaction of a partition with 1000 small (100kb-1mb) files. Currently this crawls in our code base as all the files are compacted in a single thread.

[varun-lakhyani]

Thanks for the direction @RussellSpitzer.
I’ll work on a design document with a small POC and will keep this thread updated.