[Bug] REST server-side scan planning retains fetched FileScanTasks indefinitely

[chenwyi2]

Apache Iceberg version

1.10.1

Query engine

Starrocks

Please describe the bug 🐞

Query engine

REST Catalog server-side scan planning.

Please describe the bug

The reference implementation of REST server-side scan planning keeps all planned
FileScanTask objects in the singleton InMemoryPlanningState, even after clients
have successfully fetched every plan task.

As a result, repeated successful scans cause retained heap usage to grow
approximately linearly with the number of planned files.

The retained state is stored in the following maps:

Map<String, List> planTaskToFileScanTasks;
Map<String, String> planTaskToNext;
Map<String, PlanStatus> asyncPlanningStates;

The normal successful fetch path only reads these maps:

`List fileScanTasks =
IN_MEMORY_PLANNING_STATE.fileScanTasksForPlanTask(planTask);

return FetchScanTasksResponse.builder()
.withFileScanTasks(fileScanTasks)
.withPlanTasks(IN_MEMORY_PLANNING_STATE.nextPlanTask(planTask))
.withSpecsById(table.specs())
.build();`

Neither fileScanTasksForPlanTask nor nextPlanTask removes the fetched entries.
The initial task returned by synchronous planTableScan or asynchronous
fetchPlanningResult is also retained.
Currently, the state is released only when cancelPlanning calls
InMemoryPlanningState.cancelPlan.
However, according to the REST Catalog OpenAPI specification, cancellation is not
required after scan tasks have been fetched for every plan task. This implies that a
successful fetch lifecycle must eventually release fetched task state without
requiring an explicit cancellation request.
Specification:
https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml
Relevant implementation:
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/CatalogHandlers.java
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/InMemoryPlanningState.java
Impact
In a long-running REST catalog deployment based on this implementation, we observed:

• Heap usage continuously increasing after successful scans

• Tens of millions of retained FileScanTask objects

• More than 20 GB of heap retained by planning state

• Full GC unable to reclaim the objects

• No active scans corresponding to most retained tasks

• No cancellation requests because all plan tasks had already been fetched successfully

In one observation window, approximately 25 million FileScanTask objects were
retained after about 22 hours of operation.
This may eventually cause excessive GC pressure or an OutOfMemoryError.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time