Looking for efficient way to query specific partitions from metadata

[emilie-wang]

Feature Request / Improvement

Hi pyiceberg community,

Is there a more efficient way than the API table.inspect.partitions() to query specific partitions?

The table.inspect.partitions() seems expensive as it reads all manifest files, especially when the table has many partitions, while we only need to check a few recent partitions.

I have a few questions about this use case:

• Is there any more efficient way to do this that I am missing?
• If not, can I propose something like partitions_filtered(partition_filter) to get the specific partition by reusing the work

  iceberg-python/pyiceberg/table/__init__.py

  Line 1918 in 5ee5eea

  def plan_files(self) -> Iterable[FileScanTask]:

Thanks!

[starhel]

I came across this by accident, but in my workflow it would also be useful to have an option to check whether a specific partition already exists.

[Fokko]

@emilie-wang I think that's reasonable to add. We could add a predicate as an argument that defaults to ALWAYS_TRUE.