Skip to content

Snapshot Changes should have a Streaming Interface #15659

@RussellSpitzer

Description

@RussellSpitzer

Feature Request / Improvement

Currently SnapshotChanges only provides a caching interface that eagerly materializes all file changes into in-memory lists and returns Iterable. There are use cases where returning a streaming CloseableIterable would be preferable see #13556 for example.

I think we probably should just add CloseableIterable interfaces to SnapshotChanges as well and just have the caching calls wrap them.

Query engine

None

Willingness to contribute

  • I can contribute this improvement/feature independently
  • I would be willing to contribute this improvement/feature with guidance from the Iceberg community
  • I cannot contribute this improvement/feature at this time

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementPR that improves existing functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions