PyIceberg Near-Term Roadmap

[kevinjqliu]

Feature Request / Improvement

PyIceberg 0.7.0

The main objective of 0.7.0 is to have partitioned writes (non-exhaustive list :)

• Support for merge-into / upsert: Merge into / Upsert #402
  • Support partitioned appends: Support Appends with TimeTransform Partitions #784
  • Support partial deletes: Support partial deletes #569
• Support for parallelizing writes: Parallel Table.append #428 Faster ingestion from Parquet #346
  • Support parallelized writes: Bin-pack Writes Operation into multiple parquet files, and parallelize writing WriteTasks #444
• Support table_exists on catalog: check if table exist #406 Add table_exists method to the Catalog #507, fixed in Add table_exists method to Catalog #512
• Metadata tables: Add metadata tables #511
  • Files assigned to @Gowthami03B, PR in Add Files metadata table #614
  • Snapshots assigned to @Fokko in Add Snapshots table metadata #524
  • History assigned to @ndrluis
  • Metadata log entries @kevinjqliu (issue in [feat request] Add metadata_log_entries metadata table #594): Metadata Log Entries metadata table #667
  • Manifests @geruh: PR in Add manifests metadata table #717
  • Partitions assigned to @syun64 (issue in Support get partition table with filter #24)
  • References assigned to @geruh in Add Refs metadata table #602
  • Entries assigned to @Fokko in Add entries metadata table #551
• Manifest read/write improvements:
  • Implement rolling writes: Implement rolling manifest-writers #596 feat: add RollingManifestWriter #650
  • Caching of manifests: Implement caching of manifest-files #595
• Incremental append scan: Incremental Append Scan #533

PyIceberg 0.8.0

• Table maintenance:
  • Snapshot expiration
  • Metadata rewrites
  • Compaction
  • Delete orphan files
• Catalogs:
  • Snowflake catalog: Add Snowflake catalog #687
  • Nessie catalog: Support Nessie catalog #19
  • BigLake catalog: Add support for BigLake Metastore #651
• ORC Support: ORC file format support #20
• Branch Support: Support writing to a branch #306
• Tag Support: Support creating tags  #573
• Write with Sort Order Support writing to a table with sort-order #271

PyIceberg 1.0.0

Long-term goals:

• Support Griffe to detect breaking API changes Use griffe to find breaking changes #334: detect breaking changes #394
• Implement Arrow dataset: Expose PyIceberg table as PyArrow Dataset #30
• Support table maintenance operations: Support to optimize, analyze tables and expire snapshots, remove orphan files #31
• Add View support
• Add Puffin support
• Support engine integrations (DuckDB/Daft/Polars/Ray)

[corleyma]

@kevinjqliu @Fokko Where would something like the Iceberg Spark create_changelog_view procedure fit in this roadmap? Is that something that might be tackled as part of the other procedures under table maintenance, or is it likely to come later (1.0.0), or not at all in PyIceberg?

[Fokko]

Sorry for the late reply, I was touching grass.

@kevinjqliu @Fokko Where would something like the Iceberg Spark create_changelog_view procedure fit in this roadmap? Is that something that might be tackled as part of the other procedures under table maintenance, or is it likely to come later (1.0.0), or not at all in PyIceberg?

Thanks for bringing this up @corleyma 🙌 Some related work is being done in #533 and I think PyIceberg should definitely support something like that.

@kevinjqliu @Fokko where would something like #402 go?

I've added it to the overview. Once the partial deletes + partitioned writes are in, this is supported automatically. We might want to have some community discussion on the API once those two PRs land.

[tusharchou]

@Fokko can we add issues for creating tests and documentation for the new features of 0.7.0 as good first issues?

[MehulBatra]

@Fokko can we add issues for creating tests and documentation for the new features of 0.7.0 as good first issues?

@tusharchou: Whenever you create a new feature, you need to add the unit & integration test and make the necessary changes in mkdocs as a part of that PR, but if you feel like there are some missing parts, please feel free to raise an improvement/issue and we can discuss that in the python syncup.