Andrew Lamb Weekly-ish Open Source plan - 2025-12-08

[alamb]

This is my weekly plan, mostly for my own organizational needs (as I am dropping too many things). I am making it public in the hopes that helps others to see what I am working on -- also I spend so much time in github the interface is very familiar to me and I can cross link all the issues I am working

(it is also my excuse as to why I haven't reviewed many good looking PRs)

Notes to myself: a duplicate entry unchecked means I need to go back and re-review

PR review queue (rough order)

• relation extension planner: Add relation planner extension support to customize SQL planning #17843
• Arc partition values in TableSchema #19137
• Fix PruningPredicate interaction with DynamicFilterPhysicalExpr that references partition columns #19129
• feat: Add remove_optimizer_rule to SessionContext #19209 (review)
• Fix PartialEq behavior for UnionFields arrow-rs#8937
• feat: Add DELETE/UPDATE hooks to TableProvider trait #19142 (review)
• Row group limit pruning #18868
• Add additional tests for InListExpr #19050
• Move partition handling out of PhysicalExprAdapter #19128
• Support "pre-image" for pruning predicate evaluation #18789
• Push down projection expressions into ParquetOpener #19111
• Establish the high level API for sort pushdown and the optimizer rule and support reverse files and row groups  #19064
• arrow-select: implement specialized interleave_list arrow-rs#8953
• feat: Implement the statistics_cache function #19054
• Make push_batch_with_filter up to 3x faster for primitive types arrow-rs#8951
• add specialized InList implementations for common scalar types #18832
• Row group limit pruning #18868 (rereview)
• Support reverse parquet scan and fast parquet order inversion at row group level #18817
• Emit aggregation groups in chunks to avoid blocking async runtime #18906
• feat(memory-tracking): expose API to NullBuffer, ArrayData, and Array arrow-rs#8918 (comment)
• [DRAFT] Extension Type Registry Draft #18552
• feat: implement GroupArrayAggAccumulator attempt 3 #17915

Projects I am supporting actively (high on my priority list)

• Improve DataFusion ClickBench performance: [EPIC] Make DataFusion the top of the ClickBench Parquet leaderboard #18489
• Release object store 0.13.0: Release object store 0.13.0 (breaking) - Target Nov 2025 arrow-rs-object-store#367
• DataFusion object store requests go faster with @BlakeOrth [EPIC] ListingTable object store usage improvements #17214
• Help integrate Variant with @friendlymatthew [EPIC] Support VARIANT type for unstructured data #16116

Projects on my backlog

These are ones I would like to support but don't have the capacity at the moment to push, in relative order

• Epic: Join Order Enumeration #18249 from @NGA-TRAN
• feat: support invoking table functions with tables rather than a single expression #18535

PRs that look great but need a thorough review (looking for help here 🎣 from anyone else)

• external tables for multiple locations: feat(cli): support external tables on multiple locations #17702
• writing REE arrays to parquet: Support writing RunEndEncoded as Parquet arrow-rs#8069