v0.7.8

[desmondcheongzx]

What's Changed 🚀

💥 Breaking Changes

• chore!: Remove unused max_task_backlog parameter @srilman (chore!: Remove unused max_task_backlog parameter #6591)

✨ Features

• feat: Expose Subscribers through nicer APIs @srilman (feat: Expose Subscribers through nicer APIs #6631)
• feat(dashboard): heatmap coloring for nodes. @universalmind303 (feat(dashboard): heatmap coloring for nodes. #6628)
• feat(dataframe): add DataFrame.skew() global aggregation method @kerwin-zk (feat(dataframe): add DataFrame.skew() global aggregation method #6619)
• feat(dashboard): add arrows indicating pipeline direction @universalmind303 (feat(dashboard): add arrows indicating pipeline direction #6625)
• feat: wire CheckpointId through Flotilla execution pipeline @rohitkulshreshtha (feat: wire CheckpointId through Flotilla execution pipeline #6567)
• feat: add image_hash() for image deduplication @chenghuichen (feat: add image_hash() for image deduplication #6485)
• feat: support get function from catalog @gavin9402 (feat: support get function from catalog #6524)
• feat(dataframe): add var() method to DataFrame and GroupedDataFrame @kerwin-zk (feat(dataframe): add var() method to DataFrame and GroupedDataFrame #6584)
• feat: checkpoint based on distributed key-existence filter @everySympathy (feat: checkpoint based on distributed key-existence filter #5931)
• feat(distributed): make flotilla worker actor startup timeout configurable @desmondcheongzx (feat(distributed): make flotilla worker actor startup timeout configurable #6592)

🐛 Bug Fixes

• fix: Fix nightly Daft version resolution in Ray runtime env @jeevb (fix: Fix nightly Daft version resolution in Ray runtime env #6630)
• fix(dashboard): pass partition sets to repr_json so plan matches execution topology @samstokes (fix(dashboard): pass partition sets to repr_json so plan matches execution topology #6576)
• fix(scan): skip getting bytes when range start equals end in daft async reader @gweaverbiodev (fix(scan): skip getting bytes when range start equals end in daft async reader #6602)
• fix(io): retry transient errors on initial GET request @desmondcheongzx (fix(io): retry transient errors on initial GET request #6544)
• fix(dashboard): use smart per-node stats aggregation for distributed execution @samstokes (fix(dashboard): use smart per-node stats aggregation for distributed execution #6574)
• fix(io): handle schema-evolved Iceberg columns in Parquet predicate pushdown @sankarreddy-atlan ( fix(io): handle schema-evolved Iceberg columns in Parquet predicate pushdown #6551)
• fix(dashboard): prevent flotilla workers from sending spurious lifecycle events @samstokes (fix(dashboard): prevent flotilla workers from sending spurious lifecycle events #6573)

🚀 Performance

• perf: Optimize GroupBy Map Building & List-Agg @srilman (perf: Optimize GroupBy Map Building & List-Agg #6613)
• perf(flight): Read local shuffle data directly from disk @srilman (perf(flight): Read local shuffle data directly from disk #6436)
• perf(inline-agg): add min/max accumulator types @BABTUNA (perf(inline-agg): add min/max accumulator types #6604)
• perf: inline vectorized aggregation for grouped count/sum @desmondcheongzx (perf: inline vectorized aggregation for grouped count/sum #6345)

♻️ Refactor

• refactor: migrate PaimonScanOperator to DataSource API @chenghuichen (refactor: migrate PaimonScanOperator to DataSource API #6600)
• refactor(subscriber): collapse trait to single sync on_event method @cckellogg (refactor(subscriber): collapse trait to single sync on_event method #6593)
• refactor: consolidate BatchManager as single buffer abstraction @universalmind303 (refactor: consolidate BatchManager as single buffer abstraction #6566)
• refactor: Use DAFT_REF_NAME and DAFT_SHA env vars for benchmark run metadata @jeevb (refactor: Use DAFT_REF_NAME and DAFT_SHA env vars for benchmark run metadata #6610)
• refactor: Minor parameterization for benchmarking workflows @jeevb (refactor: Minor parameterization for benchmarking workflows #6609)
• refactor(distributed): unify repartition exchange write flow across ray and flight @ohbh (refactor(distributed): unify repartition exchange write flow across ray and flight #6499)

📖 Documentation

• docs: add custom SQL filter documentation for Lance @Jay-ju (docs: add custom SQL filter documentation for Lance #5916)

👷 CI

• ci(test): reduce Ray resource requests in chained skip_existing tests @desmondcheongzx (ci(test): reduce Ray resource requests in chained skip_existing tests #6633)
• ci(test): reduce Ray resource requests in skip_existing tests @desmondcheongzx (ci(test): reduce Ray resource requests in skip_existing tests #6629)
• ci(deps): fix CI failures from dependabot bump chore(deps): bump the all group with 49 updates #6570 @desmondcheongzx (ci(deps): fix CI failures from dependabot bump #6570 #6596)

🔧 Maintenance

• chore(cargo): inherit edition/version from workspace in root package @yew1eb (chore(cargo): inherit edition/version from workspace in root package #6329)
• chore!: Remove unused max_task_backlog parameter @srilman (chore!: Remove unused max_task_backlog parameter #6591)
• chore(dashboard): add ?debug query param for SSE event console logging @samstokes (chore(dashboard): add ?debug query param for SSE event console logging #6577)

Full Changelog: v0.7.7...v0.7.8

---

This discussion was created from the release v0.7.8.