v0.7.5

[desmondcheongzx]

What's Changed 🚀

💥 Breaking Changes

• perf!: lazy-load lance to speed up import daft @desmondcheongzx (perf!: lazy-load lance to speed up import daft #6328)
• feat!: add ddof parameter to stddev (default ddof=1) @aaron-ang (feat!: add ddof parameter to stddev (default ddof=1) #6142)

✨ Features

• feat(io): support numeric range expansion in glob patterns @singularityDLW (feat(io): support numeric range expansion in glob patterns #6127)
• feat(dashboard): add query result preview tab @samstokes (feat(dashboard): add query result preview tab #6327)
• feat: Add read_text API to support reading text files @plotor (feat: Add read_text API to support reading text files #6111)
• feat: Add protocol aliases for IOConfig @universalmind303 (feat: Add protocol aliases for IOConfig #6252)
• feat(observability): Improve distributed operator names @srilman (feat(observability): Improve distributed operator names #6324)
• feat(metrics): add node_plan_id for distributed stats attribution @cckellogg (feat(metrics): add node_plan_id for distributed stats attribution #6332)
• feat: adds support for daft native extensions via stable C ABI @rchowell (feat: adds support for daft native extensions via stable C ABI #6301)
• feat(lance): refine distributed scalar index support with new types and parameters @huleilei (feat(lance): refine distributed scalar index support with new types and parameters #5817)
• feat!: add ddof parameter to stddev (default ddof=1) @aaron-ang (feat!: add ddof parameter to stddev (default ddof=1) #6142)
• feat: use human-readable query names instead of UUIDs @samstokes (feat: use human-readable query names instead of UUIDs #6310)
• feat(metrics): add node phase attribute for multi-phase pipeline nodes @cckellogg (feat(metrics): add node phase attribute for multi-phase pipeline nodes #6290)
• feat(dashboard): add wall-clock and CPU duration to operator cards @samstokes (feat(dashboard): add wall-clock and CPU duration to operator cards #6300)
• feat: Add Python Function data source info in explain @plotor (feat: Add Python Function data source info in explain #6119)
• feat(dashboard): physical plan tree with live operator stats @samstokes (feat(dashboard): physical plan tree with live operator stats #6299)
• feat(io): implement write_sql with SQLDataSink and explicit dtype support @huleilei (feat(io): implement write_sql with SQLDataSink and explicit dtype support #5979)
• feat(dashboard): add interactive tree visualizer for query plans @samstokes (feat(dashboard): add interactive tree visualizer for query plans #6295)

🐛 Bug Fixes

• fix(windows): pin transitive aws deps to fix windows linking bug @universalmind303 (fix(windows): pin transitive aws deps to fix windows linking bug #6404)
• fix(video): add missing pillow dependency for video keyframes @desmondcheongzx (fix(video): add missing pillow dependency for video keyframes #6355)
• fix(dashboard): request notification permission before enabling notifications @samstokes (fix(dashboard): request notification permission before enabling notifications #6362)
• fix(dashboard): make dashboard subscriber non blocking @cckellogg (fix(dashboard): make dashboard subscriber non blocking #6336)
• fix(dashboard): ensure query_end fires when generator is abandoned early @samstokes (fix(dashboard): ensure query_end fires when generator is abandoned early #6326)
• fix(tests): make Ray conda UDF tests robust to unavailable Python micro versions @cckellogg (fix(tests): make Ray conda UDF tests robust to unavailable Python micro versions #6341)
• fix(sql): reject duplicate CTE names in WITH clause @singularityDLW (fix(sql): reject duplicate CTE names in WITH clause #6331)
• fix: Use Tokio FS for Flight Server Reads @srilman (fix: Use Tokio FS for Flight Server Reads #6308)
• fix(io): add connection pool semaphore to Azure Blob Storage backend @singularityDLW (fix(io): add connection pool semaphore to Azure Blob Storage backend #6305)
• fix(dashboard): sort extra stats alphabetically to prevent flicker @samstokes (fix(dashboard): sort extra stats alphabetically to prevent flicker #6309)
• fix: Support Union and Optional in DataType.infer_from_type @srilman (fix: Support Union and Optional in DataType.infer_from_type #6304)
• fix: make max_concurrency control coroutine concurrency for async UDFs @colin-ho (fix: make max_concurrency control coroutine concurrency for async UDFs #6302)
• fix: Fix native executor closing @colin-ho (fix: Fix native executor closing #6282)
• fix(docs): Remove duplicate Voice AI analytics listing in SUMMARY.md @copilot-swe-agent[bot] (fix(docs): Remove duplicate Voice AI analytics listing in SUMMARY.md #6274)

🚀 Performance

• perf: Support multi-task parallel glob scan for from_glob_path @plotor (perf: Support multi-task parallel glob scan for from_glob_path #5897)
• perf: Drop tasks early in distributed limits @srilman (perf: Drop tasks early in distributed limits  #6350)
• perf(image): make image ops run in parallel via rayon @universalmind303 (perf(image): make image ops run in parallel via rayon #6333)
• perf(upload): use a zero copy approach for url_upload @universalmind303 (perf(upload): use a zero copy approach for url_upload #6334)
• perf(parquet): late materialization via RowFilter and RowSelection @desmondcheongzx (perf(parquet): late materialization via RowFilter and RowSelection #6311)
• perf!: lazy-load lance to speed up import daft @desmondcheongzx (perf!: lazy-load lance to speed up import daft #6328)

♻️ Refactor

• refactor: decouple extension ABI from arrow-rs with Arrow C Data Interfaces @rchowell (refactor: decouple extension ABI from arrow-rs with Arrow C Data Interfaces #6337)
• refactor(schema): change Field.name from String to Arc<str> @desmondcheongzx (refactor(schema): change Field.name from String to Arc<str> #6351)
• refactor(deps): remove 11 dependencies by inlining builder dependencies @universalmind303 (refactor(deps): remove 11 dependencies by inlining builder dependencies #6364)
• refactor(arrow2): goodbye arrow2 @universalmind303 (refactor(arrow2): goodbye arrow2 #6363)
• refactor(arrow2): migrate remaining parquet code to arrow-rs @universalmind303 (refactor(arrow2): migrate remaining parquet code to arrow-rs #6361)
• refactor(arrow2): remove parquet2 from daft @desmondcheongzx (refactor(arrow2): remove parquet2 from daft #6339)
• refactor(arrow2): remove parquet2 read pipeline, make arrow-rs the default reader @desmondcheongzx (refactor(arrow2): remove parquet2 read pipeline, make arrow-rs the default reader #6335)
• refactor(arrow2): add arrow-rs parquet reader @desmondcheongzx (refactor(arrow2): add arrow-rs parquet reader #6294)
• refactor(arrow2): daft-csv crate @universalmind303 (refactor(arrow2): daft-csv crate #6298)
• refactor(arrow2): daft-functions-serde @universalmind303 (refactor(arrow2): daft-functions-serde #6303)
• refactor(arrow2): introduce DaftParquetMetadata adapter to decouple crates from parquet2 @desmondcheongzx (refactor(arrow2): introduce DaftParquetMetadata adapter to decouple crates from parquet2 #6292)
• refactor(arrow2): refactor daft-json to arrow-rs @universalmind303 (refactor(arrow2): refactor daft-json to arrow-rs #6297)
• refactor(arrow2): move the auto conversion logic into array init @universalmind303 (refactor(arrow2): move the auto conversion logic into array init #6289)

📖 Documentation

• docs(optimization): add join strategies guide @desmondcheongzx (docs(optimization): add join strategies guide #6367)
• docs: add custom catalog implementation guide @desmondcheongzx ( docs: add custom catalog implementation guide #6366)
• docs(skills): add Claude Code compatible guides and update README @Jay-ju (docs(skills): add Claude Code compatible guides and update README #6116)

🔧 Maintenance

• chore: removes unnecessary pushdown builder code @rchowell (chore: removes unnecessary pushdown builder code #6403)
• chore: simplify data source implementation @rchowell (chore: simplify data source implementation #6397)
• chore: add claude/worktrees to gitignore @universalmind303 (chore: add claude/worktrees to gitignore #6402)
• chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 @singularityDLW (chore(deps): drop pyarrow 8.0.0 support, bump minimum to >= 15.0.0 #6378)
• chore: Enable write_json related test cases for Ray Runner @plotor (chore: Enable write_json related test cases for Ray Runner #6387)
• chore: Upgrade PyO3 to 0.28 @srilman (chore: Upgrade PyO3 to 0.28 #6383)
• chore: update Slack invite links across docs and README @everettVT (chore: update Slack invite links across docs and README #6377)
• chore: Replace RepartitionSpec::IntoPartitions with custom path @srilman (chore: Replace RepartitionSpec::IntoPartitions with custom path #6349)
• chore: Improve Common Crawl Benchmark @srilman (chore: Improve Common Crawl Benchmark  #6307)
• chore(tests): migrate internal usages of daft.udf to cls/func @universalmind303 (chore(tests): migrate internal usages of daft.udf to cls/func #6348)
• chore(dashboard): add README with build and architecture notes @samstokes (chore(dashboard): add README with build and architecture notes #6323)
• chore(lints): add workspace lints inheritance to missing subcrates @yew1eb (chore(lints): add workspace lints inheritance to missing subcrates #6330)
• chore: Fix some typo issues in the code or docs @plotor (chore: Fix some typo issues in the code or docs #6320)
• chore(deps): bump the all group with 4 updates @dependabot[bot] (chore(deps): bump the all group with 4 updates #6314)
• chore: refactor pipeline node ownership in start @srilman (chore: refactor pipeline node ownership in start #6259)
• chore: Move EXPLAIN ANALYZE to Python @srilman (chore: Move EXPLAIN ANALYZE to Python #6254)

⬆️ Dependencies

• chore(deps): bump the all group with 4 updates @dependabot[bot] (chore(deps): bump the all group with 4 updates #6314)

Full Changelog: v0.7.4...v0.7.5

---

This discussion was created from the release v0.7.5.