v0.7.26

Full Changelog: v0.7.25...v0.7.26

Release Notes

This is a smaller release to apply a dependency upgrade before the larger v0.8.0 release.

What's Changed

Build

• build: bump Ray to 2.59 by @NellyWhads in #7567

v0.7.25

What's Changed

✨ Features

• feat(lerobot): support v2.0/v2.1 dataset layouts by @XuQianJin-Stars in #7381
• feat(iceberg): add overwrite_filter for static partition overwrite by @atovk in #7419
• feat(optimizer): push limit through row-preserving join sides by @hello-peter-tang in #7439
• feat(io): add WebDataset reader by @rohankumardubey in #7493

🐛 Bug Fixes

• fix: clean up unscheduled UDF actors by @everettVT in #7346
• fix: IsIn/Decimal/UnresolvedColumn expressions fail to translate to SQL by @jiangxt2 in #7278
• fix(functions): make coalesce() argument validation reachable by @DogerW666 in #7404
• fix(catalog): declare noop auth for Gravitino Iceberg REST catalogs by @kyo-tom in #7417
• fix(distributed): surface background errors while input is pending by @zhouwenjiajia in #7396
• fix(parquet): error instead of nulling columns when field ids are absent by @jackylee-ch in #7446
• fix(catalog): avoid shared mutable default for from_postgres extensions by @DogerW666 in #7426
• fix(sql): escape single quotes in SQL literal rendering by @chuqCTC in #7269
• fix(sql): reject negative literals for unsigned table function options by @jackylee-ch in #7479
• fix(utf8): preserve backslashes in regexp_replace replacement by @jonasdedden in #7472
• fix: Fix startup of IO-Integration Docker Compose by @srilman in #7499

🚀 Performance

• perf(file): avoid opening files for file_size by @yongster in #7481

♻️ Refactor

• refactor(shuffle): unify distributed shuffle backend config onto daft-local-plan ShuffleBackend by @BABTUNA in #7402

📖 Documentation

• docs(sql): document table function options by @jackylee-ch in #7355
• docs: Update extras installation docs by @srilman in #7391
• docs: remove stale TableState references by @MeteorSkyOne in #7457

✅ Tests

• test: use runtime concurrency in morsel plan assertions by @ConeyLiu in #7454

👷 CI

• ci(integration): build Iceberg test image from python:3.10-bookworm by @hello-peter-tang in #7489

New Contributors

• @DogerW666 made their first contribution in #7404
• @atovk made their first contribution in #7419
• @MeteorSkyOne made their first contribution in #7457
• @chuqCTC made their first contribution in #7269
• @yongster made their first contribution in #7481
• @NellyWhads made their first contribution in #7487
• @rohankumardubey made their first contribution in #7493
• @jonasdedden made their first contribution in #7472

Full Changelog: v0.7.24...v0.7.25

v0.7.24

What's Changed

✨ Features

• feat: add read_blob function for reading files as raw bytes by @hello-peter-tang in #7318
• feat(utf8): add split_part string function by @ByJH in #7371

🐛 Bug Fixes

• fix: ls pagination by @prayanshchh in #7372

🔧 Maintenance

• chore: Update Rust deps for dependabot warnings by @srilman in #7363
• chore(deps): bump opendal to 0.58.1 by @Xuanwo in #7358
• chore: Remove some duplicate dependencies by @srilman in #7370
• chore: Bump some dependencies by @srilman in #7378

New Contributors

• @Xuanwo made their first contribution in #7358
• @prayanshchh made their first contribution in #7372
• @hello-peter-tang made their first contribution in #7318
• @ByJH made their first contribution in #7371

Full Changelog: v0.7.23...v0.7.24

v0.7.23

What's Changed

✨ Features

• feat(iceberg): inject DataFile column stats into DataSourceTask for predicate short-circuit by @jiangxt2 in #7301

🐛 Bug Fixes

• fix(dataframe): preserve explain() plan for all write sinks by @desmondcheongzx in #6580

🚀 Performance

• perf: Optimize stats aggregation within Swordfish by @srilman in #7359

📖 Documentation

• docs: migrate tutorials and examples to @daft.func / @daft.cls by @everettVT in #6729

✅ Tests

• test(iceberg): cover read_iceberg against a local catalog by @jackylee-ch in #7357

🔧 Maintenance

• chore: Update dashboard frontend dependencies by @srilman in #7360
• chore: Ignore Dependabot warnings in benchmarks by @srilman in #7361

Full Changelog: v0.7.22...untagged-8f79647338339e50cd69

v0.7.22

What's Changed

✨ Features

• feat(catalog): support Gravitino Iceberg REST catalogs by @kyo-tom in #7245
• feat(io): add HDFS support via OpenDAL services-hdfs by @Lucas61000 in #7202
• feat(io): export HdfsConfig from daft.io by @Lucas61000 in #7298
• feat(sql): support CREATE TABLE statement by @Lucas61000 in #7186
• feat(functions): add try_divide for PySpark parity by @BABTUNA in #7210
• feat(file): trace media file reads by @everettVT in #7312
• feat(file): trace video frame decoding by @everettVT in #7313
• feat(sql): support read_parquet ignore_corrupt_files by @jackylee-ch in #7133
• feat(sql): resolve read_iceberg io_config from table properties and default config by @jackylee-ch in #7273
• feat(sql): support ignore_corrupt_files in read_csv by @jackylee-ch in #7264

🐛 Bug Fixes

• fix(core): handle all-null list literals by @everySympathy in #7168
• fix(parquet): honor row-group constraints in count_rows() count pushdown by @FANNG1 in #7288
• fix: Gravitino catalog S3 credential key mismatch and _has_table error handling by @jiangxt2 in #7279
• fix(io): decode video with a single PyAV generator so .mkv/.webm yield frames by @AnayGarodia in #7229
• fix(io): support writing to hdfs:// URLs that include a port by @Lucas61000 in #7307
• fix(ai): route OpenAI chat completions parse through beta namespace by @colin-ho in #6995
• fix: truncate default IOConfig fields in explain() output by @shreynath in #7327

♻️ Refactor

• refactor(shuffle): migrate repartition onto shared backend and unify pipeline dispatch by @BABTUNA in #7275
• refactor(io): migrate Delta Lake scans to DataSource by @ihopenre-eng in #7300
• refactor(io): migrate Hudi scans to DataSource by @Lucas61000 in #7324
• refactor: migrate IcebergScanOperator to DataSource by @Lucas61000 in #7311

📖 Documentation

• docs: add motion trimming physical AI example by @ykdojo in #7341
• docs: remove dead HDF5 example notebook link by @everettVT in #7352

✅ Tests

• test(ai): mock beta chat completions parse by @everettVT in #7315

New Contributors

• @jiangxt2 made their first contribution in #7279
• @ihopenre-eng made their first contribution in #7300
• @shreynath made their first contribution in #7327

Full Changelog: v0.7.21...v0.7.22

v0.7.21

What's Changed

🐛 Bug Fixes

• fix(ai): keep UDF options out of OpenAI requests by @FANNG1 in #7277

📖 Documentation

• docs(benchmark): add hand-tracking workload A/B benchmark for lerobot batched decode by @ykdojo in #7267
• docs: add physical AI examples (hand tracking + reward scoring) by @ykdojo in #7271

Full Changelog: v0.7.20...v0.7.21

v0.7.20

What's Changed

✨ Features

• feat(iceberg): push down starts_with and is_nan/not_nan filters by @jackylee-ch in #7266

🐛 Bug Fixes

• fix(dashboard): support OUT_DIR on a different filesystem from the source tree by @alejandro-soto-franco in #7246
• fix(daft-file): release GIL in PyDaftFile methods to prevent deadlock by @zhouwenjiajia in #7257
• fix: fd and thread leak from tqdm progress bar in Jupyter by @ykdojo in #7262

♻️ Refactor

• refactor(shuffle): dedup local shuffle backend enums for gather and into_partitions by @BABTUNA in #7221

🔧 Maintenance

• chore(deps): lower opt-level for opendal-service-oss release builds by @alejandro-soto-franco in #7249
• chore(config): deprecate min_cpu_per_task by @XiaoHongbo-Hope in #7125

New Contributors

• @alejandro-soto-franco made their first contribution in #7249
• @zhouwenjiajia made their first contribution in #7257

Full Changelog: v0.7.19...v0.7.20

v0.7.19

What's Changed

✨ Features

• feat: add mcap pip extra and dependency checks for read_mcap by @everettVT in #7220
• feat(writers): add single_file kwarg to write_parquet by @aaron-ang in #7004

🐛 Bug Fixes

• fix: normalize HF bucket tree paths by @everettVT in #7218
• fix(session): allow set_provider to accept Provider instance for auto… by @DieselZhang in #7041
• fix(telemetry): prevent interpreter-shutdown segfault from Scarf telemetry by @FANNG1 in #7203
• fix(io): route oss:// through S3-compatible storage options for write_lance by @AnayGarodia in #7228
• fix: Reuse file format config in Iceberg and Delta by @srilman in #7163

📖 Documentation

• docs: add GooseFS connector documentation by @XuQianJin-Stars in #7193
• docs: Clean up formatting in Common Crawl docs by @srilman in #7237
• docs(checkpoint): add user guide and connector cross-references by @rohitkulshreshtha in #6985

👷 CI

• ci: Include New Contributers in Automated Draft Releases by @srilman in #7234

New Contributors

• @DieselZhang made their first contribution in #7041
• @FANNG1 made their first contribution in #7203
• @AnayGarodia made their first contribution in #7228

Full Changelog: v0.7.18...v0.7.19

v0.7.18

What's Changed 🚀

✨ Features

• feat: Remove dynamodb dependency for concurrent writes in deltalake on s3 @arpatilmh (#7187)
• feat(lerobot): batch video-frame decode by shard @ykdojo (#7184)
• feat(io): export GooseFSConfig from daft.io @XuQianJin-Stars (#7194)

🚀 Performance

• perf(inline-agg): pack two-string-column keys into u64 for typed FNV grouping @BABTUNA (#6924)

Full Changelog: v0.7.17...v0.7.18

v0.7.17

What's Changed 🚀

✨ Features

• feat: streamline HDF5 file access @everettVT (#7160)
• feat: Use HF Bucket as as alternative for Common Crawl @lhoestq (#7103)
• feat: Support Xet reads for HuggingFace IO @srilman (#7183)
• feat(inline-agg): add AnyValue accumulator @BABTUNA (#7036)
• feat(io): add GooseFS support via OpenDAL services-goosefs @XuQianJin-Stars (#7109)
• feat(ai): support text file inputs in transformers prompter @YuangGao (#7174)
• feat: asof_join_aligned distributed @euanlimzx (#7177)
• feat: add HDF5 file support @everettVT (#7159)
• feat(ai): add transformers provider for prompt() @YuangGao (#7152)
• feat(lerobot): Add daft.datasets.lerobot for working with LeRobot v3 datasets @srilman (#7090)
• feat(session): resolve catalog-qualified identifiers in namespace methods @Lucas61000 (#7171)

🐛 Bug Fixes

• fix: add pillow to video extra @everettVT (#7213)
• fix: Format nested Rust errors as a simplified backtrace in Python @srilman (#7178)
• fix(iceberg): pass ignore_corrupt_files through table reads @jackylee-ch (#7147)
• fix(io): default https scheme for scheme-less S3 endpoint_url @plusplusjiajia (#7111)

♻️ Refactor

• refactor(inline-agg): consolidate Sum/Product/Min/Max into one macro @BABTUNA (#7151)

🔧 Maintenance

• chore(deps): bump the all group with 4 updates @dependabot[bot] (#7190)
• chore: Rename GoosefsConfig to GooseFSConfig @srilman (#7192)

⬆️ Dependencies

• chore(deps): bump the all group with 4 updates @dependabot[bot] (#7190)

Full Changelog: v0.7.16...v0.7.17