Skip to content

0.5.0

Latest
Latest

Choose a tag to compare

View all tags
@yihua yihua released this 10 Sep 10:34
· 16 commits to main since this release
release-0.5.0
c6c4bac
This commit was signed with the committer’s verified signature.
yihua Y Ethan Guo
GPG key ID: 6BE1D454C90F5EA5
Verified
Learn about vigilant mode.

What's changed

🚀 Features

  • Add get_file_slices_splits_between API (#411) by @yunchipang
  • Expose datafusion custom table provider to python (#412) by @brayanjuls
  • Remove table version 5 support (#409) by @yunchipang
  • Support picking up storage options from HOODIE_ENV_ env vars (#410) by @yunchipang
  • Add table v8+ timeline support for loading old and new table versions (#395) by @codope
  • Add hfile reader (#490) by @xushiyan
  • Support reading log files from files partition in metadata table (#492) by @xushiyan
  • Add API to read file slice from base file and list of log files (#446) by @yunchipang
  • Support reading metadata table (files) (#499) by @xushiyan
  • Implement table v8 file slicing for read path (#501) by @xushiyan
  • Add streaming table and file group APIs (#508) by @xushiyan
  • Support column project and row filter pushdown (#510) by @xushiyan
  • Support column stats pruning (#516) by @xushiyan
  • Modernize Table API with async-only design (#537) by @xushiyan
  • Support IN and NOT IN operators in partition filter (#566) by @yunchipang
  • Export HudiTableConfig and HudiReadConfig as python enums (#433) by @yunchipang
  • Unify Hudi read APIs around ReadOptions and add Python bindings (#587) by @xushiyan
  • (core) Enrich FileSlice with stats and streamline config flow (#592) by @xushiyan
  • (core) Add general config alias support with ordering fields as primary config (#598) by @xushiyan
  • (core) Add lance base file format support (#596) by @xushiyan
  • (datafusion) Support Lance scan and broaden MOR snapshot e2e coverage (#608) by @xushiyan
  • (core) Select the file group reader by version, defaulting to 2 (#677) by @linliu-code
  • (core) Serve merge-on-read reads with file group reader version 2 (#660) by @linliu-code
  • (core) Read the metadata table as Arrow, and expose hudi-rs over a C ABI for JVM callers (#700) by @linliu-code
  • (reader) Merge-aware pushdown gate, row-group pruning, and read-volume counters (#708) by @linliu-code
  • (core) Serve Hudi metadata tables through reader v2, sharded and memory-bounded (#706) by @linliu-code
  • (deps) Upgrade to arrow 58, DataFusion 54, object_store 0.13, and Lance 11 (#711) by @yihua
  • (benchmark) Add AWS EC2 setup for the TPC-H harness (#718) by @yihua
  • (release) Verify the artifacts a release actually published (#759) by @yihua

🐛 Bug Fixes

  • Handle non partition in MDT files partition (#505) by @xushiyan
  • Avoid dashmap deadlock for correlated queries (#539) by @xushiyan
  • Make timestamp keygen work for partition pruning (#462) by @yunchipang
  • Order partition schema by config and error on missing fields (#581) by @sushiljacksparrow
  • Stop .gitignore from swallowing test fixtures (#676) by @linliu-code
  • (core) Gate the log-block scan on instant state (#678) by @linliu-code
  • (core) Fold partial updates in both ordering directions (#679) by @linliu-code
  • (core) Keep the custom merger's own columns in the required schema (#704) by @linliu-code
  • (python) Compile the datafusion feature into every python build (#714) by @yihua
  • (release) Align the release guide and scripts with their actual behavior (#717) by @yihua
  • (release) Set the macOS deployment target for the x86_64 wheel build (#722) by @yihua
  • (benchmark) Stop the parquet run overwriting the Hudi benchmark results (#747) by @yihua
  • (core) Decline base-read pushdown when a predicate reads a repaired column (#748) by @linliu-code
  • (ci) Install a modern protoc for the manylinux wheel build (#751) by @yihua
  • (release) Reject an expired signing key in the src release verification (#753) by @yihua
  • (ci) Publish the release only after every artifact has built (#755) by @yihua
  • (python) Declare the readme so the published package has a description (#762) by @yihua

🚜 Refactor

  • Rename variable names in splitting table APIs (#424) by @yunchipang
  • Replace custom .to::() with into() for HudiConfig conversions (#432) by @yunchipang
  • Move .crc filtering logic from table to storage layer (#458) by @yunchipang
  • Define avro model for parsing commit metadata (#477) by @xushiyan
  • Add targeted MDT lookup APIs for partition pruning (#502) by @xushiyan
  • Consolidate file metadata population for snapshot and incremental queries (#586) by @xushiyan
  • (core) Move parquet io into base file reader (#599) by @xushiyan

📚 Documentation

  • Update release guide (#287) by @xushiyan
  • Add AGENTS.md as canonical AI coding agent guide (#591) by @xushiyan
  • Deduplicate AI agent guidance with sub-directory AGENTS.md (#594) by @xushiyan
  • Correct read config scoping and MOR streaming, document reader v2 knobs (#709) by @yihua
  • Fix stale API claims in README, reader spec, and agent guides (#716) by @yihua

⚡ Performance

  • (core) Skip decoding out-of-range log blocks and add unit test (#420) by @codope
  • Setup tpch benchmark infra (#538) by @xushiyan
  • Improve datafusion integration (#548) by @xushiyan
  • (core) Copy an HFile record rather than its whole block per record (#697) by @linliu-code
  • (core) Bound the merge chunk rather than moving the merge off the executor (#689) by @linliu-code

🛠️ Build

  • Use uv for Python package management (#391) by @codephage2020
  • (deps) Bump pytest from 8.3.5 to 8.4.1 in /python (#355) by @dependabot[bot]
  • (deps) Bump astral-sh/setup-uv from 5 to 6 (#399) by @dependabot[bot]
  • (deps) Bump actions/checkout from 4 to 5 (#416) by @dependabot[bot]
  • (deps) Bump actions/download-artifact from 4 to 5 (#417) by @dependabot[bot]
  • (deps) Bump mypy from 1.15 to 1.17.1 in /python (#413) by @dependabot[bot]
  • (deps) Bump actions/labeler from 5 to 6 (#443) by @dependabot[bot]
  • (deps) Bump actions/setup-python from 5 to 6 (#444) by @dependabot[bot]
  • (deps) Bump actions/setup-node from 4 to 5 (#442) by @dependabot[bot]
  • (deps) Bump pytest from 8.4.1 to 8.4.2 in /python (#441) by @dependabot[bot]
  • (deps) Fix integration test setup issue with pyarrow and rust versions (#467) by @xushiyan
  • (deps) Bump astral-sh/setup-uv from 6 to 7 (#465) by @dependabot[bot]
  • (deps) Bump coverage from 7.8 to 7.10.7 in /python (#456) by @dependabot[bot]
  • (deps) Bump mypy from 1.17.1 to 1.18.2 in /python (#454) by @dependabot[bot]
  • (deps) Bump actions/setup-node from 5 to 6 (#469) by @dependabot[bot]
  • (deps) Bump ruff from 0.12 to 0.14.1 in /python (#472) by @dependabot[bot]
  • Use build wrapper to handle macos 26 SDK loading (#479) by @xushiyan
  • (deps) Bump apache/skywalking-eyes from 0.7.0 to 0.8.0 (#468) by @dependabot[bot]
  • (deps) Bump actions/checkout from 5 to 6 (#485) by @dependabot[bot]
  • (deps) Bump actions/download-artifact from 5 to 7 (#496) by @dependabot[bot]
  • (deps) Bump actions/upload-artifact from 4 to 6 (#495) by @dependabot[bot]
  • (deps) Bump actions/cache from 4 to 5 (#494) by @dependabot[bot]
  • (deps-dev) Bump mypy from 1.18.2 to 1.19.1 in /python (#497) by @dependabot[bot]
  • (deps) Update tokio requirement from ~1.45 to ~1.48 (#470) by @dependabot[bot]
  • (deps) Upgrade apache-avro to v0.21.0 (#503) by @xushiyan
  • (deps) Upgrade datafusion, arrow, and other deps (#504) by @xushiyan
  • (deps) Upgrade rust and make deps verisons more compatible (#506) by @xushiyan
  • (deps-dev) Bump ruff from 0.14.1 to 0.14.13 in /python (#526) by @dependabot[bot]
  • (deps) Update strum requirement from 0.27 to 0.28 (#533) by @dependabot[bot]
  • (deps) Bump actions/upload-artifact from 6 to 7 (#531) by @dependabot[bot]
  • (deps) Upgrade datafusion to 52, require python 3.10 (#535) by @xushiyan
  • (deps) Bump actions/download-artifact from 7 to 8 (#532) by @dependabot[bot]
  • (deps) Bump docker/setup-buildx-action from 3 to 4 (#540) by @dependabot[bot]
  • (deps) Update strum_macros requirement from 0.27 to 0.28 (#541) by @dependabot[bot]
  • (deps) Update zip requirement from 4 to 8 (#545) by @dependabot[bot]
  • (deps-dev) Bump datafusion from 52.0.0 to 52.3.0 in /python (#544) by @dependabot[bot]
  • (deps) Bump actions/checkout from 5 to 6 (#568) by @dependabot[bot]
  • (deps) Bump codecov/codecov-action from 5 to 6 (#567) by @dependabot[bot]
  • (deps-dev) Bump pytest from 8.4.2 to 9.0.3 in /python (#577) by @dependabot[bot]
  • (deps-dev) Bump mypy from 1.19.1 to 1.20.2 in /python (#584) by @dependabot[bot]
  • (deps-dev) Bump ruff from 0.14.13 to 0.15.12 in /python (#585) by @dependabot[bot]
  • Upgrade rust dev toolchain to 1.94 (#595) by @xushiyan
  • (deps) Bump actions/cache from 5 to 6 (#635) by @dependabot[bot]
  • (deps) Bump actions/checkout from 6 to 7 (#632) by @dependabot[bot]
  • (deps-dev) Bump pytest from 9.0.3 to 9.1.1 in /python (#634) by @dependabot[bot]
  • (deps) Bump codecov/codecov-action from 6 to 7 (#628) by @dependabot[bot]
  • (deps-dev) Bump ruff from 0.15.12 to 0.15.18 in /python (#633) by @dependabot[bot]
  • (deps) Bump astral-sh/setup-uv from 8.3.2 to 10.0.1 (#626) by @dependabot[bot]
  • (deps-dev) Bump coverage from 7.10.7 to 7.16.0 in /python (#625) by @dependabot[bot]
  • (deps) Bump docker/setup-buildx-action from 4.2.0 to 4.3.0 (#617) by @dependabot[bot]
  • (deps-dev) Bump mypy from 1.20.2 to 2.3.1 in /python (#615) by @dependabot[bot]
  • Raise MSRV to 1.94.1 to match what the dependency graph requires (#715) by @yihua
  • Adopt the MSRV-aware resolver and check the MSRV in CI (#721) by @yihua
  • (python) Bound the datafusion extra to the 54.x major (#720) by @yihua
  • Mark hudi-jvm-ffi as not published (#719) by @yihua

🧪 Testing

⚙️ Miscellaneous Tasks

New Contributors

Contributors

yihua, sushiljacksparrow, and 10 other contributors
Assets 2
Loading