Skip to content

v0.18.0

Choose a tag to compare

@amotl amotl released this 22 Sep 23:36
· 68 commits to main since this release

What's Changed

  • The filesystem and blob stack builds as its own distribution, dlt-filesystem.
    It moves to packages/dlt-filesystem/ and declares its own dependencies, so a dlt
    user can install the readers, the strict lister and the Arrow readinto shim without
    omniload's dependency list: 136 distributions against 250. Nothing an existing install
    line resolves to changes. omniload, omniload[filesystem], omniload[iterable] and
    omniload[full] each resolve to the same set as before, because the extras now forward
    to the new distribution. The install hints a missing long-tail decoder prints name
    dlt-filesystem[iterable], which is the extra that supplies it.

  • Tests: the memory-store fixtures keep to their own store. Two test modules
    subclass MemoryFileSystem to hold their fixtures apart from fsspec's
    process-global store, but store is a class attribute the subclass inherits
    rather than replaces, so their clear() emptied the store the remote-filesystem
    tests write into at import time. Each subclass now carries its own.

  • Tests: the Arrow readinto shim's write-handle test reads the instance. fsspec
    2026.9.0 mirrors readinto onto ArrowFile for every mode, so hasattr no longer
    separates what fsspec provides from what the shim injects. The shim itself is unchanged
    and stays, because the supported floor is fsspec>=2024.6.

  • Filesystem: Vortex files, read and write. .vortex (and .vortex.gz) files
    read on every filesystem source, and the local file:// destination writes them.
    A remote or gzipped file is staged to a local temporary file first, because the
    Vortex library opens files by path only. Opt-in through the new vortex extra, which needs Python 3.11 or
    newer and is included in full.

Details

  • fix(deps): update dependency lakefs-spec to <0.16 by @renovate[bot] in #365
  • fix(deps): update dependency ty to v0.0.82 by @renovate[bot] in #366
  • fix(deps): update dependency fsspec to >=2024.6,<2026.10 by @renovate[bot] in #367
  • Chore: Format code. Satisfy linter. by @amotl in #370
  • Tests: Pin the Arrow readinto contract and isolate the memory-store fixtures by @hampsterx in #372
  • chore(deps): update astral-sh/setup-uv action to v10.2.0 by @renovate[bot] in #374
  • Filesystem: Build the package as its own distribution, dlt-filesystem by @hampsterx in #375
  • Add Vortex file format by @amotl in #373
  • fix(deps): update dependency ty to v0.0.83 by @renovate[bot] in #376

Full Changelog: v0.17.0...v0.18.0