Skip to content

Add zstd decompression #23

Description

@andrew

Several package formats we want to open are zstd-compressed and we currently have no reader for it:

  • .tar.zst directly
  • .conda (anaconda.org v2 format) is a zip containing pkg-<name>.tar.zst and info-<name>.tar.zst
  • .deb on current Debian and Ubuntu ships data.tar.zst / control.tar.zst inside the ar container

Go stdlib has no zstd. github.com/klauspost/compress/zstd is pure Go, no cgo, actively maintained, and widely used; we already carry ulikunitz/xz as a compression dep so there is precedent. The alternative is github.com/DataDog/zstd which wraps the C library via cgo and is faster but breaks the pure-Go build.

Once a zstd.Reader is wired into the tar path the same way gzip/bzip2/xz are, .tar.zst is just another suffix in detectFormat, and zstd is another case in archiveFormat for the content-sniff fallback (magic will report zstd once git-pkgs/magic#1 lands). .conda and .deb need their own container handlers on top (zip-of-tars and ar-of-tars respectively) and can follow separately; this issue is just the decompressor.

Follows #21.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions