Skip to content

Tensorseed 0.2.0

Choose a tag to compare

@flyingmongoose flyingmongoose released this 13 Nov 17:55
· 90 commits to main since this release

Tensorseed 0.2.0

Tensorseed mirrors open Hugging Face models and datasets for preservation.
0.2.0 lets one client follow several signed catalogs at once and adds a
browser explorer at explore.tensorseed.dev
that verifies the catalog signature and its Chia anchor without installing
anything. Formats and commands may still change before 1.0.

Highlights

  • Multi-catalog: follow several signed, chain-anchored catalogs with
    [[catalogs]] or tensorseed catalog add. Revisions that agree across
    catalogs share one swarm, and list shows how many catalogs agree.
  • Catalog explorer at explore.tensorseed.dev: browse and search the
    official catalog, open an entry, and check the minisign signature and
    the Chia DataLayer anchor in the browser (WebAssembly built from the
    same Rust verification code as the CLI). Add another catalog by URL,
    key, and store id to compare.
  • Default download order is the BitTorrent swarm, then Hugging Face, then
    HTTP mirrors, configurable with [fetch] source_order.
  • tensorseed hf login, logout, and status, and automatic reuse of an
    existing Hugging Face CLI token.
  • pull --include and --exclude globs for partial pulls.
  • .deb and .rpm packages, shell completions, and man pages.

Download targets

The same five archives as 0.1.0 (Linux x86_64 and ARM64 static musl,
Linux x86_64 glibc 2.17, Windows x86_64, Windows ARM64), signed source
archives (tensorseed-0.2.0-src.tar.gz and .zip), and new .deb and
.rpm packages for x86_64 and aarch64. All are listed in SHA256SUMS,
signed with the release key (minisign, key id F5D2D3C4C27D459D).

Added

  • Read-only catalog explorer (explorer/, Svelte 5, TailwindCSS,
    DaisyUI, built with pnpm, no third party scripts) deployed at
    explore.tensorseed.dev. It verifies the catalog with a WebAssembly
    build of tensorseed-verify (tensorseed-wasm), shows chain status,
    per-entry files, hashes, licenses, and torrent details, and has a
    "use another catalog" form with agreement across catalogs.
    Catalog signature and chain checks moved into the tensorseed-verify
    crate so the CLI and the explorer share one implementation.
  • tensorseed pull --include GLOB and --exclude GLOB (repeatable) select
    files by path. Globs use the same rules as serve (* matches any
    run of characters, including /). --exclude wins when both match.
    The swarm uses librqbit only_files; HTTP mirrors and the Hub use the
    same filter. pull.json records the full file list plus skipped and
    unselected. verify on a partial pull passes when selected files
    match; missing unselected files fail only with --strict. serve and
    --seed-after seed the selected subset with only_files, and do not
    announce missing files as a complete torrent.
  • Default download order is the BitTorrent swarm, then Hugging Face,
    then HTTP mirrors. [fetch] source_order is a list of "swarm",
    "hub", and "mirrors" (validated, no duplicates) and applies to
    pull and serve auto-fetch. --no-swarm, --swarm-only, and
    dead-mirror memory are unchanged. A 401, 403, or 404 from the Hub
    falls through to the mirrors without marking the host dead.
  • Hugging Face tokens are also loaded from tensorseed hf login and from
    the Hugging Face CLI (HF_TOKEN_PATH, $HF_HOME/token, or
    ~/.cache/huggingface/token). Resolution order: token_env, then
    token_file, then the tensorseed login file, then the CLI token.
    tensorseed hf login reads a token from a hidden prompt or stdin and
    writes ~/.config/tensorseed/hf_token (0600 on Unix, a user-only ACL
    on Windows). hf logout removes that file. hf status calls
    whoami-v2 and prints the username and token source, never the token.
  • just dist-packages builds .deb and .rpm packages (x86_64 and
    aarch64) from the static musl binaries with nfpm. Each package
    installs /usr/bin/tensorseed, a disabled tensorseed-serve systemd
    unit running as the tensorseed system user, and
    /etc/tensorseed/tensorseed.toml.example (copied to tensorseed.toml
    on first install). just dist includes the packages in SHA256SUMS.
  • tensorseed completions <shell> prints a bash, zsh, fish, or PowerShell
    completion script. just generate-docs also writes man pages
    (tensorseed.1 and one page per subcommand). Linux packages install
    bash, zsh, and fish completions (fish to
    /usr/share/fish/vendor_completions.d) and the man pages to
    /usr/share/man/man1. Release tarballs and zip archives include
    completions/ and man/.
  • Follow several catalogs at once with [[catalogs]] (name, label,
    publisher key, store id, base URL, chain check, optional priority).
    A config with no [[catalogs]] still uses the top-level [catalog],
    [file], and [chia] block as the sole subscription. Revisions are
    keyed by Hugging Face repo and commit; agreeing hashes share one swarm
    and can use every agreeing catalog's HTTP sources. list prints
    "N of M catalogs agree". The same repo and commit with different
    hashes is a hard error unless --catalog is given. Tombstones are per
    catalog: a revision remains pullable and seedable if another
    subscription still lists it. serve seeds a revision that is on disk
    and still valid in at least one non-tombstoned subscription.
    tensorseed catalog list, catalog add, and catalog remove manage
    subscriptions; catalog info and catalog chain-status accept
    --catalog.

Fixed

  • The BitTorrent DHT binds IPv4-only when the configured listen address is
    IPv4, and stores its routing table under Tensorseed's data directory
    (%LOCALAPPDATA%\tensorseed\dht.json on Windows). A previous default
    used librqbit's dual-stack [::] bind and its own cache path, which
    failed to start the engine on Windows and aborted pull before HTTP
    sources could run. If the engine still fails to start, pull now
    falls back to HTTP unless --swarm-only is set.

Verify a release

Release archives ship with SHA256SUMS and SHA256SUMS.minisig, signed
by the release key (minisign, key id F5D2D3C4C27D459D). This is not the
catalog publisher key. Do not verify binaries with the catalog key.

RWSdRX3CxNPS9Vl1JNjDnZ04OmM/3xrfp9vqG5FO8TNpK8Sbg6ENAI4P
minisign -Vm SHA256SUMS -P RWSdRX3CxNPS9Vl1JNjDnZ04OmM/3xrfp9vqG5FO8TNpK8Sbg6ENAI4P
sha256sum -c --ignore-missing SHA256SUMS

The public key file is also published as tensorseed-release.pub (and in
keys/tensorseed-release.pub in the repository).

Known limitations

  • The Windows ARM64 build has not been run on ARM Windows hardware.
  • There is no macOS build; build from source with Rust 1.87 or newer.
  • Catalog and on-disk formats may change before 1.0.
  • takedown and doctor still print "not implemented yet".