hubdl

hubdl is a multi-hub model and dataset downloader for Linux servers, distributed on PyPI.

It combines:

Hugging Face listing + aria2 workflow (inspired by hfd)
ModelScope API listing (from battle-tested local msd tooling)
A shared core: integrity-aware resume, hard rate limits, path sandbox, directory flock, L3 SHA-256 and L4 repository reports

Disclaimer: hubdl is not affiliated with Hugging Face, Inc. or ModelScope / Alibaba DAMO.
You are responsible for complying with each model's license and each Hub's terms of service.

Install & use

Official install: create a venv, then install from PyPI. Install system aria2 for downloads.
Do not install into the system/global Python (sudo pip is discouraged).

# 1) System packages (Debian / Ubuntu)
sudo apt update
sudo apt install -y python3 python3-venv aria2 ca-certificates

# 2) Install from PyPI into a venv
python3 -m venv .venv
source .venv/bin/activate          # Windows: .\.venv\Scripts\Activate.ps1
pip install -U pip
pip install hubdl

# 3) Verify
hubdl --version
hubdl hf gpt2 --local-dir ./gpt2 --dry-run

Task	Command
Install (official)	`pip install hubdl`
Pin a version	`pip install "hubdl==0.1.2"`
Upgrade	`pip install -U hubdl`

Package: https://pypi.org/project/hubdl/ · CLI entry points: hubdl, hfdx, msd

Features

Feature	Description
Multi-hub	Hugging Face (`hf`) and ModelScope (`ms`) behind one core
Dual CLI aliases	`hfdx` → HF, `msd` → ModelScope (drop-in style entrypoints)
Fast listing	HF: siblings fast path + resumable tree walk; MS: paginated `repo/files`
Download engine	Default aria2c (multi-connection, `.aria2` resume); optional wget
Rate limit	`--rate-mbps` (megabit/s) → aria2 overall limit
Integrity	L0 resume → L1 size → L3 SHA-256 (when available) → L4 repo report
Safe resume	Incomplete files with `size < expected` are not deleted
Path sandbox	Reject `..`, absolute paths, and writes into control dirs (`.hubdl`, …)
Single instance	`flock` on `.hubdl/hubdl.lock` (Linux)
Ops modes	`--dry-run`, `--verify-only`, `--offline`, `--refresh`
Structured outputs	`plan.json`, `verify_report.json`, `last_run.json`, `hash_cache.tsv`

CLI entry points

Command	Hub	Notes
`hubdl hf <REPO>`	Hugging Face	Primary unified CLI
`hubdl ms <REPO>`	ModelScope	Primary unified CLI
`hfdx <REPO>`	Hugging Face	Alias of `hubdl hf`
`msd <REPO>`	ModelScope	Alias of `hubdl ms`

Also accepted: hubdl huggingface …, hubdl modelscope ….

Current version: 0.1.2 (PyPI · CHANGELOG)
Metadata directory: <local-dir>/.hubdl/
Manifest format: hubdl-manifest-v1 — size TAB sha256|- TAB path

Requirements

System (Linux recommended)

Package	Required	Role
Python ≥ 3.10	Yes	Runtime
aria2c	Recommended	Default download engine
wget	Optional	`--tool wget`
flock / fcntl	Recommended	Single-instance lock (Linux)
ca-certificates	Recommended	HTTPS

# Debian / Ubuntu
sudo apt update
sudo apt install -y python3 python3-venv python3-pip aria2 ca-certificates
# optional
sudo apt install -y wget

Python

No third-party runtime dependencies (stdlib only). Dev extras: pytest, pytest-cov.

Installation

Method	Use when
PyPI `pip install hubdl`	Normal use (official)
GitHub Release wheel	Offline / no PyPI access
`pip install git+https://…`	Install from a specific commit or tag
Editable clone	Developing hubdl itself

Release process for maintainers: docs/RELEASE.md · docs/PUBLISHING.md.

From PyPI (official)

sudo apt install -y python3 python3-venv aria2 ca-certificates   # Linux, once
python3 -m venv .venv
source .venv/bin/activate   # Windows: .\.venv\Scripts\Activate.ps1
pip install -U pip
pip install hubdl
hubdl --version
hfdx --help
msd --help

PyPI: https://pypi.org/project/hubdl/
Console scripts: hubdl, hfdx, msd
Python runtime dependencies: none (stdlib only). Downloader: system aria2c.

From a GitHub Release wheel (offline)

Download the .whl from Releases:

python3 -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install ./hubdl-0.1.2-py3-none-any.whl
hubdl --version

From Git (optional)

python3 -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install "git+https://github.com/mrhegit/hubdl.git@v0.1.2"
# or track main:
# pip install "git+https://github.com/mrhegit/hubdl.git"
hubdl --version

Editable install (contributors)

git clone https://github.com/mrhegit/hubdl.git && cd hubdl
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
hubdl --version
pytest -q

Without activating the venv: .venv/bin/hubdl (Linux) or .venv\Scripts\hubdl.exe (Windows).
.venv/ is gitignored and must not be committed.

Quick start

Assumes Install & use is done and the venv is active (hubdl --version works).

Hugging Face

# Plan only (no download, no delete)
hubdl hf gpt2 --local-dir ./gpt2 --dry-run
# equivalent alias
hfdx gpt2 --local-dir ./gpt2 --dry-run

# Full download (L3 default = auto)
hubdl hf Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05

# Private / gated model
export HF_TOKEN=hf_xxxxxxxx
hubdl hf meta-llama/Llama-3.2-1B \
  --local-dir ./llama \
  --hf_username YOUR_HF_NAME

# Mirror endpoint
export HF_ENDPOINT=https://hf-mirror.com
hubdl hf gpt2 --local-dir ./gpt2

ModelScope

hubdl ms Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05 --dry-run
# equivalent alias
msd Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen05 --dry-run

# Shared LAN: hard cap 35 Mb/s, single connection
hubdl ms deepseek-ai/DeepSeek-V4-Flash-0731 \
  --local-dir /data1/models/DeepSeek-V4-Flash-0731 \
  --profile lan-shared

Verify only

# Re-check integrity (verify-only maps auto → full)
hubdl hf owner/model --local-dir ./m --verify-only --verify-hash on
hubdl ms owner/model --local-dir ./m --verify-only --offline

Background job (nohup)

mkdir -p /data1/model_download_logs
LOG=/data1/model_download_logs/hubdl_$(date +%Y%m%d_%H%M%S).log
nohup hubdl ms deepseek-ai/DeepSeek-V4-Flash-0731 \
  --local-dir /data1/models/DeepSeek-V4-Flash-0731 \
  --profile lan-shared \
  --progress none \
  > "$LOG" 2>&1 &
echo $! > /data1/model_download_logs/hubdl.pid

Architecture

Option A: dual CLI + shared library.

hubdl/
├── src/hubdl/
│   ├── cli.py                 # hubdl / hfdx / msd entrypoints
│   ├── core/                  # shared pipeline
│   │   ├── sandbox.py         # path jail, reserved dirs
│   │   ├── plan.py            # size + optional L3
│   │   ├── verify.py          # L3 + L4 report
│   │   ├── download.py        # aria2 / wget
│   │   ├── cache.py           # hash_cache.tsv
│   │   ├── httputil.py        # host pin, retries
│   │   └── pipeline.py        # orchestration
│   └── hubs/
│       ├── huggingface.py     # list + resolve URLs
│       └── modelscope.py
├── scripts/                   # repo wrappers without install
└── tests/

Layer	Responsibility
CLI	Parsing, profiles, hub selection, exit codes
core	Plan, verify, download, lock, atomic IO, hash policy
hubs.*	File listing + URL resolution only

CLI reference

Positional

Argument	Description
`hub`	Only for `hubdl`: `hf` / `ms` (or long names)
`repo_id`	`owner/name`; HF also allows legacy single-segment ids (e.g. `gpt2`)

Common options

Option	Default	Description
`--local-dir DIR`	`./<repo_name>`	Root for weights + `.hubdl/`
`--revision REV`	`main` (hf) / `master` (ms)	Branch / tag / commit
`--dataset`	off	Treat repo as a dataset
`--token TOKEN`	env (see below)	Access token
`--hf_token TOKEN`		Alias of `--token` for HF
`--hf_username NAME`		HF gated hint (Bearer still uses token)
`--endpoint URL`	hub default	Override API/download base
`--include PAT`	all	Include glob (`*`); repeatable
`--exclude PAT`	none	Exclude glob; repeatable
`--tool aria2c\|wget`	`aria2c`	Download backend
`-x N`	`4`	aria2 connections per file
`-j N`	`5`	aria2 concurrent files
`--rate-mbps N`	unlimited	Overall cap in megabit/s
`--max-download-limit L`		Pass-through aria2 limit string (e.g. `4480K`)
`--dry-run`		Fetch + plan only
`--verify-only`		L3+L4 only; no download; no delete
`--offline`		No network list (requires existing manifest); only with `--verify-only`
`--refresh`		Force re-list (ignore fingerprint cache)
`--verify-hash MODE`	hf: `auto`, ms: `on`	`auto` \| `on` \| `off`
`--no-verify-hash`		Same as `--verify-hash off`
`--verify-mode MODE`	`auto`	`full` \| `light` \| `auto` (`verify-only` + `auto` → `full`)
`--verify-jobs N\|auto`	auto	SHA-256 parallelism
`--no-fix-bad`		Never delete bad files
`--trust-aria2-checksum`	off	Seed hash cache from size-aligned downloads (use carefully)
`--progress MODE`	`aria2`	`aria2` \| `none` (`line` reserved)
`--profile NAME`		See Profiles
`--allow-http`	off	Allow `http://` endpoints
`--proxy URL`	env (see below)	HTTP(S) proxy only (`http://host:port`); `none`/`direct`/`off`/`false`/`0` = force direct. No SOCKS
`--no-proxy LIST`	env	Comma-separated bypass hosts (aria2 + simple API match)
`-h` / `--help`		Help
`--version`		Print version

Rate limit note

Spoken “35 megabit/s” → --rate-mbps 35 → roughly 4480K in aria2 (N × 1024 / 8 KiB/s).

Profiles

Profile	Effect
`lan-shared`	`--rate-mbps 35` if unset, `-x 1 -j 1`
`fast`	No rate limit, `-x 4 -j 5`; on HF sets `verify-hash=off`
`safe-verify`	`--verify-only`, `--verify-mode full`, `--verify-hash on`, no delete

Workflow & modes

download (default)

fetch list → plan (size + optional L3)
           → scrub orphan .aria2 (size-ok payload) → re-plan if needed
           → download round=1 → plan
           → if still need: download round=2 → plan
           → verify L3+L4

ModelScope multi-conn notes (aria2 only):

Rule	Behavior
Small files (`size ≤ 32 MiB` or unknown)	Per-URI `split=1` / single connection (avoids intermittent 404 under multi-range)
Large weights	Keep user `-x` multi-connection
Global `--min-split-size`	`16M` (HF + MS); reduces pointless splits on tiny objects
Round 2 (if need remains)	MS: force single-conn recovery (`-x1 -j1`); HF: same strategy retry
Orphan `.aria2`	If payload size already matches expected, control file is removed (size-only; L3 still owns content trust)
`wget`	No multi-range policy; prefer `aria2c` for MS reliability

dry-run

fetch → plan → exit 0

verify-only

fetch | offline → verify → exit 0/2

(auto verify-mode becomes full so stale plan.json is not trusted lightly.)

Exit codes

Code	Meaning
0	Success (download+verify, dry-run, or verify-only pass)
1	Download still incomplete (`need` non-empty)
2	L3/L4 verification failed
3	Directory lock conflict
4	Invalid args / runtime error
130	Interrupted (Ctrl+C)

Integrity (L0–L4)

Level	Meaning
L0	Transfer resume: aria2 `.aria2` / wget `-c`
L1	Local file size equals expected size from listing
L3	Local SHA-256 equals expected (LFS / API when present)
L4	All listed files present, no leftover `.aria2`, counts and matched bytes consistent

--verify-hash auto (HF default): enable L3 if any of:

share of files with sha ≥ 30%, or
all files ≥ 20 MiB have sha, or
files covering top ~90% of total bytes all have sha

ModelScope default is on (API usually provides Sha256).

Files without a digest are recorded as size_ok_nohash when L3 is enabled for the run.

Delete policy

Local state	Plan	Delete
Missing	need	No
Has `.aria2`	need (in progress)	No (payload kept); size-ok orphan `.aria2` may be scrubbed
`size < expected`	need (resume)	No
`size > expected`	need	Yes if `fix_bad`
Size OK, hash OK	skip	No
Size OK, hash bad	need	Yes if `fix_bad`
Hash IO error	need	No
Path into `.hubdl` / `..` / absolute	rejected	No
`--dry-run` / `--verify-only`	—	Forced no

Deletes only occur inside --local-dir after sandbox checks.

Metadata directory

Path: <local-dir>/.hubdl/ (mode preferably 0700).

File	Purpose
`manifest`	Listing v1: size, sha256 or `-`, path
`repo_meta.json`	Counts, endpoint, hub, fingerprint fields
`list_fingerprint`	Skip re-list when options unchanged
`plan.json`	Last plan statistics
`need_download.txt`	Paths still needed
`last_need.txt`	Snapshot of last download batch
`aria2_urls.txt`	aria2 input (may briefly hold Authorization; cleared after download)
`hash_cache.tsv`	L3 cache (path, size, mtime, expected, digest)
`verify_report.json`	Last L3+L4 report
`download.log`	Download log (tee / append)
`progress.json`	Progress snapshot
`last_run.json`	Phase + exit code
`hubdl.lock`	Single-instance lock
`verify_hash.flag`	Last decided L3 on/off (CLI still re-evaluated each run)
`repo_metadata.json`	Raw HF metadata when using HF hub
`list_state` / `manifest.partial`	Temporary HF tree-walk checkpoints

Do not publish or artifact-upload .hubdl/ without scrubbing secrets.

Environment variables

Variable	Applies to	Description
`HF_TOKEN` / `HUGGING_FACE_HUB_TOKEN`	HF	Access token
`HF_USERNAME`	HF	Username hint for gated repos
`HF_ENDPOINT`	HF	Mirror base (default `https://huggingface.co`)
`MODELSCOPE_API_TOKEN` / `MODELSCOPE_TOKEN`	MS	Access token
`MODELSCOPE_ENDPOINT`	MS	Default `https://www.modelscope.cn`
`HUBDL_PROXY`	both	Preferred hubdl proxy URL (or `none` to force direct)
`HUBDL_NO_PROXY`	both	Preferred no-proxy list
`https_proxy` / `HTTPS_PROXY` / `http_proxy` / `HTTP_PROXY` / `ALL_PROXY` / `all_proxy`	both	Standard proxy env (after `HUBDL_PROXY`); applied to API + aria2 + wget
`no_proxy` / `NO_PROXY`	both	Bypass list
`HUBDL_ALLOW_HTTP`	both	`1` allows plain `http://` endpoints

Proxy notes

Priority: CLI --proxy > HUBDL_PROXY > https_proxy / http_proxy / ALL_PROXY.

Only HTTP(S) forward proxies are supported (aria2 / urllib / wget common capability).
SOCKS is rejected — point hubdl at a local HTTP port (Clash/V2Ray mixed / HTTP port), not socks5://.

# Recommended: local HTTP proxy (API + download, full path)
export https_proxy=http://127.0.0.1:7890
hubdl hf gpt2 --local-dir ./gpt2

# Explicit CLI (overrides env)
hubdl ms org/model --local-dir ./m --proxy http://127.0.0.1:7890 --no-proxy localhost,127.0.0.1

# Force direct when the shell has a global proxy
hubdl hf gpt2 --local-dir ./gpt2 --proxy none
# same: export https_proxy=none

Child downloaders do not inherit ambient proxy env; hubdl injects or clears explicitly.
aria2 also clears protocol-specific options so conf/https_proxy cannot override --proxy none.
Credentials in the proxy URL are redacted in logs (still visible to same-uid ps); prefer unauthenticated localhost proxies.
last_run.json stores scheme only (http / https / none).

Hub differences

	Hugging Face	ModelScope
List API	metadata + siblings / recursive tree	`/api/v1/{models\|datasets}/…/repo/files`
Content hash source	Often LFS `sha256` / `oid`	API `Sha256` field
Default `--verify-hash`	`auto`	`on`
Default revision	`main`	`master`
Legacy repo id	Yes (`gpt2`)	`owner/name` only
Gated repos	Token required; username recommended	Token via env/`--token`
Large file counts	Resumable tree walk	Paged API (hard stop ~200 pages)

Capabilities are not 100% symmetric; that is intentional.

Security

Summary (see also SECURITY.md):

Sandbox: repository paths cannot target .hubdl, .hfdx, .msd, .hfd, or .git.
HTTP pin: requests must stay on the configured endpoint host; cross-host redirects are blocked (reduces token exfiltration).
aria2: --check-integrity=true; URL list is wiped after download.
Tokens: prefer env vars; do not commit tokens or pack .hubdl/ into CI artifacts.
Mirrors: a mirror can still observe your token on the first hop — treat mirrors as trusted parties.
fix_bad: can delete overflow/hash-bad files inside local_dir only; use --no-fix-bad when unsure.

Troubleshooting

Symptom	What to check
Exit 3 / lock	Another hubdl on the same `--local-dir`; remove stale process before touching lock
Exit 2 / verify fail	Open `.hubdl/verify_report.json` → `bad` list; re-run download or `--verify-only`
Exit 1 / incomplete	Re-run the same command to resume; check `.hubdl/download.log`
Slow / limited speed	Is `--rate-mbps` or `lan-shared` set? Try adjusting `-x` / `-j`
401 / gated	Token validity; HF username; accept model license on the website
Hash slow	First full L3 is disk-bound; raise `--verify-jobs`; rely on `hash_cache.tsv` later
Treesize WARN (HF)	Incomplete listing or filter; try `--refresh` or official endpoint
flock WARN on Windows	Expected; use Linux for production locks
aria2 not found	`sudo apt install aria2` or `--tool wget`

tail -f <local-dir>/.hubdl/download.log
cat <local-dir>/.hubdl/last_run.json
cat <local-dir>/.hubdl/verify_report.json
cat <local-dir>/.hubdl/plan.json

Development

source .venv/bin/activate   # or Windows Activate.ps1
pip install -e ".[dev]"
pytest -q
# with coverage:
pytest --cov=hubdl --cov-report=term-missing
python -m hubdl hf --help

Project layout for contributors mirrors the Architecture section.

Tests

Layer	Coverage
Unit	sandbox, hash policy, sha normalize, patterns, HTTP host pin, atomic IO, cache, lock
Component	manifest IO, plan (missing/incomplete/overflow/hash/cache), verify pass/fail/light
Mocked hub	ModelScope page list, HF siblings + gated
Pipeline	offline `--verify-only`, dry-run, download incomplete (mocked aria2)
CLI	defaults, profiles, offline rules, entry aliases

pytest -q
pytest --cov=hubdl --cov-report=term-missing

CI runs on Python 3.10–3.12 (GitHub Actions). Before a PR:

pytest -q
hubdl hf gpt2 --dry-run --local-dir /tmp/hubdl-gpt2 (network)
hubdl ms Qwen/Qwen2.5-0.5B-Instruct --dry-run --local-dir /tmp/hubdl-qwen --no-verify-hash

Roadmap

Shared core + HF / MS backends
Dual CLI aliases (hfdx, msd)
Official PyPI distribution (pip install hubdl)
CI + Trusted Publishing (OIDC) to PyPI / GitHub Release
Stronger single-line progress HUD (hfd-style)
Optional migration from legacy .msd / .hfdx meta dirs
Broader integration tests against mocked Hub APIs
Optional standalone binary packaging

Credits & license

Inspired by / thanks to:

hfd by padeoe — Hugging Face listing + aria2 workflow
Local msd / hfdx tooling — L3/L4, sandbox, operational flags

License: Apache License 2.0
Attribution: NOTICE
Security policy: SECURITY.md

Model weights and Hub content remain under their respective licenses.

Language

Language	File
English (default)	README.md
简体中文	README.zh-CN.md

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
.github/workflows		.github/workflows
docs		docs
scripts		scripts
src/hubdl		src/hubdl
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
README.zh-CN.md		README.zh-CN.md
SECURITY.md		SECURITY.md
pyproject.toml		pyproject.toml

Folders and files

Latest commit

History

Repository files navigation