v0.8.0 — LLM models and the GPU kit

LLM models for offline GPU hosts

A new Models page downloads open models from Hugging Face for the GPU benchmark image and hands them over in parts under 1 GB.

• Curated list for NVIDIA H200 — smoke test (Qwen2.5-0.5B), 8B baseline in BF16 and FP8, MoE (gpt-oss-20b, Qwen3-30B-A3B), mid-size dense (phi-4, Qwen3-32B), large single-GPU (gpt-oss-120b, Llama-3.3-70B FP8) and multi-GPU (Qwen2.5-72B, Qwen3-235B-A22B FP8), with size, precision and the number of GPUs each needs. Or enter any Hugging Face model id.
• Only what vLLM loads is downloaded (config, tokenizer, *.safetensors), from one resolved commit. Weights are checked against the sha256 Hugging Face publishes, small files against their Git blob id; a mismatch never becomes "ready".
• Resumable: downloads continue after a network error, a restart or an upgrade (HTTP range requests). Space is checked before a download starts.
• No copies on disk: a model is stored once as a folder; its download is an uncompressed tar stream generated from that folder and split into parts of at most 900 MB.
• install-<model>.sh checks the parts, unpacks with cat | tar -x straight into /opt/models/<model>, and verifies every file. download-<model>.sh fetches all parts with resume and checksum checks.
• Only open models: gated repositories (Meta Llama, Google Gemma) are refused, and no tokens are stored. The list uses RedHatAI's open copies of Llama.

New setting LAYERSMITH_MODEL_DIR (default $DATA/models, also under Settings → Storage) and LAYERSMITH_HF_ENDPOINT for a mirror. See docs/models.md.

GPU kit: one command on the GPU host

A GPU benchmark build page now has a GPU kit: tick models, prepare the image parts, download download-all.sh and run it where LayerSmith is reachable. It fetches the image parts, model parts and scripts, resumes, and verifies every part. On the GPU host:

bash setup.sh

setup.sh checks the host and the free space per filesystem (archive, podman's $TMPDIR scratch space, the image store, the models directory) before it touches anything, restores and loads the image, installs each model into /opt/models, generates the CDI spec with sudo nvidia-ctk cdi generate if it is missing, and starts the dashboard. Every step is skipped once done, so it can be re-run. --no-start, --models-dir and TMPDIR= are supported. start_container.sh starts the dashboard again later (GPUS=0, MODELS_DIR=…, or any gpu-bench command).

Also

• The GPU benchmark guide mounts /opt/models, gives the real image size (about 10–15 GB unpacked, 4.6 GB archive and what loading needs), and covers statfs /models: no such file or directory, unresolvable CDI devices and no space left on device during podman load.
• The generated scripts run on bash 4.2 (RHEL/CentOS 7) and newer, and pass shellcheck.

Verified: a real download of Qwen2.5-0.5B-Instruct from Hugging Face, fetched with download-*.sh and installed with install-*.sh, gives weights byte-identical to Hugging Face's published sha256. setup.sh is covered end to end in the test suite with stand-ins for podman and the NVIDIA tools; it has not been run on a real GPU host yet.

Upgrade notes

• Unfinished model downloads resume when LayerSmith starts. As in 0.7.0, a build running during an upgrade is marked failed — let it finish first.
• The new llm_models table is created automatically.

ghcr.io/r0lfi/layersmith:0.8.0