From 345599878fd8520c3043acac2a0672b52d745a8d Mon Sep 17 00:00:00 2001 From: Michael Dwan Date: Mon, 4 May 2026 17:22:50 -0600 Subject: [PATCH 1/2] docs: flesh out example READMEs for managed weights The existing READMEs listed commands without explaining what managed weights are, what the cog.yaml fields mean, or common gotchas like .dockerignore. An MLE picking these up as a starting point would have to reverse-engineer everything from the YAML comments. - resnet: added conceptual overview, field-by-field cog.yaml docs, file layout, .dockerignore explanation, adaptation guide - managed-weights: clarified this is a test fixture (not a model template), linked to resnet as the real starting point, restructured for scannability - managed-weights/cog.yaml: replaced hardcoded personal registry URL with placeholder, removed image arg from push commands --- examples/managed-weights/README.md | 63 +++++++------ examples/managed-weights/cog.yaml | 5 +- examples/resnet/README.md | 137 +++++++++++++++++++++++++++-- 3 files changed, 172 insertions(+), 33 deletions(-) diff --git a/examples/managed-weights/README.md b/examples/managed-weights/README.md index 14fe78abee..0a68342b74 100644 --- a/examples/managed-weights/README.md +++ b/examples/managed-weights/README.md @@ -1,69 +1,82 @@ # examples/managed-weights -A minimal cog model used to exercise the v1 managed-weights OCI pipeline -end-to-end. It produces an OCI image index carrying a model image manifest -and per-weight manifests. +Test fixture for the v1 managed-weights OCI pipeline. This isn't a model you'd +deploy -- it's an end-to-end exercise of weight import, packing, pushing, and +runtime validation. -The predictor validates weight files on disk against `weights.lock` -(generated by `cog weights import`), errors on any missing files, and -returns a per-weight status summary from predict(). +If you're looking for a starting point for a real model, see +[`examples/resnet`](../resnet/) instead. -## Populating `weights/` +## What this does -The weight directory is git-ignored because it's ~5 GB. Populate it by -cloning the HuggingFace repo and copying everything except `.git/`: +The predictor doesn't do inference. Instead, it reads `weights.lock` at setup, +validates that every expected file exists on disk with the correct size and +digest, and returns a per-weight status summary from `predict()`. It's a +smoke test for the weight pipeline. + +The `cog.yaml` declares two weight sources to exercise both code paths: + +- **`parakeet`** -- local directory (`uri: weights`), filtered with `include` + globs. You populate this manually by cloning from HuggingFace. +- **`minilm`** -- HuggingFace repo (`uri: hf://sentence-transformers/all-MiniLM-L6-v2`), + filtered with `exclude` globs. Downloaded automatically by `cog weights import`. + +## Setup + +### Populate the local weights + +The `weights/` directory is git-ignored (~5 GB). Clone the HuggingFace repo +and copy everything except `.git/`: ```bash -# One-time: clone the weights somewhere outside this repo git clone https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3 ~/hf/parakeet -# Copy everything except .git into examples/managed-weights/weights/ mkdir -p examples/managed-weights/weights rsync -a --exclude=.git/ ~/hf/parakeet/ examples/managed-weights/weights/ ``` -You can substitute any directory of model files; the pipeline is +You can substitute any directory of model files -- the pipeline is content-agnostic. -## Importing weights - -After populating (or changing) `weights/`, regenerate the lockfile: +### Import weights and generate the lockfile ```bash cd examples/managed-weights cog weights import ``` -This writes `weights.lock`. The predictor's `setup()` reads this file and -validates that all expected files exist at the declared targets. +This fetches the HuggingFace weights (minilm), hashes everything (both local +and remote), and writes `weights.lock`. ## Running the pipeline +### Option A: Full build + push + Start a local registry (or point at any registry you control): ```bash docker run -d --rm -p 5000:5000 --name cog-test-registry registry:3 ``` -Build and push the full bundle. Presence of `weights:` in `cog.yaml` -triggers the OCI bundle format automatically. +Build and push. The `weights:` block in `cog.yaml` triggers the OCI bundle +format automatically: ```bash cd examples/managed-weights -cog push localhost:5000/managed-weights +cog push ``` -Or run the weight pipeline in isolation (no model image): +### Option B: Weight pipeline only (no model image) ```bash cd examples/managed-weights cog weights build -cog weights push localhost:5000/managed-weights +cog weights push ``` -## Testing locally +### Testing locally -Build the image and run it with weights bind-mounted: +Build the image and run with weights bind-mounted: ```bash cd examples/managed-weights @@ -88,5 +101,5 @@ crane manifest localhost:5000/managed-weights:latest | jq . crane ls localhost:5000/managed-weights ``` -Weight manifests are pushed under tags of the shape +Weight manifests are pushed under tags like `weights--<12-hex-digest>` (see `pkg/model/weight_pusher.go`). diff --git a/examples/managed-weights/cog.yaml b/examples/managed-weights/cog.yaml index b3601bbc0f..2086ac55b3 100644 --- a/examples/managed-weights/cog.yaml +++ b/examples/managed-weights/cog.yaml @@ -8,10 +8,9 @@ # cog weights import # # Build and push the full bundle: -# cog push localhost:5000/managed-weights +# cog push -image: registry.cloudflare.com/3515b24d58ec616d11f4ce4290a02ac4/md/examples/managed-weights -# image: localhost:5000/md/examples/managed-weights +image: /managed-weights build: gpu: false diff --git a/examples/resnet/README.md b/examples/resnet/README.md index e3701f64a5..ce4434b3c8 100644 --- a/examples/resnet/README.md +++ b/examples/resnet/README.md @@ -3,23 +3,150 @@ ResNet50 image classifier (microsoft/resnet-50 from HuggingFace) packaged with v1 managed weights. Takes an image, returns top-3 ImageNet classes. -## Usage +Use this as a starting point for packaging a real model with managed weights. -Import weights from HuggingFace and generate the lockfile: +## What are managed weights? + +Managed weights separate your model weights from your model image. Instead of +baking multi-GB weight files into the Docker image (slow builds, huge layers), +cog packs them into dedicated OCI layers that get mounted at runtime. + +The key idea: your `predict.py` reads weights from a path like +`/src/weights/resnet50`, but those files don't live inside the Docker image -- +they arrive separately and get overlaid at that path when the container starts. + +## File layout + +``` +examples/resnet/ +├── cog.yaml # model config -- declares weights, build settings +├── predict.py # predictor -- loads weights from target path +├── requirements.txt # python deps +├── weights.lock # generated by `cog weights import` -- don't hand-edit +├── .dockerignore # keeps weights/ out of the Docker build context +├── .gitignore # keeps weights/ out of git +├── hotdog.png # test image +└── cat.png # test image +``` + +After running `cog weights import`, you'll also have: + +``` +weights/resnet50/ # the actual weight files (git-ignored) +├── config.json +├── model.safetensors +└── preprocessor_config.json +``` + +## How `cog.yaml` works + +```yaml +weights: + - name: resnet50 + source: + uri: hf://microsoft/resnet-50 # where to fetch from + exclude: # files to skip + - "pytorch_model.bin" + - "flax_model.msgpack" + - "tf_model.h5" + - "README.md" + - ".gitattributes" + target: /src/weights/resnet50 # where files appear in the container +``` + +**`name`** -- an identifier for this weight set. Used in lockfile entries and +OCI tags. Pick something short and descriptive. + +**`source.uri`** -- where the weights come from. Two formats: +- `hf:///` -- pulls from HuggingFace Hub +- A local directory path (e.g. `weights/`) -- uses files already on disk + +**`source.exclude`** -- glob patterns for files to skip. Most HF repos ship +weights in multiple formats (PyTorch, TF, Flax, ONNX). Exclude the ones you +don't need -- it'll save gigabytes. + +**`target`** -- the absolute path where weight files land inside the container. +Your `predict.py` loads from this path. Must start with `/`. + +## Getting started + +### 1. Import weights + +This fetches weight files from HuggingFace into `weights/` locally and +generates `weights.lock`: ```sh cd examples/resnet cog weights import ``` -Run a prediction locally (weights are bind-mounted): +The lockfile records digests and sizes for every file. It's how cog knows +whether weights have changed on subsequent imports. Commit `weights.lock` +to version control -- don't commit the `weights/` directory itself. + +### 2. Run a prediction locally ```sh cog predict -i image=@hotdog.png ``` -Build and push to a registry: +Locally, cog bind-mounts `weights/` into the container at the `target` path. +You don't need to push anything to test. + +### 3. Build and push ```sh -cog push /resnet +cog push +``` + +This builds the model image and pushes it to the registry specified by `image:` +in `cog.yaml`, alongside the weight layers as an OCI image index. The weights +and model image are separate artifacts in the registry -- the image index ties +them together. + +## Important: `.dockerignore` + +The `.dockerignore` in this directory excludes `weights/` from the Docker build +context. This is critical. Without it, Docker sends the entire weights directory +(potentially many GB) to the build daemon on every `cog build`, and the weights +would get baked into the image -- defeating the whole point. + +If you're starting a new model, make sure your `.dockerignore` includes: + +``` +weights/ +.cog/weights-cache/ ``` + +## Adapting this for your own model + +1. Copy this directory as a starting point +2. Edit `cog.yaml`: + - Change `source.uri` to your HuggingFace repo (or a local path) + - Adjust `exclude` patterns for the formats you don't need + - Set `target` to wherever your code expects to find the weights + - Set `image` to your registry destination (required for `cog push`) +3. Edit `predict.py` to load your model from `WEIGHTS_DIR` +4. Update `requirements.txt` with your dependencies +5. Run `cog weights import` to fetch weights and generate the lockfile +6. Test with `cog predict` +7. Push with `cog push` + +### Using local weights instead of HuggingFace + +If you already have weights on disk (downloaded separately, trained locally, +etc.), point the source at a local directory: + +```yaml +weights: + - name: my-model + source: + uri: my-weights-dir/ + include: + - "*.safetensors" + - "*.json" + target: /src/weights/my-model +``` + +Then run `cog weights import` as usual -- it'll hash the local files and +generate the lockfile. From dcb6b2a6a57f0cdc8545be3e6dc59211be87e90f Mon Sep 17 00:00:00 2001 From: Michael Dwan Date: Mon, 4 May 2026 17:28:24 -0600 Subject: [PATCH 2/2] fix: correct weight storage description -- files live in cache, not project dir HF-sourced weights go to ~/.cache/cog/weights/, not a weights/ dir in the project. cog predict assembles ephemeral hardlink trees under .cog/mounts/ from the cache. Updated both READMEs to describe the actual flow. --- examples/managed-weights/README.md | 5 ++-- examples/resnet/README.md | 41 ++++++++++++------------------ 2 files changed, 19 insertions(+), 27 deletions(-) diff --git a/examples/managed-weights/README.md b/examples/managed-weights/README.md index 0a68342b74..b9a99d6708 100644 --- a/examples/managed-weights/README.md +++ b/examples/managed-weights/README.md @@ -45,8 +45,9 @@ cd examples/managed-weights cog weights import ``` -This fetches the HuggingFace weights (minilm), hashes everything (both local -and remote), and writes `weights.lock`. +This fetches the HuggingFace weights (minilm), ingresses everything (both +local and remote files) into the content-addressed store at +`~/.cache/cog/weights/`, and writes `weights.lock`. ## Running the pipeline diff --git a/examples/resnet/README.md b/examples/resnet/README.md index ce4434b3c8..0791ae5c09 100644 --- a/examples/resnet/README.md +++ b/examples/resnet/README.md @@ -23,20 +23,18 @@ examples/resnet/ ├── predict.py # predictor -- loads weights from target path ├── requirements.txt # python deps ├── weights.lock # generated by `cog weights import` -- don't hand-edit -├── .dockerignore # keeps weights/ out of the Docker build context -├── .gitignore # keeps weights/ out of git +├── .dockerignore # keeps local weight dirs out of the Docker build context +├── .gitignore # keeps local weight dirs and .cog/ out of git ├── hotdog.png # test image └── cat.png # test image ``` -After running `cog weights import`, you'll also have: - -``` -weights/resnet50/ # the actual weight files (git-ignored) -├── config.json -├── model.safetensors -└── preprocessor_config.json -``` +Weight files themselves don't live in the project directory. `cog weights import` +downloads them into a content-addressed store at `~/.cache/cog/weights/` (override +with `$COG_CACHE_DIR`). When you run `cog predict`, cog assembles a temporary +directory under `.cog/mounts/` using hardlinks from the store and bind-mounts it +into the container at the `target` path. The mount dir is cleaned up when the +container stops. ## How `cog.yaml` works @@ -72,7 +70,7 @@ Your `predict.py` loads from this path. Must start with `/`. ### 1. Import weights -This fetches weight files from HuggingFace into `weights/` locally and +This downloads weight files from HuggingFace into the local cache and generates `weights.lock`: ```sh @@ -82,7 +80,7 @@ cog weights import The lockfile records digests and sizes for every file. It's how cog knows whether weights have changed on subsequent imports. Commit `weights.lock` -to version control -- don't commit the `weights/` directory itself. +to version control. ### 2. Run a prediction locally @@ -90,8 +88,8 @@ to version control -- don't commit the `weights/` directory itself. cog predict -i image=@hotdog.png ``` -Locally, cog bind-mounts `weights/` into the container at the `target` path. -You don't need to push anything to test. +Locally, cog assembles the weight files from the cache and bind-mounts them +into the container at the `target` path. You don't need to push anything to test. ### 3. Build and push @@ -106,17 +104,10 @@ them together. ## Important: `.dockerignore` -The `.dockerignore` in this directory excludes `weights/` from the Docker build -context. This is critical. Without it, Docker sends the entire weights directory -(potentially many GB) to the build daemon on every `cog build`, and the weights -would get baked into the image -- defeating the whole point. - -If you're starting a new model, make sure your `.dockerignore` includes: - -``` -weights/ -.cog/weights-cache/ -``` +The `.dockerignore` excludes `weights/` and `.cog/weights-cache/` from the +Docker build context. This matters if you're using local directory weight +sources -- without it, Docker would send the full weight directory to the +build daemon on every `cog build`. ## Adapting this for your own model