Skip to content

Troubleshooting

Jump to bottom
Holden Salomon edited this page Jun 14, 2026 · 11 revisions

Troubleshooting

Container exits immediately

Check logs:

docker compose logs winnow

Common causes:

  • Missing required env var — IMMICH_URL or API_KEY not set
  • Cannot reach Immich — check the URL and that Immich is running; use http:// not https:// unless you have TLS set up

"Immich API key is invalid or expired (401 Unauthorized)"

Your API key has been revoked or expired. Generate a new one in Immich → Account Settings → API Keys and update API_KEY in your .env.

Getting more log detail

Set VERBOSE=true in .env (or pass -e VERBOSE=true) to enable DEBUG-level output on the console. Useful for tracing exactly which images are being fetched, filtered, and selected.

The log file always captures DEBUG regardless of VERBOSE. It is written to the output volume:

docker exec winnow cat /app/frigate_train/winnow.log

"No people found" / nothing processed

  • Make sure Immich has completed face recognition and you have named people in your library
  • YEARS_FILTER defaults to 10 years — increase it if your tagged photos are older
  • MIN_FACE_COUNT (default 3) skips people with few photos — lower or remove it

Frigate upload fails

  • Confirm FRIGATE_URL is reachable from inside the container: docker exec winnow curl $FRIGATE_URL/api/stats
  • Check Frigate v0.16+ — older versions don't have the face training API
  • Set DRY_RUN=true to verify selection without uploading

Model fails to download

winnow downloads InsightFace Buffalo_L (~300 MB) on first run if it isn't already cached.

  • Ensure the container has internet access
  • Confirm the models volume is mounted and writable (/models inside the container)
  • If behind a proxy, set HTTP_PROXY / HTTPS_PROXY env vars

Running on CPU (no GPU)

Use the :cpu image tag (ghcr.io/sudolulo/winnow:cpu) which omits the GPU base entirely. Or set FORCE_CPU=true with any other tag to disable GPU at runtime.

Everything works on CPU but embedding computation is slower — about 8–10× vs GPU for InsightFace (see Benchmarks).

Memory: Set a minimum container memory limit of 2 GB (mem_limit: 2g).

GPU not being used

NVIDIA

  1. Confirm the GPU is visible inside the container:

    docker exec winnow nvidia-smi

    If this fails, the container doesn't have GPU access — check the deploy.resources.reservations.devices block in compose.yml and that the NVIDIA Container Toolkit is installed on the host.

  2. Check which ONNX providers are available:

    docker exec winnow /app/.venv/bin/python3 -c \
  "import onnxruntime as ort; ort.preload_dlls(cuda=True, cudnn=True); print(ort.get_available_providers())"

    Expected (GPU working): ['CUDAExecutionProvider', 'CPUExecutionProvider', ...]
    GPU not working: only ['CPUExecutionProvider'] listed.

  3. If only CPU providers appear even though nvidia-smi works, you may be running an older image with a packaging bug. Update:

    docker compose pull && docker compose up -d

AMD (ROCm)

  1. Confirm devices are passed through — your compose.yml should have:

    devices:
  - /dev/kfd
  - /dev/dri
group_add:
  - video
  - render

  2. Verify ROCm sees the GPU:

    docker exec winnow /app/.venv/bin/python3 -c \
  "import onnxruntime as ort; print(ort.get_available_providers())"

    Expected: ['ROCmExecutionProvider', 'CPUExecutionProvider']

Intel Arc / iGPU

  1. Confirm the DRI device is passed through:

    devices:
  - /dev/dri
group_add:
  - render

  2. Ensure OPENVINO_DEVICE=GPU is set — without it, OpenVINO runs on the CPU (via its optimised CPU plugin), not the iGPU, even with /dev/dri passed through.

  3. Verify OpenVINO sees the device:

    docker exec winnow /app/.venv/bin/python3 -c \
  "from openvino.runtime import Core; print(Core().available_devices)"

    Expected: ['CPU', 'GPU']. If only ['CPU'] appears, the Level Zero / OpenCL runtime can't see the device — check group membership and device permissions.

Check startup logs for provider selection

winnow logs ONNX providers available: [...] and the selected execution provider at DEBUG level on every start. Enable verbose mode and inspect the log:

docker exec winnow cat /app/frigate_train/winnow.log | grep -i "provider\|execution\|cuda\|rocm\|openvino"

Same images uploaded every run

The upload tracker is stored in DATA_DIR (/app/data by default) as winnow_tracker.db (SQLite). If this volume isn't persisted between runs, the tracker resets and images are re-uploaded.

Make sure /app/data is mounted to a persistent host path.

Upgrading from 0.4.x: winnow automatically migrates the legacy JSON tracker files (frigate_uploaded_ids.json / frigate_rejected_ids.json) to SQLite on the first run and renames them to .json.bak. No manual action is required.

Re-uploading a specific person

To clear the upload history for one person and start fresh:

RESET_PERSON=John

Remove this after one run. It deletes winnow-managed Frigate training files for that person, wipes their upload history, then processes normally. Manually-added Frigate files are never touched.

Duplicate person warning at startup

If winnow prints a warning like:

⚠  Duplicate person names detected in Immich:
  John → 2 people: abc12345… (47 assets), def67890… (12 assets)
    Processing largest only (47 assets). Skipping 1 smaller duplicate(s)...

Immich has created separate person records for the same individual (a common side-effect of its face clustering). By default winnow processes only the largest and skips the rest.

To permanently merge them inside Immich, set MERGE_DUPLICATE_PEOPLE=true for one run. The person with the most assets absorbs the others. This modifies Immich and cannot be undone — only do this after verifying the duplicates are the same person.

Image quality issues

The quality thresholds are pre-calibrated for Frigate's ArcFace requirements. Support will not be provided for problems caused by non-default values. That said, if you understand the tradeoffs:

  • Selected images are too blurry: Raise BLUR_THRESHOLD (default 120.0) — e.g. 200 requires sharper images
  • Too many images rejected for blur: Lower BLUR_THRESHOLD — e.g. 80 accepts blurrier images
  • Too many images rejected for small face size: Lower MIN_FACE_WIDTH (default 90) — e.g. 50 accepts smaller face crops
  • Selected images have poor-quality face crops: Raise MIN_FACE_WIDTH — e.g. 150 requires larger face crops
  • Too many images rejected for low detection confidence: Lower MIN_CONFIDENCE (default 0.7) — e.g. 0.5 accepts lower-confidence detections
  • Training set includes marginal detections: Raise MIN_CONFIDENCE — e.g. 0.85 requires more confident detections
  • Rejected images being re-tried: Set RETRY_REJECTED=true for one run

Clone this wiki locally