Skip to content

Troubleshooting

Jump to bottom
Holden Salomon edited this page Jun 13, 2026 · 11 revisions

Troubleshooting

Container exits immediately

Check logs:

docker compose logs winnow

Common causes:

  • Missing required env var — IMMICH_URL or API_KEY not set
  • Cannot reach Immich — check the URL and that Immich is running; use http:// not https:// unless you have TLS set up

"Immich API key is invalid or expired (401 Unauthorized)"

Your API key has been revoked or expired. Generate a new one in Immich → Account Settings → API Keys and update API_KEY in your .env.

Getting more log detail

Set VERBOSE=true in .env (or pass -e VERBOSE=true) to enable DEBUG-level output on the console. Useful for tracing exactly which images are being fetched, filtered, and selected.

The log file always captures DEBUG regardless of VERBOSE. It is written to the output volume:

docker exec winnow cat /app/frigate_train/winnow.log

"No people found" / nothing processed

  • Make sure Immich has completed face recognition and you have named people in your library
  • YEARS_FILTER defaults to 10 years — increase it if your tagged photos are older
  • MIN_FACE_COUNT skips people with few photos — lower or remove it

Frigate upload fails

  • Confirm FRIGATE_URL is reachable from inside the container: docker exec winnow curl $FRIGATE_URL/api/stats
  • Check Frigate v0.16+ — older versions don't have the face training API
  • Set DRY_RUN=true to verify selection without uploading

Models fail to download

winnow downloads InsightFace and HuggingFace (SigLIP) models on first run.

  • Ensure the container has internet access
  • Confirm the model volume is mounted and writable
  • If behind a proxy, set HTTP_PROXY / HTTPS_PROXY env vars

Running on CPU (no GPU)

Use the :cpu image tag (ghcr.io/sudolulo/winnow:cpu) which omits the GPU base entirely. Or set FORCE_CPU=true with any other tag to disable GPU at runtime.

Everything works on CPU but embedding computation is slower — typically tens of seconds per person instead of under a second on GPU.

Memory: Set a minimum container memory limit of 2 GB (mem_limit: 2g). If you have a large library — hundreds of images per person — allow more.

GPU not being used

NVIDIA

  1. Confirm the GPU is visible inside the container:

    docker exec winnow nvidia-smi

    If this fails, the container doesn't have GPU access — check the deploy.resources.reservations.devices block in compose.yml and that the NVIDIA Container Toolkit is installed on the host.

  2. Check which ONNX providers are available:

    docker exec winnow /app/.venv/bin/python3 -c \
  "import onnxruntime as ort; ort.preload_dlls(cuda=True, cudnn=True); print(ort.get_available_providers())"

    Expected (GPU working): ['CUDAExecutionProvider', 'CPUExecutionProvider', ...]
    GPU not working: only ['CPUExecutionProvider'] listed.

  3. If only CPU providers appear even though nvidia-smi works, you may be running an image older than 0.2.11 — a packaging bug caused onnxruntime-gpu to be silently overwritten. Update to fix it:

    docker compose pull && docker compose up -d

AMD (ROCm)

  1. Confirm devices are passed through — your compose.yml should have:

    devices:
  - /dev/kfd
  - /dev/dri
group_add:
  - video
  - render

  2. Verify ROCm sees the GPU:

    docker exec winnow /app/.venv/bin/python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

    ROCm exposes itself as CUDA in PyTorch — True here means ROCm is working.

Intel Arc / iGPU

  1. Confirm the DRI device is passed through:

    devices:
  - /dev/dri
group_add:
  - render

  2. Ensure OPENVINO_DEVICE=GPU is set — without it, OpenVINO defaults to CPU even with device passthrough.

  3. Verify OpenVINO sees the device:

    docker exec winnow /app/.venv/bin/python3 -c \
  "from openvino.runtime import Core; print(Core().available_devices)"

    Expected: ['CPU', 'GPU']. If only ['CPU'] appears, the Level Zero / OpenCL runtime can't see the device — check group membership and device permissions.

Set VERBOSE=true and check startup logs

winnow logs ONNX providers available: [...] and the selected execution provider at DEBUG level on every start. Enable verbose mode and inspect the log:

docker exec winnow cat /app/frigate_train/winnow.log | grep -i "provider\|execution\|cuda\|rocm\|openvino"

Same images uploaded every run

The upload tracker is stored in CACHE_DIR (/app/.if_cache by default). If this volume isn't persisted between runs, the tracker resets and images are re-uploaded.

Make sure /app/.if_cache is mounted to a persistent host path.

Re-uploading a specific person

To clear the upload history for one person and start fresh:

RESET_PERSON=John

Remove this after one run. It deletes winnow-managed Frigate training files for that person, wipes their upload history, then processes normally. Manually-added Frigate files are never touched.

Image quality issues

  • Selected images are too blurry: Raise BLUR_THRESHOLD (default 120.0) — e.g. 200 requires sharper images
  • Too many images rejected for blur: Lower BLUR_THRESHOLD — e.g. 80 accepts blurrier images
  • Too many images rejected for small face size: Lower MIN_FACE_WIDTH (default 90) — e.g. 50 accepts smaller face crops
  • Selected images have poor-quality face crops: Raise MIN_FACE_WIDTH — e.g. 150 requires larger face crops
  • Too many images rejected for low detection confidence: Lower MIN_CONFIDENCE (default 0.7) — e.g. 0.5 accepts lower-confidence detections
  • Training set includes marginal detections: Raise MIN_CONFIDENCE — e.g. 0.85 requires more confident detections
  • Rejected images being re-tried: Set RETRY_REJECTED=true for one run

Clone this wiki locally