Skip to content

0.1.0-alpha.13 — CUDA OOM falls back to CPU instead of crashing

Choose a tag to compare

@star-ga star-ga released this 17 May 00:25
· 193 commits to main since this release

Targeted hotfix on top of 0.1.0a12. Surfaced by a live install on a machine with another GPU-resident model.

Fixed

  • route() no longer crashes on CUDA OOM. sentence-transformers' default device pick is CUDA when a GPU is visible. If the GPU is already full (e.g. a local LLM in Ollama owns the VRAM), SentenceTransformer raised torch.AcceleratorError: CUDA error: out of memory and the whole call failed before the first prompt could even be encoded.
  • _Runtime.__init__ now catches GPU-init failures broadly (matches out of memory, cuda, cudaerror, no cuda in the exception message), prints a one-line stderr notice, and retries the model load with device='cpu'.

Added

  • MIND_NERVE_DEVICE env var — set to cpu to force CPU unconditionally even when a GPU is visible. Useful for hosts sharing the GPU with other tenants.

Behavior on this machine after the fix

$ mind-nerve route 'deploy the staging build to production' --top-k 5
mind-nerve: GPU init failed (AcceleratorError), falling back to CPU
{ "query": ..., "routes": [
    { "name": "deploy",                 "score": 0.7157 },
    { "name": "Deploy to Render",       "score": 0.6703 },
    { "name": "build-and-deployment-rules", "score": 0.6569 },
    ...
] }

PyPI: https://pypi.org/project/mind-nerve/0.1.0a13/