Option to set own context in launch menu?

[PiotrZadka]

Hey, thanks for the awesome work!
I was wondering, because I understand the idea of this is to run headless for maximum vram, however would it be possible to add option in launch menu to reduce context for each option, so that we can run it while in desktop mode (ex. plasma in cachy os) if we plan to use the server on same machine and not connect to it?

[noonghunna]

@PiotrZadka — shipped just now in PR #79 (commit da0c7e7 on master). After git pull, two new env knobs flow through bash scripts/switch.sh ... cleanly without editing any compose:

# Reserve VRAM for desktop + browser + other GPU consumers:
GPU_MEMORY_UTILIZATION=0.85 bash scripts/switch.sh vllm/long-text

# Or trim context to free more headroom:
MAX_MODEL_LEN=90000 bash scripts/switch.sh vllm/long-text

# Or both:
MAX_MODEL_LEN=90000 GPU_MEMORY_UTILIZATION=0.85 \
  bash scripts/switch.sh vllm/long-text

Existing defaults preserved for anyone not setting the env — bit-for-bit identical CLI args reach vLLM if you don't override. Zero behavior change for the headless-server audience.

Coverage: 18 vLLM composes (everything in models/*/vllm/compose/). llama-cpp composes have a different flag shape and aren't touched in this PR.

Sub-24 GB tuning notes (from validating the change)

A few things that surfaced when I tested the override mechanism:

• GPU_MEMORY_UTILIZATION=0.80 is generally too aggressive on TQ3 KV paths (long-text.yml, dual-turbo.yml, etc.) — vLLM's profiling phase consumes more than the 0.05 saved budget and engine init reports No available memory for the cache blocks. Stay 0.85-0.92 for TQ3.
• fp8 KV paths (tools-text.yml, dual.yml, long-vision.yml) tolerate 0.80 better.
• Prefer dropping MAX_MODEL_LEN first over GPU_MEMORY_UTILIZATION — clean KV-budget reduction with predictable behavior, vs profiling-overhead interactions.

docs/SINGLE_CARD.md "Running alongside a desktop / sub-24 GB usable VRAM" has the full guidance + example values.

If you want to share which compose you're targeting + a nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader snapshot from your desktop session, happy to suggest concrete MAX_MODEL_LEN / GPU_MEMORY_UTILIZATION values that would fit your setup. And if anything doesn't behave the way the docs claim, that's a bug — drop a comment and we'll iterate.

[PiotrZadka]

Thank you!