Repository navigation
v2.0.8 - Request Queue, Worker Slots, Discord Web Search, Adaptive Rate Limiting
Major Features
Built-in Request Queue (Prevents "Worker limit reached" Errors)
NIMbus now includes a production-ready request queue to prevent NVIDIA NIM's hard limit of 32 concurrent requests per worker:
- Priority lanes: HIGH (Discord bot), NORMAL (API proxy), LOW (background)
- Configurable limits: Max concurrent (<=32), max queue size (default 600), timeout (default 300s), worker threads (default 4)
- Two-phase timeout: Wait for worker pickup, then wait indefinitely for processing to complete
- Observability:
GET /queue/statusendpoint, worker utilization in/status, DEBUG-level queue depth logs
NVIDIA Worker Slot Accounting
- Tracks per-worker concurrent requests (32 slots per NVIDIA worker)
- Buffered and streaming requests acquire worker slots for their full lifecycle
- Worker limit/available exposed via
/statusendpoint - Rate limiter adapts based on worker availability
Discord Web Search (Enabled by Default)
- Bot can search web via DuckDuckGo and fetch pages for up-to-date answers
- Multi-page pagination with deduping
- Configurable:
DISCORD_ENABLE_WEB_SEARCH,MAX_RESULTS,MAX_ITERATIONS - Results with disclaimer: "This response used online resources, please verify"
Adaptive Rate Limiting
- Runtime NVIDIA RPM backoff:
NIM_RPM_INITIAL(default 40),NIM_RPM_DROP(default 10),NIM_RPM_MIN(default 20),NIM_RPM_HOLD_INITIAL/_MAX(default 5/10s) - Reset with
<nimrpm:reset>chat tag - Backs off automatically on 429 responses
HTTP Client Timeouts
- Configurable
HTTP_READ_TIMEOUT,HTTP_WRITE_TIMEOUT,HTTP_CONNECT_TIMEOUT
NIM Core Settings
NIM_MAX_TOKENS(default 202000) - max output tokensNIM_REASONING_EFFORT(low/medium/high, default high)NIM_REASONING_EFFORT_MAPPINGS- JSON mapping for custom effort levels
Model Swapper
- Dynamic model switching via
SWAPPER_ENABLEDchat tag<modelswap:model-name> SWAPPER_TEST_PROMPTandSWAPPER_TEST_TIMEOUTfor validation- Short names resolved against NVIDIA's live catalog
MCP Cache TTL
MCP_CACHE_TTLfor MCP web page fetch caching (default 600s, max 3600, 0 = disabled)
Fast Prefix Detection
FAST_PREFIX_DETECTIONoptimization (enabled by default)
Complete Setup Wizard Coverage
- New sections: Provider Rate Limiting & Adaptive, HTTP Timeouts, NIM Core Settings, Model Swapper, MCP Cache, Discord Web Search
- Update mode includes all sections
Fixes & Improvements
Retry & Error Handling
- Detect NVIDIA "ResourceExhausted" (rate limit) and retry with exponential backoff
- Handle model capability errors: auto-retry with system-to-user, thinking-to-non-thinking conversion
- Fixed exception tuple bug for
JSONDecodeError/OSError - Improved SSE error events on terminal failures
- Discord message processing: up to 3 retries with exponential backoff (1s base)
Separate Retry Count for Resource Exhaustion
- New
RESOURCE_EXHAUSTED_RETRIES(default 10) for NVIDIA worker limit / ResourceExhausted errors (0 = endless)
Discord Bot Enhancements
- Native replies:
message.reply()for native Discord threading - Reply awareness: captures replied-to content; responds to replies even with
require_mention=true - Complete history: user messages saved unconditionally
Configuration & Defaults
- Default model: Nemotron 3 Super 120B (
nvidia/nemotron-3-super-120b-a12b) - Setup wizard configures: queue, Discord web search, replies, adaptive rate limiting, HTTP timeouts, NIM settings, model swapper, MCP cache
- New per-user
user_reasoning_effort_mapping.jsonfor custom effort mappings
New Environment Variables
# Request Queue
REQUEST_QUEUE_ENABLED=true
REQUEST_QUEUE_MAX_CONCURRENT=32
REQUEST_QUEUE_MAX_SIZE=600
REQUEST_QUEUE_TIMEOUT=300.0
REQUEST_QUEUE_NUM_WORKERS=4
REQUEST_QUEUE_DISCORD_PRIORITY=2
REQUEST_QUEUE_API_PRIORITY=1
# Discord Web Search
DISCORD_ENABLE_WEB_SEARCH=true
DISCORD_WEB_SEARCH_MAX_RESULTS=10
DISCORD_WEB_SEARCH_MAX_ITERATIONS=10
# Resource Exhaustion Retries
RESOURCE_EXHAUSTED_RETRIES=10
# Adaptive Rate Limiting
NIM_RPM_INITIAL=40
NIM_RPM_DROP=10
NIM_RPM_MIN=20
NIM_RPM_HOLD_INITIAL=5.0
NIM_RPM_HOLD_MAX=10.0
# HTTP Client Timeouts
HTTP_READ_TIMEOUT=300
HTTP_WRITE_TIMEOUT=10
HTTP_CONNECT_TIMEOUT=2
# NIM Core Settings
NIM_MAX_TOKENS=202000
NIM_REASONING_EFFORT=high
NIM_REASONING_EFFORT_MAPPINGS=
# Model Swapper
SWAPPER_ENABLED=false
SWAPPER_TEST_PROMPT=Please reply with pong only, nothing else
SWAPPER_TEST_TIMEOUT=120.0
# MCP Cache
MCP_CACHE_TTL=600
# Fast Prefix Detection
FAST_PREFIX_DETECTION=trueInstallation
Windows (recommended): Download nimbus.exe from this release (uploaded separately)
nimbus.exe --init # Interactive setup
nimbus.exe # Start proxy