Skip to content

v2.0.8 - Request Queue, Worker Slots, Discord Web Search, Adaptive Rate Limiting

Choose a tag to compare

@cyberofficial cyberofficial released this 27 Jun 05:01
· 99 commits to master since this release

Major Features

Built-in Request Queue (Prevents "Worker limit reached" Errors)

NIMbus now includes a production-ready request queue to prevent NVIDIA NIM's hard limit of 32 concurrent requests per worker:

  • Priority lanes: HIGH (Discord bot), NORMAL (API proxy), LOW (background)
  • Configurable limits: Max concurrent (<=32), max queue size (default 600), timeout (default 300s), worker threads (default 4)
  • Two-phase timeout: Wait for worker pickup, then wait indefinitely for processing to complete
  • Observability: GET /queue/status endpoint, worker utilization in /status, DEBUG-level queue depth logs

NVIDIA Worker Slot Accounting

  • Tracks per-worker concurrent requests (32 slots per NVIDIA worker)
  • Buffered and streaming requests acquire worker slots for their full lifecycle
  • Worker limit/available exposed via /status endpoint
  • Rate limiter adapts based on worker availability

Discord Web Search (Enabled by Default)

  • Bot can search web via DuckDuckGo and fetch pages for up-to-date answers
  • Multi-page pagination with deduping
  • Configurable: DISCORD_ENABLE_WEB_SEARCH, MAX_RESULTS, MAX_ITERATIONS
  • Results with disclaimer: "This response used online resources, please verify"

Adaptive Rate Limiting

  • Runtime NVIDIA RPM backoff: NIM_RPM_INITIAL (default 40), NIM_RPM_DROP (default 10), NIM_RPM_MIN (default 20), NIM_RPM_HOLD_INITIAL/_MAX (default 5/10s)
  • Reset with <nimrpm:reset> chat tag
  • Backs off automatically on 429 responses

HTTP Client Timeouts

  • Configurable HTTP_READ_TIMEOUT, HTTP_WRITE_TIMEOUT, HTTP_CONNECT_TIMEOUT

NIM Core Settings

  • NIM_MAX_TOKENS (default 202000) - max output tokens
  • NIM_REASONING_EFFORT (low/medium/high, default high)
  • NIM_REASONING_EFFORT_MAPPINGS - JSON mapping for custom effort levels

Model Swapper

  • Dynamic model switching via SWAPPER_ENABLED chat tag <modelswap:model-name>
  • SWAPPER_TEST_PROMPT and SWAPPER_TEST_TIMEOUT for validation
  • Short names resolved against NVIDIA's live catalog

MCP Cache TTL

  • MCP_CACHE_TTL for MCP web page fetch caching (default 600s, max 3600, 0 = disabled)

Fast Prefix Detection

  • FAST_PREFIX_DETECTION optimization (enabled by default)

Complete Setup Wizard Coverage

  • New sections: Provider Rate Limiting & Adaptive, HTTP Timeouts, NIM Core Settings, Model Swapper, MCP Cache, Discord Web Search
  • Update mode includes all sections

Fixes & Improvements

Retry & Error Handling

  • Detect NVIDIA "ResourceExhausted" (rate limit) and retry with exponential backoff
  • Handle model capability errors: auto-retry with system-to-user, thinking-to-non-thinking conversion
  • Fixed exception tuple bug for JSONDecodeError/OSError
  • Improved SSE error events on terminal failures
  • Discord message processing: up to 3 retries with exponential backoff (1s base)

Separate Retry Count for Resource Exhaustion

  • New RESOURCE_EXHAUSTED_RETRIES (default 10) for NVIDIA worker limit / ResourceExhausted errors (0 = endless)

Discord Bot Enhancements

  • Native replies: message.reply() for native Discord threading
  • Reply awareness: captures replied-to content; responds to replies even with require_mention=true
  • Complete history: user messages saved unconditionally

Configuration & Defaults

  • Default model: Nemotron 3 Super 120B (nvidia/nemotron-3-super-120b-a12b)
  • Setup wizard configures: queue, Discord web search, replies, adaptive rate limiting, HTTP timeouts, NIM settings, model swapper, MCP cache
  • New per-user user_reasoning_effort_mapping.json for custom effort mappings

New Environment Variables

# Request Queue
REQUEST_QUEUE_ENABLED=true
REQUEST_QUEUE_MAX_CONCURRENT=32
REQUEST_QUEUE_MAX_SIZE=600
REQUEST_QUEUE_TIMEOUT=300.0
REQUEST_QUEUE_NUM_WORKERS=4
REQUEST_QUEUE_DISCORD_PRIORITY=2
REQUEST_QUEUE_API_PRIORITY=1

# Discord Web Search
DISCORD_ENABLE_WEB_SEARCH=true
DISCORD_WEB_SEARCH_MAX_RESULTS=10
DISCORD_WEB_SEARCH_MAX_ITERATIONS=10

# Resource Exhaustion Retries
RESOURCE_EXHAUSTED_RETRIES=10

# Adaptive Rate Limiting
NIM_RPM_INITIAL=40
NIM_RPM_DROP=10
NIM_RPM_MIN=20
NIM_RPM_HOLD_INITIAL=5.0
NIM_RPM_HOLD_MAX=10.0

# HTTP Client Timeouts
HTTP_READ_TIMEOUT=300
HTTP_WRITE_TIMEOUT=10
HTTP_CONNECT_TIMEOUT=2

# NIM Core Settings
NIM_MAX_TOKENS=202000
NIM_REASONING_EFFORT=high
NIM_REASONING_EFFORT_MAPPINGS=

# Model Swapper
SWAPPER_ENABLED=false
SWAPPER_TEST_PROMPT=Please reply with pong only, nothing else
SWAPPER_TEST_TIMEOUT=120.0

# MCP Cache
MCP_CACHE_TTL=600

# Fast Prefix Detection
FAST_PREFIX_DETECTION=true

Installation

Windows (recommended): Download nimbus.exe from this release (uploaded separately)

nimbus.exe --init    # Interactive setup
nimbus.exe           # Start proxy