Skip to content

v2.0.12 - Adaptive Rate Limit Redesign & Thinking Fallbacks

Choose a tag to compare

@cyberofficial cyberofficial released this 18 Jul 07:17
· 53 commits to master since this release

v2.0.12 - Date: 2026-07-18

Which Build for me?

Nimbus Lite - You Dont care about the web_search mcp
Nimbus - You care about websearch mcp

Added

Rate Limiting (Adaptive Cap on 429)

  • Local adaptive RPM cap - On 429, caps effective RPM to the current sliding-window rate (actual throughput). After NIM_RPM_RESET (default NIM_RPM_RESET=5) successful requests, restores +1 RPM incrementally until reaching initial RPM.
  • Progressive decrement at cap - If 429 occurs again while capped, decrements RPM by 1 (floor = 1). Recovery follows same gradual restore.
  • Server-header-driven proactive limit - x-ratelimit-limit updates current RPM when not locally capped; x-ratelimit-reset updates rate window.
  • Reactive block from retry-after - 429 responses trigger blocking via retry-after header; <nimrpm:reset> clears block and cap immediately.
  • Auto-restore on success streak - Success counter resets on 429; after N successes without 429, recovers 1 RPM (when capped) or clears cap (when recovered).

Thinking / Effort Fallback for Nemotron Models

  • Effort-flag fallback - Models that accept enable_thinking but reject *_effort flags (low/medium/high_effort) now auto-retry with effort flags stripped, keeping enable_thinking=true. Model added to runtime cache on first error.
  • Reasoning_budget handling - reasoning_budget errors detected and handled via existing thinking-param fallback path.
  • Log buffering for streaming - THINKING/REPLY log lines buffered (120 chars) to reduce log spam under high throughput; flushed on finish_reason or stream end.

Inline Commands: Session ID for Effort Tracking

  • Per-session effort via x-claude-code-session-id - <nimeffort:level> and <nimeffort> now key off the x-claude-code-session-id header (falls back to API key). Fixes effort persistence across API key rotations.

Changed

Rate Limiter Redesign (providers/rate_limit.py)

  • Removed legacy adaptive params - NIM_RPM_INITIAL, NIM_RPM_DROP, NIM_RPM_MIN, NIM_RPM_HOLD_INITIAL, NIM_RPM_HOLD_MAX deleted from code, config, .env.example, wizard.
  • NIM_RPM_RESET semantics changed - Was: auto-restore after N seconds without 429 (default 300s). Now: auto-restore after N successful requests without 429 (default NIM_RPM_RESET=5).
  • Single RPM tracker - _current_rpm replaces _effective_rpm + _initial_rpm + _hold_delay + _drop_count. Source: server header or fallback.
  • New state - _capped_rpm (int | None) tracks local adaptive cap; _success_count tracks consecutive non-429 requests.
  • API changes - on_rate_limit_hit() → on_rate_limited(), reset_adaptive_backoff() → reset_reactive_block(), added on_success(), current_rpm() property.
  • get_status() output changed - Now returns current, max, initial_max, remaining, reset_in_seconds, is_blocked, blocked_seconds_remaining, worker_limit, worker_available, capped_rpm, initial_rpm. Removed effective_rpm, hold_delay, drop_count.

Environment & Config

  • New HOST variable - HOST=0.0.0.0 (bind address) added to .env.example.
  • NIM_ENABLE_THINKING=true - Global thinking toggle added to .env.example (read directly by provider, not via Pydantic settings).
  • FABLE_OVERRIDE="" - Uncommented in .env.example (was commented).
  • NIM_RPM_RESET=5 - Default changed from 300 (seconds) to 5 (requests).
  • Direct env vars section - Added TIKTOKEN_CACHE_DIR, CLAUDE_CONFIG_DIR, XDG_CONFIG_HOME, UV_PROJECT_ENVIRONMENT, WEB_SEARCH_DEBUG=false, PLAYWRIGHT_BROWSERS_PATH to .env.example (read directly by components, not via settings).
  • Optimization flags use Field(validation_alias=...) - fast_prefix_detection, enable_network_probe_mock, enable_title_generation_skip, enable_suggestion_mode_skip, enable_filepath_extraction_mock, enable_recap_skip now explicitly declare env aliases.

Model Override & Request Building

  • model_override passed to build_request_body() - Ensures thinking params match the swapped model, not the original Claude tier model.
  • Session ID from request header - _handle_nimeffort and _handle_nimeffort_status use request_data.session_id (from x-claude-code-session-id) instead of API key.

Documentation Updates (README.md)

  • Adaptive rate limiting section rewritten - Now describes server-header-driven proactive limit + reactive retry-after block + auto-restore on success streak.
  • Env var table updated - Removed legacy NIM_RPM_* vars; added NIM_RPM_RESET, PROVIDER_RATE_LIMIT, PROVIDER_RATE_WINDOW.
  • <nimrpm:reset> description updated - "Clear reactive rate limit block immediately (clears retry-after block)".

Fixed

  • Effort tracking tied to API key - Fixed by using x-claude-code-session-id header for per-session effort storage (commit a485eee).
  • Model override ignored thinking params - model_override now passed to build_request_body() so swapped models get correct thinking config (commit df7c5cc).
  • Rate limiter used hardcoded defaults - Fixed by removing legacy params; now reads only PROVIDER_RATE_LIMIT, PROVIDER_RATE_WINDOW, NIM_RPM_RESET from settings (commit 15ed3ed).
  • Stale rate limit on header updates - x-ratelimit-limit now updates _current_rpm only when not locally capped; x-ratelimit-reset updates window dynamically (commit 236d67f).

Removed

  • Legacy adaptive rate limit env vars - NIM_RPM_INITIAL, NIM_RPM_DROP, NIM_RPM_MIN, NIM_RPM_HOLD_INITIAL, NIM_RPM_HOLD_MAX removed from codebase, .env.example, settings, wizard, README.
  • ProviderConfig legacy fields - rpm_drop, rpm_min, hold_initial, hold_max removed from providers/base.py.
  • Settings legacy fields - nim_rpm_initial, nim_rpm_drop, nim_rpm_min, nim_rpm_hold_initial, nim_rpm_hold_max removed from config/settings.py.
  • Test-compat _rate_limit property - Kept temporarily in rate_limit.py for test compatibility (deprecated).

Files Touched

Core Rate Limiting

  • providers/rate_limit.py - Complete rewrite: adaptive cap, success-streak restore, header-driven limits, reactive retry-after block
  • providers/base.py - Removed legacy ProviderConfig fields; rpm_reset now int (requests)
  • config/settings.py - Removed legacy nim_rpm_* fields; added HOST, NIM_ENABLE_THINKING; optimization flags use Field(validation_alias=...)
  • api/app.py, api/dependencies.py - Removed legacy params from GlobalRateLimiter.get_instance() calls
  • setup_wizard.py - Removed legacy RPM prompts; added NIM_RPM_RESET (requests) prompt
  • .env.example - Added HOST, NIM_ENABLE_THINKING, FABLE_OVERRIDE, NIM_RPM_RESET; removed legacy vars; added direct env vars section
  • README.md - Rewrote adaptive rate limiting section; updated env var table; updated <nimrpm:reset> description

Provider & Request Building

  • providers/provider.py - Effort-flag fallback cache/logic; _rebuild_without_effort_flags(); log buffering (120 chars); on_success()/on_rate_limited() calls in stream/buffered paths
  • providers/request.py - model_override passed to build_request_body()
  • providers/thinking_effort_unsupported_cache (module-level cache added in provider.py)

Inline Commands / Sessions

  • api/routes.py - _handle_nimeffort, _handle_nimeffort_status use request_data.session_id (header x-claude-code-session-id) with API key fallback; removed blank lines
  • api/models/anthropic.py - session_id field on request model (already existed, now used)

Config & Documentation

  • reasoning_config.json - Added reasoning_budget entries (diff shows +12 lines)