Repository navigation
v2.0.12 - Adaptive Rate Limit Redesign & Thinking Fallbacks
v2.0.12 - Date: 2026-07-18
Which Build for me?
Nimbus Lite - You Dont care about the web_search mcp
Nimbus - You care about websearch mcp
Added
Rate Limiting (Adaptive Cap on 429)
- Local adaptive RPM cap - On 429, caps effective RPM to the current sliding-window rate (actual throughput). After
NIM_RPM_RESET(defaultNIM_RPM_RESET=5) successful requests, restores +1 RPM incrementally until reaching initial RPM. - Progressive decrement at cap - If 429 occurs again while capped, decrements RPM by 1 (floor = 1). Recovery follows same gradual restore.
- Server-header-driven proactive limit -
x-ratelimit-limitupdates current RPM when not locally capped;x-ratelimit-resetupdates rate window. - Reactive block from
retry-after- 429 responses trigger blocking viaretry-afterheader;<nimrpm:reset>clears block and cap immediately. - Auto-restore on success streak - Success counter resets on 429; after N successes without 429, recovers 1 RPM (when capped) or clears cap (when recovered).
Thinking / Effort Fallback for Nemotron Models
- Effort-flag fallback - Models that accept
enable_thinkingbut reject*_effortflags (low/medium/high_effort) now auto-retry with effort flags stripped, keepingenable_thinking=true. Model added to runtime cache on first error. - Reasoning_budget handling -
reasoning_budgeterrors detected and handled via existing thinking-param fallback path. - Log buffering for streaming -
THINKING/REPLYlog lines buffered (120 chars) to reduce log spam under high throughput; flushed on finish_reason or stream end.
Inline Commands: Session ID for Effort Tracking
- Per-session effort via
x-claude-code-session-id-<nimeffort:level>and<nimeffort>now key off thex-claude-code-session-idheader (falls back to API key). Fixes effort persistence across API key rotations.
Changed
Rate Limiter Redesign (providers/rate_limit.py)
- Removed legacy adaptive params -
NIM_RPM_INITIAL,NIM_RPM_DROP,NIM_RPM_MIN,NIM_RPM_HOLD_INITIAL,NIM_RPM_HOLD_MAXdeleted from code, config,.env.example, wizard. NIM_RPM_RESETsemantics changed - Was: auto-restore after N seconds without 429 (default 300s). Now: auto-restore after N successful requests without 429 (defaultNIM_RPM_RESET=5).- Single RPM tracker -
_current_rpmreplaces_effective_rpm+_initial_rpm+_hold_delay+_drop_count. Source: server header or fallback. - New state -
_capped_rpm(int | None) tracks local adaptive cap;_success_counttracks consecutive non-429 requests. - API changes -
on_rate_limit_hit()→on_rate_limited(),reset_adaptive_backoff()→reset_reactive_block(), addedon_success(),current_rpm()property. get_status()output changed - Now returnscurrent,max,initial_max,remaining,reset_in_seconds,is_blocked,blocked_seconds_remaining,worker_limit,worker_available,capped_rpm,initial_rpm. Removedeffective_rpm,hold_delay,drop_count.
Environment & Config
- New
HOSTvariable -HOST=0.0.0.0(bind address) added to.env.example. NIM_ENABLE_THINKING=true- Global thinking toggle added to.env.example(read directly by provider, not via Pydantic settings).FABLE_OVERRIDE=""- Uncommented in.env.example(was commented).NIM_RPM_RESET=5- Default changed from 300 (seconds) to 5 (requests).- Direct env vars section - Added
TIKTOKEN_CACHE_DIR,CLAUDE_CONFIG_DIR,XDG_CONFIG_HOME,UV_PROJECT_ENVIRONMENT,WEB_SEARCH_DEBUG=false,PLAYWRIGHT_BROWSERS_PATHto.env.example(read directly by components, not via settings). - Optimization flags use
Field(validation_alias=...)-fast_prefix_detection,enable_network_probe_mock,enable_title_generation_skip,enable_suggestion_mode_skip,enable_filepath_extraction_mock,enable_recap_skipnow explicitly declare env aliases.
Model Override & Request Building
model_overridepassed tobuild_request_body()- Ensures thinking params match the swapped model, not the original Claude tier model.- Session ID from request header -
_handle_nimeffortand_handle_nimeffort_statususerequest_data.session_id(fromx-claude-code-session-id) instead of API key.
Documentation Updates (README.md)
- Adaptive rate limiting section rewritten - Now describes server-header-driven proactive limit + reactive retry-after block + auto-restore on success streak.
- Env var table updated - Removed legacy
NIM_RPM_*vars; addedNIM_RPM_RESET,PROVIDER_RATE_LIMIT,PROVIDER_RATE_WINDOW. <nimrpm:reset>description updated - "Clear reactive rate limit block immediately (clears retry-after block)".
Fixed
- Effort tracking tied to API key - Fixed by using
x-claude-code-session-idheader for per-session effort storage (commita485eee). - Model override ignored thinking params -
model_overridenow passed tobuild_request_body()so swapped models get correct thinking config (commitdf7c5cc). - Rate limiter used hardcoded defaults - Fixed by removing legacy params; now reads only
PROVIDER_RATE_LIMIT,PROVIDER_RATE_WINDOW,NIM_RPM_RESETfrom settings (commit15ed3ed). - Stale rate limit on header updates -
x-ratelimit-limitnow updates_current_rpmonly when not locally capped;x-ratelimit-resetupdates window dynamically (commit236d67f).
Removed
- Legacy adaptive rate limit env vars -
NIM_RPM_INITIAL,NIM_RPM_DROP,NIM_RPM_MIN,NIM_RPM_HOLD_INITIAL,NIM_RPM_HOLD_MAXremoved from codebase,.env.example, settings, wizard, README. - ProviderConfig legacy fields -
rpm_drop,rpm_min,hold_initial,hold_maxremoved fromproviders/base.py. - Settings legacy fields -
nim_rpm_initial,nim_rpm_drop,nim_rpm_min,nim_rpm_hold_initial,nim_rpm_hold_maxremoved fromconfig/settings.py. - Test-compat
_rate_limitproperty - Kept temporarily inrate_limit.pyfor test compatibility (deprecated).
Files Touched
Core Rate Limiting
providers/rate_limit.py- Complete rewrite: adaptive cap, success-streak restore, header-driven limits, reactive retry-after blockproviders/base.py- Removed legacyProviderConfigfields;rpm_resetnowint(requests)config/settings.py- Removed legacynim_rpm_*fields; addedHOST,NIM_ENABLE_THINKING; optimization flags useField(validation_alias=...)api/app.py,api/dependencies.py- Removed legacy params fromGlobalRateLimiter.get_instance()callssetup_wizard.py- Removed legacy RPM prompts; addedNIM_RPM_RESET(requests) prompt.env.example- AddedHOST,NIM_ENABLE_THINKING,FABLE_OVERRIDE,NIM_RPM_RESET; removed legacy vars; added direct env vars sectionREADME.md- Rewrote adaptive rate limiting section; updated env var table; updated<nimrpm:reset>description
Provider & Request Building
providers/provider.py- Effort-flag fallback cache/logic;_rebuild_without_effort_flags(); log buffering (120 chars);on_success()/on_rate_limited()calls in stream/buffered pathsproviders/request.py-model_overridepassed tobuild_request_body()providers/thinking_effort_unsupported_cache(module-level cache added in provider.py)
Inline Commands / Sessions
api/routes.py-_handle_nimeffort,_handle_nimeffort_statususerequest_data.session_id(headerx-claude-code-session-id) with API key fallback; removed blank linesapi/models/anthropic.py-session_idfield on request model (already existed, now used)
Config & Documentation
reasoning_config.json- Addedreasoning_budgetentries (diff shows +12 lines)