chore: upstream sync - #7
Merged
Merged
Conversation
Realtime cost calculation computed totals but never populated logging_obj.cost_breakdown, so spend logs and the UI Metrics/Cost Breakdown showed no input/output cost details. Co-authored-by: Cursor <cursoragent@cursor.com>
…rflowing Tremor's Table forwards className to a wrapper div rather than the inner table element, so the table-fixed class never reached the table and it stayed table-layout: auto. Across 16 whitespace-nowrap columns that expanded the table far past the viewport Give each spend-logs column an explicit pixel size and drive the table width from getCenterTotalSize(), matching the Virtual Keys table. The shared DataTable applies this only when columns declare sizes, so the other consumers keep their existing fluid layout
… into litellm_/cranky-hamilton-21b5d0
git diff --name-only includes deleted paths, so a PR that removes a litellm/**/*.py file feeds the gone path to ruff format --check, which exits 123 with 'No such file or directory'. Add --diff-filter=ACMR so only added/copied/modified/renamed files are checked, matching the pattern already used in test-litellm-ui-build.yml.
… of overflowing The Request Logs page pushed the whole page past the viewport horizontally. The cause was the app shell flex layout: <main className="flex-1"> is a flex item, and flex items default to min-width: auto, so they refuse to shrink below their content's intrinsic width. The logs table is intrinsically ~2300px across its 16 nowrap columns, so main grew to that width and dragged the page with it; the table's own overflow-x-auto wrapper never got the chance to scroll Add min-w-0 to main so it can shrink to the available width, at which point the existing overflow-x-auto wrapper engages and the table scrolls inside its card. This applies to every dashboard page, not just logs Also drop the dead max-w-screen class on the logs container (not a real Tailwind utility, so it was a no-op), and revert the earlier column-sizing attempt which targeted table-layout rather than the actual containment problem
…ense ones Now that the page-overflow bug is fixed by letting the main pane shrink, bring back per-column sizing purely to control widths. Columns declare explicit pixel sizes and the table derives its min-width from getCenterTotalSize(), so it stretches to fill a wide card but scrolls once the columns no longer fit. The shared DataTable applies this only when columns declare sizes, leaving the other consumers on their existing fluid layout Trim the columns that were eating horizontal space without earning it: Request ID and Key Hash drop ~30% (Key Hash now narrower than Key Alias, which is the more useful of the two), and Duration and TTFT shrink to fit their short numeric values
… non-admins Non-admin callers could create or update a personal key (no team_id) with arbitrary access_group_ids, mcp_toolsets, vector_stores, or search_tools in object_permission. The server persisted the values without ownership validation; runtime authorization then trusted the IDs because they were stored on the key, allowing cross-tenant access to other teams' restricted models, MCP toolsets, and vector stores. The personal-key gate now mirrors the team-key path. enforce_member_can_assign_access_groups raises 403 for non-admin teamless callers. validate_key_mcp_servers_against_team rejects non-empty mcp_toolsets on personal non-admin keys. A new validate_key_vector_stores_against_team enforces the same rule for vector_stores. validate_key_search_tools_against_team gains the same gate for search_tools. The four validators are wired into /key/generate, /key/update, and /key/regenerate. Proxy admins keep their existing carve-out across all fields; team keys are unaffected. Endpoint-level regression coverage lives in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py (six new parametrised cases through generate_key_fn and _validate_update_key_data) and helper-level coverage in tests/test_litellm/proxy/management_helpers/. Deleting any of the validator calls in _common_key_generation_helper or unmoving the enforce gate in _validate_update_key_data breaks the suite.
Replace bare Optional[dict] on the object_permission validator surfaces with a typed TypedDict mirror of LiteLLM_ObjectPermissionBase. The TypedDict shape matches the Pydantic model field-for-field and supports .get() and item assignment, so the mutation in _rewrite_object_permission_mcp_identifiers continues to work at runtime (TypedDict is a plain dict). Propagated through the surfaces this PR touches: _object_permission_to_dict, _validate_mcp_servers_for_key_update, validate_key_mcp_servers_against_team, validate_key_search_tools_against_team, validate_key_vector_stores_against _team, the five _extract_requested_* helpers, and the two _rewrite_object_permission_mcp_* mutators. attach_object_permission_to_dict, handle_update_object_permission_common, and _set_object_permission keep their wider dict typing because they handle the full key/team data_json, which is a superset of ObjectPermissionDict and pre-dates this PR. No behavior change. 373 tests pass; ruff strict + type discipline gates green.
The error string is already produced by the f-string interpolation; the trailing .format() call on it was redundant. Add a regression test that the message renders the model name verbatim.
get_model_list always returns a list, never None, so the is-None branch could not execute. Collapse to the single reachable message.
…oned format and re-encryption migration (BerriAI#31215) * feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration * test(proxy): add behavior scenarios for credential migration endpoints * fix(proxy): scan covered tables in encryption check, fix CI lint and route types * fix(proxy): migrate callback_settings credentials, clear CI lint/recursion gates, add encryption endpoint+CLI tests * fix(proxy): correct dry-run/real-run migrated vs residual-legacy counters in config and SSO walkers * fix(proxy): make callback-vars residual detection gate-independent in encryption check
…del_error_cleanup chore(router): simplify unknown-model error message construction
chore(ci): re-issue 31566 for full CI run
…0) (BerriAI#31533) * fix(ui): keep virtual-keys filters across delete and refresh (LIT-4080) Filtering virtual keys by User ID and then deleting a key reset the filter to show all keys, and re-clicking Fetch did not re-apply it. The page ran two competing fetch paths: useKeys (React Query) fetched the page unfiltered while a separate useFilterLogic hook held its own filteredKeys list and, on any refresh, only re-applied Team and Organization client-side, silently dropping the User ID and Key Alias filters. Delete refreshed through the unfiltered useKeys path, so the filtered view collapsed back to everything VirtualKeysTable now owns its filter state and feeds every filter (team, organization, key alias, user id, key hash) straight into the useKeys options, so the filters are part of the React Query key. Any refetch or invalidation re-runs the same filtered query, which makes the reset-on-delete bug structurally impossible. Free-text inputs are debounced with @tanstack/react-pacer, sorting and pagination are server-side, and changing a filter or sort resets to page 1 Delete now invalidates keyKeys.lists() from key_info_view, matching the create path, instead of prop-drilling a refetch; the window "storage" refetch effect is removed. The dual-path useFilterLogic hook (and its test) are deleted Regression coverage: VirtualKeysTable threads an active User ID filter into the useKeys query and clears it on reset, useKeys encodes filter options in its query key so a filter change refetches, and key_info_view invalidates the keys list on delete * refactor(ui): simplify virtual-keys table data flow VirtualKeysTable now fetches its own teams and organizations via useOrganizations and the existing all-teams query instead of taking them as props, so the prop-drill through UserDashboard and the two page callers (page.tsx, ApiKeysDashboard) is gone along with their redundant organization state and fetch Filter state collapses from a useState plus a useDebouncedState mirror into a single source whose debounced copy is derived with useDebouncedValue, and one typed toKeyListFilters adapter maps it to the key/list query options. Behavior is unchanged; same 300ms debounce and the same reset timing The unused onSortChange/currentSort props and their sync effect are removed since no caller passed them, leaving sorting fully internal Adds a created_by_user alias-over-email regression test that fails if the display precedence is swapped * test(ui): add required last_active to useKeys mock fixtures The KeyResponse type requires last_active, so the typed mockKeys fixtures were missing it. Add it so the file type-checks cleanly. * chore(ui): ratchet lint budgets after virtual-keys refactor Deleting filter_logic.tsx and simplifying VirtualKeysTable lowered the no-explicit-any (2026 to 2016) and complexity (128 to 127) counts, so the eslint-metrics.json baseline was stale and failed the frontend-lint budget gate. Regenerate it, and drop the now-dead filter_logic.tsx suppression entry for the file this PR removed. * fix(ui): show a loading state for data-backed filter dropdowns The Team ID and Organization ID filters source their options from async hooks (teams / organizations). While that data was still loading the dropdowns rendered 'No results found', so they looked empty rather than loading. Add an opt-in loading flag to FilterOption that the searchable select surfaces as a spinner and a 'Loading...' empty state, and wire it from the teams and organizations query loading states. While loading, the filter no longer caches an empty initial-options list, so the real options appear once the data arrives.
…rriAI#31638) * perf(ui): load virtual-keys team filter from the fast v2 endpoint The virtual-keys table sourced all teams through fetchAllTeams, which hits the unpaginated /team/list. On a proxy with 125 teams that call takes ~9.5s, so the Team ID filter and the team-alias/budget columns sat empty for that whole window. The key list itself does not carry team_alias or team_max_budget, so the table genuinely needs a team lookup and cannot just drop the fetch. Add useAllTeams, which pages the fast /v2/team/list to completion (~0.6s per 100-team page, so ~1.2s for 125 vs ~9.5s), and point VirtualKeysTable at it instead of fetchAllTeams. The allTeams shape, the filter searchFn, the column lookups, and the loading indicator are all unchanged; only the source endpoint changes. fetchAllTeams stays for its other callers. * test(ui): tighten team-filter test readability and robustness Address adversarial review of the added tests. Rename the single-page useAllTeams test to match what it asserts (one request for a one-page result) rather than implying it guards the early-return, and drop the unread, misleading total: 125 from the mock page response. Scope the created_by alias-over-email assertion to the key's table row so it checks the visible cell value; the hover popover that also holds the email is portaled out of the row, so the previous document-wide negative assertion was relying on antd's lazy popover mounting. * fix(ui): scope useAllTeams cache by access token The previous /team/list query keyed on accessToken, so a user switch in the same SPA session produced a distinct cache entry. useAllTeams dropped that, so team IDs and aliases could be briefly reused across users until the staleTime expired. Put accessToken back in the query key to restore per-identity isolation, and add a regression test that a token switch triggers a refetch rather than serving the cached list.
…ws tool_calls (BerriAI#31633) * fix(databricks): split parallel tool calls so each tool message follows tool_calls Databricks OpenAI-compatible serving (e.g. GPT models) 400s with "messages with role 'tool' must be a response to a preceeding message with 'tool_calls'" when an assistant turn makes parallel tool calls. LiteLLM faithfully sends one assistant message holding all tool_calls followed by one 'tool' message per result, so every result after the first is preceded by another 'tool' message rather than the assistant tool_calls message, which Databricks rejects. Re-emit each result immediately after an assistant message that carries only its matching tool_call, turning assistant(tool_calls=[A, B]), tool(A), tool(B) into assistant(tool_calls=[A]), tool(A), assistant(tool_calls=[B]), tool(B). The rewrite is a no-op when the turn is already valid (single call), the group is incomplete, or ids don't line up, so no tool call is ever dropped. Scoped to non-Claude models, matching the existing OpenAI-shaped transformation path. * style(databricks): use builtin list generics in parallel tool-call split Switch the List[...] annotations introduced by _split_parallel_tool_calls to lowercase list[...] so the UP006 strict-rule budget stays within its ceiling.
…s (VERIA-392) (BerriAI#31469) /key/generate validated the caller's delegation ceiling against data.max_budget only. The per-window entries in data.budget_limits bypassed the check, so a non-admin caller could mint a key whose 1-day window vastly exceeded their own max_budget. The data.permissions dict also went unvalidated for non-admin callers, so they could self-grant capabilities like allow_pii_controls (and on Enterprise, get_spend_routes). Both gates now live in _common_key_generation_helper, covering /key/generate and /key/service-account/generate. The existing empty {} default on permissions still passes for non-admin callers.
…riAI#31604) increment_spend_counters was parallelized in BerriAI#31578, but the dominant per-request cost under high concurrency is the pre-call budget enforcement in common_checks, which still ran a Redis-first get_current_spend per scope (team, team windows, key windows, org, tag, user, team member, end user) one sequential await after another inside the auth span. The per-scope reads target distinct counter keys with no cross-scope ordering dependency, so they now run concurrently under asyncio.gather. Key metadata.tags injection still runs before the gather so the tag budget check sees it, and every scope settles before the first error in scope-priority order propagates, preserving the previous rejection semantics. Resolves LIT-4090
…erriAI#31630) Enforce that every `budget_limits[*].max_budget` is a finite number; applies to every caller including proxy admin and runs before the role / ceiling checks. Six parametrized regression tests cover NaN / +inf / -inf for both non-admin and admin callers.
…erriAI#31631) Mirror the scalar `max_budget` guard in `_common_key_generation_helper` for the per-window check: a CLI session token caller (carrying `max_budget=None`) cannot set `budget_limits` on a personal key. Pass `team_table` into the helper so it can detect the personal-key shape; reject before the `delegation_ceiling is None` early return. Four new regression tests cover the personal-key reject, the team-key happy path, the team-key over-team-budget path, and the proxy-admin exemption.
…reuse Defines the TypedDict mirror of LiteLLM_ObjectPermissionBase under litellm/types/object_permission.py so SDK-side modules can adopt the type without violating the SDK-must-not-import-from-proxy layering rule. litellm/proxy/_types.py re-exports it for existing callers. Supports the validator-surface retypes in the preceding proxy refactor commit.
… on large uploads (BerriAI#31653) * fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads PR BerriAI#31036 switched the vertex batch file upload from a single GCS media upload to a chunked resumable session. The resumable path sends the body as many sequential PUTs, each waiting a full round-trip to GCS before the next, so a multi-GB upload accumulates hundreds of round-trips and overruns the client/load-balancer request timeout, surfacing as 499s (client closed connection) on files as small as 500MB. This was a regression from the last-known-good commit, where the upload completed as one continuous request. Revert the batch upload to a single uploadType=media request, but stage the transformed payload to a temp file first so peak memory stays bounded (the goal of the resumable rewrite) without the per-chunk round-trips. The temp file is closed deterministically (TemporaryFile unlinks on close), not left to the GC. The now-unused resumable chunked-upload plumbing is removed. Also swap the per-row transform's stdlib json for orjson (parse + serialize), which is ~4x faster on this hot path; the streaming body now emits compact orjson bytes. The request stays synchronous, so the returned file object is real and POST /v1/batches keeps working immediately against the uploaded object. Tests: single media request carries the whole payload with a real Content-Length (no chunked transfer-encoding); failed upload raises; the staged temp file is closed deterministically; byte-for-byte transform parity. * test(vertex_ai/files): mock single media upload POST instead of removed resumable method test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type. * fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports Forward the per-request timeout through _stage_and_upload_media / _astage_and_upload_media to the GCS POST. Every other upload branch forwards it; the new media path was dropping it, so a caller-provided timeout was silently ignored (the files path passes 600s by default, but a custom request_timeout would not have reached this upload). Regression test asserts the resolved timeout reaches the request (mutation-verified). Revert the orjson swap in the batch transform: importing orjson at module load in this core-path file broke `import litellm` on environments without orjson (the Windows import test). Back to stdlib json; the upload leg dominates large uploads anyway, so the transform-side win was marginal. Fix import ordering in llm_http_handler.py (I001) introduced by the new imports. * fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file Addresses a disk-exhaustion concern: staging the full transformed batch body to a local temp file before the GCS request meant an authenticated user could fill the proxy's temp volume with large concurrent uploads (on top of Starlette's input spool). GCS's simple/media upload accepts chunked transfer-encoding, so stream the transform straight to the single media request instead. Each block is produced on a worker thread (the transform never runs on the event loop) and sent chunked, so the body is neither buffered in memory nor written to disk, and the upload is still one continuous request (no per-chunk round-trips, no 499). Drops the temp-file staging, the tempfile/IO imports, and Content-Length computation. Regression test asserts the upload streams (chunked transfer-encoding, no Content-Length) and creates no temp file; mutation-verified that reintroducing staging fails it.
…I#31227) * fix(proxy): count only active users toward license seat limit SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected * fix(proxy): floor billable user count at zero count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call Addresses Greptile P1 on the PR * refactor(proxy): count teams via TeamRepository in available_users * style: ruff format changed files at line-length 120
…n endpoint (BerriAI#31657) * fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint The OAuth token endpoint stored a user's per-server token under the identity returned by _extract_user_id_from_request, which read only the Authorization header and did getattr(cached, "user_id") on a raw user_api_key_cache lookup with no model_type rehydration and no DB fallback. That silently returned None in two common cases on a multi-replica gateway: the LiteLLM key arrives on x-litellm-api-key (what MCP clients such as Claude Desktop and Claude Code send) rather than Authorization, and a cross-replica cache hit deserializes to a plain dict rather than a UserAPIKeyAuth, so getattr finds no attribute. When it returned None the token was not persisted. This was survivable until the authorization_code v2 migration began stripping the caller's Authorization for migrated per-user OAuth servers and routing the preemptive 401 existence check through the stored token, so a persist miss now hard-fails: the egress challenges with 401 on every reconnect (the client sees "rejected them on reconnect" or a successful connect with zero tools). Resolve identity through get_key_object, the canonical resolver that reads the cache with model_type and falls back to the DB, and accept the key from x-litellm-api-key as well as Authorization. The silent persist skip is now a warning. The caller-Authorization stripping stays as is, since reinstating it would reopen the cross-user credential override it was added to prevent. * fix(mcp): reject blocked or expired keys when resolving the token-endpoint identity The OAuth token endpoint is unauthenticated, and get_key_object resolves a key row without the blocked/expiry checks the main user_api_key_auth pipeline runs (that pipeline is bypassed here). So a holder of a revoked or expired LiteLLM key could POST a valid upstream authorization code with that key in x-litellm-api-key/Authorization and write or overwrite the stored per-user OAuth token for that key's user. The cache-only resolver this replaced incidentally dropped blocked keys (blocking purges the cache entry), so moving to the authoritative cache-then-DB resolution removed that accidental shield. Validate the resolved key before trusting its identity: return None when blocked or expired, so the upsert is skipped. Deleted keys are already rejected, since get_key_object raises on a missing row. Regression tests cover the blocked and expired cases and fail without the guard.
…nal_key_metadata fix(proxy): reject team-scoped object_permission on personal keys for non-admins
Pass transcription_cost through additional_costs so cost_breakdown's input_cost + output_cost + additional_costs sums to total_cost instead of silently folding it into total_cost only. Co-authored-by: Cursor <cursoragent@cursor.com>
…tream errors Two bugs from the upstream-error fixes: the success handler has no status-code awareness, so removing raise_for_status() left it firing for every upstream 4xx/5xx too, meaning the new failure hook and the existing success handler both logged the same request (corrupting SpendLogs/cost tracking). Separately, the failure hook was passed the raw httpx.HTTPStatusError, which ProxyLogging's alerting only excludes HTTPException/ProxyException from, so a normal upstream 403 would trigger a "High" severity llm_exceptions alert. Gates the success handler (both non-streaming and end-of-stream) to status_code < 400, and reports upstream failures to post_call_failure_hook as an HTTPException instead of the raw httpx error, matching how auth/rate-limit errors are already excluded from alerting. Co-authored-by: Cursor <cursoragent@cursor.com>
… DB is down at startup (BerriAI#31951) * fix(proxy): keep serving reads from the read replica when the primary DB is down at startup RoutingPrismaWrapper.connect() connected the writer first and let a writer failure propagate, so a proxy that started during a primary outage ended up with no Prisma client at all (startup swallows the error under allow_requests_on_db_unavailable): DB-stored models never loaded and every inference request failed with 400 Invalid model name, even with a healthy DATABASE_URL_READ_REPLICA. Workers recycled via MAX_REQUESTS_BEFORE_RESTART hit this mid-outage and stayed broken for the rest of the outage. connect() now degrades on a writer-only failure: reads (key auth, DB-stored model loads) are served by the reader, writes fail at call time, and the DB health watchdog keeps retrying the writer reconnect, which clears the degraded flag once the primary recovers. A full outage (both sides down) still raises as before. Resolves LIT-4159 * fix(proxy): clear degraded-writer flag when the reconnect probe finds the writer already healthy The direct-reconnect path returns early when the writer probe succeeds (engine already reconnected by another path, e.g. an IAM token refresh), skipping recreate_prisma_client, which was the only runtime path clearing _writer_unavailable. The stale flag made the watchdog fire reconnect attempts against a healthy writer on every cooldown cycle until restart. Clear the flag in the early-return branch and cover it with a regression test that fails without the change
…292bcc build: restore maturin backend to bundle the Rust bridge in the wheel
chunk_processor now reads response.status_code to gate end-of-stream success logging. These mocks used AsyncMock(spec=httpx.Response), which spec's against the class and doesn't expose status_code since it's an instance attribute, not a class attribute, so accessing it raised AttributeError. Sets status_code=200 explicitly on the success-path mocks. Co-authored-by: Cursor <cursoragent@cursor.com>
…etrics fix(cost): store cost breakdown for /v1/realtime sessions
…or_normalisation fix(proxy): return upstream error bodies unchanged in passthrough
The CircleCI proxy containers passed --detailed_debug, and litellm's log level defaults to DEBUG when LITELLM_LOG is unset, so CI produced very verbose debug output for no reason. Drop --detailed_debug and set LITELLM_LOG=ERROR on the proxy containers so real failures still surface without the debug noise
…6c3e bump: litellm-enterprise 0.1.46 -> 0.1.47
chore(ci): build ui for release
…nd cache metrics (BerriAI#32126) * feat(prometheus): add api_provider label to token, latency, request and cache metrics The token (input/output/total), latency (llm_api, time_to_first_token, request_total, request_queue_time), proxy request (total/failed) and cache metrics were emitted from the same call sites as litellm_spend_metric and litellm_requests_metric, which already carry api_provider, yet these were missing it. That left no way to break tokens, latency, request counts or cache hits down by upstream provider even though the provider is already on the payload as custom_llm_provider. Add api_provider to each metric's label allow-list. The success path already populates enum_values.api_provider from standard_logging_payload, so those metrics emit it with no further plumbing. The cache label is added to the shared _cache_metric_labels list, so alongside litellm_cache_hits_metric and litellm_cache_misses_metric it also covers litellm_cached_tokens_metric and the provider prompt-cache read/creation token metrics; the label-presence test asserts all of them. For the client-side failure path, where a deployment may not have been resolved, derive it best-effort from litellm_params.custom_llm_provider, a partial standard_logging_object, or inference from the requested model name via litellm.get_llm_provider, falling back to empty rather than guessing. Resolves LIT-4178 * fix(prometheus): satisfy ruff BLE001 budget and update enterprise label assertions - Suppress the strict-rule BLE001 budget breach with a justified noqa; the broad except in the failure-path provider extraction is intentional defense-in-depth (covered by test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors), not dead code to delete - Update tests/enterprise assertions for litellm_tokens_metric, litellm_input_tokens_metric, litellm_output_tokens_metric, the three latency metrics, and the proxy request counters to expect the new api_provider label, matching what litellm_mapped_enterprise_tests caught in CI --------- Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
…riAI#31983) * feat(mcp): add entra_obo profile to the token_exchange (OBO) arm Microsoft Entra On-Behalf-Of uses the RFC 7523 jwt-bearer grant rather than RFC 8693, so the existing token_exchange arm cannot mint tokens against Entra. This adds an entra_obo profile on TokenExchangeConfig that switches the request form to Entra's jwt-bearer OBO dialect (the inbound token as assertion, the target resource carried in scope, and requested_token_use=on_behalf_of), while reusing the shared caching, single-flight, TTL, and fail-closed machinery. The exchanger is renamed from Rfc8693TokenExchanger to OboTokenExchanger since it now serves both dialects The profile is threaded through the config-load path, the DB credentials blob, and the MCPCredentials request schema, so an operator can select it from YAML or the management API. It rides in the existing credentials JSON blob, so there is no new auth_type and no DB migration Resolves LIT-4163 * feat(mcp): propagate the Entra Conditional Access step-up challenge on the OBO 401 An entra_obo exchange that the IdP rejects for Conditional Access returns a 4xx with error=interaction_required and a claims blob the client must satisfy to step up. The arm dropped both and emitted a static RFC 9728 challenge, so a CA-protected Entra upstream was unreachable through the gateway. The provider now reads the RFC 6749 error code and the claims string off the rejection body (error_description is still never carried; it can leak IdP internals), threads them through SubjectTokenRejected -> CredError.unauthorized, and the challenge builder folds them into WWW-Authenticate: the machine error only when it is a plain OAuth token (guards against header injection from a hostile body) and the claims base64-encoded in a claims parameter, the convention MSAL-family clients decode. With neither field the header is byte-identical to the static challenge. The multi-server aggregate still absorbs a step-up 401 to an empty listing; only single-server routes surface it * fix(mcp): use error=insufficient_claims for the Entra step-up challenge Per Microsoft's claims-challenge format, a WWW-Authenticate carrying a claims challenge must set error=insufficient_claims (the value MSAL-family clients key on to recognize the challenge and replay the claims), not the raw token-endpoint code. The challenge now sets insufficient_claims whenever a claims blob is present and keeps invalid_token otherwise, with a step-up-accurate error_description in the claims case. The presence of claims now drives the error value, so the raw oauth_error no longer needs threading from the provider through CredError to the edge; that plumbing is removed (the provider still reads the error code for its gateway-fault classification). Both the error value and the base64 claims are fixed-alphabet, so nothing from the IdP body reaches the header unescaped. Cross-checked field-by-field against Microsoft Learn; a bogus jwt-bearer OBO POST to the real login.microsoftonline.com/common endpoint confirmed Entra recognizes the grant and returns the error shape the parser reads. Follow-up: also emit authorization_uri alongside resource_metadata for strict non-MCP MSAL clients (RFC 9728 resource_metadata already serves MCP) * fix(mcp): filter blank scopes on the config-load path so entra_obo fails closed The DB-build path already runs YAML/DB scopes through _extract_scopes (which drops blanks), but the config-load path read server_config["scopes"] raw. A YAML `scopes: [""]` therefore reached the exchanger as a ("",) tuple: non-empty, so the entra_obo `not config.scopes` precondition skipped its fail-closed misconfigured path and the form builder POSTed an empty scope to the IdP instead of failing before any network call. Config-load now filters blanks the same way, so an all-blank list normalizes to None and the precondition fails closed. Regression test loads a blank-scope entra_obo server and asserts the exchange returns misconfigured without POSTing
…ore comments (BerriAI#32152) * fix: zero out crash-class basedpyright rules across litellm/ * feat(lint): add LIT009 banning inert type: ignore comments * docs: require bracketed rule and reason on every suppression * chore(lint): ratchet budgets down and zero crash-class pyright limits * fix: narrow auto router routelayer through a local before calling * test: add regression tests for crash-class fixes * fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
…"allow" (BerriAI#32158) * fix(headroom guardrail): log real token/compression stats instead of "allow" The headroom guardrail fetched tokens_before/tokens_after/compression_ratio from Headroom's /v1/compress response but only surfaced them via a debug-level log line, so spend_logs.guardrail_information showed guardrail_response: "allow" with no way to tell whether compression actually ran or by how much. _call_compress now returns the token/compression stats alongside the compressed messages and success flag, and apply_guardrail logs them via add_standard_logging_guardrail_information_to_request_data when compression succeeds. Raw message content is intentionally excluded from what's logged - only token counts, compression ratio, and applied transform names. * fix(ci): apply ruff format to headroom.py * fix(review): remove comment per repo's no-comments-unless-asked convention Addresses codex review feedback - CLAUDE.md says not to add comments unless explicitly asked; the sensitive-logging guarantee is already expressed by the stats dict only pulling specific keys, not messages.
…ng /v1/messages responses (BerriAI#32160) * fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses * fix(anthropic_messages): forward aclose to inner streaming iterator * fix(anthropic_messages): forward aclose through the streaming response wrapper The proxy's streaming cleanup closes the handler's return value via hasattr(response, "aclose"); the new wrapper hid the upstream generator's aclose, so provider connections could linger on client disconnect. The wrapper now delegates aclose to the wrapped stream and AgenticAnthropicStreamingIterator closes its inner and follow-up streams. Also adds test coverage for the agentic streaming branch --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ithout message_stop (BerriAI#32159) * fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop * fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors The bytes branch of _is_message_stop_chunk used a plain substring match, so a content_block_delta whose partial_json contained the literal text message_stop would look like a real terminal event and suppress the synthetic incomplete-stream error. Match the SSE event header line instead. Also treat a provider-emitted error event as terminal so a stream that ends with an upstream error is not followed by a second, contradictory synthetic incomplete-stream error. * test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ests (BerriAI#32016) * test(e2e): migrate access-control and inference-endpoint regression tests Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config * Update endpoints_client.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…es, batches, prometheus, and langfuse eviction (BerriAI#32165) * fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects spend_tracking/spend_e2e_client.py imported SpendTagsResponse and TagSpend from models, but neither was ever defined, so importing the client raised ImportError and pytest aborted collection for the whole e2e session. The tag-spend tests had never run. Model /spend/tags as it actually answers: a bare array of per-tag aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field that also wouldn't match the array shape; it now reads .root, matching how spend_logs consumes its RootModel. * test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction Adds regression nets and gap-surfacing tests: A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document that reasoning_effort='none' and thinking type='disabled' are silently dropped (LIT-3686 / GH BerriAI#27453) A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py): parametrized regression net asserting real completion content, not just a 200, across the configured providers for /chat/completions and /responses (GH BerriAI#28991) A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is honored and prompt-cache read tokens grow on a repeated cacheable prefix A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call rate limiter runs, then asserts no unattributed spend row is left behind by the internal input-file retrieval (LIT-3266) A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct key_alias and asserts each alias gets its own labeled series on /metrics A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict) regression proving eviction must not close an httpx client still held by an in-flight caller (LIT-3221 / GH BerriAI#13034) Extends tests/e2e/models.py with the typed request and response fields these tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage fields, spend-log api_key) Co-authored-by: Cursor <cursoragent@cursor.com> * test(e2e): drop unused litellm-regression-tests submodule The e2e suite migrated the regression cases into this repo; nothing imports the submodule at runtime (only a provenance comment references it), so the .gitmodules entry and gitlink pointing at a personal repo would just make upstream CI init a submodule it never uses. Remove both to keep the change test-only. * test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client close-on-eviction bug (LIT-3221 / GH BerriAI#13034). That is a non-trivial fix (thread cleanup vs shared client teardown) and belongs in its own PR, not this e2e coverage PR, so revert the file to its base state. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* chore: add latest model rule to CLAUDE.md * chore: correct grammar mistake * chore: make the rule more concise * chore: replace rule instead * chore: revise wording to override memories, etc. * chore: slightly adjust wording to be more precise
The test posted /config/update, slept a fixed 20s, then fired a single chat request with no readiness check or retry. When the single-process proxy was momentarily not accepting connections in that window, the request failed with a bare openai.APIConnectionError and took the whole job down, since the suite runs against one shared container with pytest -x Gate the chat request behind a /health/liveliness poll, retry it on connection errors only so real HTTP errors and the Langfuse assertion still fail the test, close the previously leaked aiohttp session, and target 127.0.0.1 instead of the 0.0.0.0 bind address. In CI, give the proxy container --restart on-failure so an intermittent crash recovers instead of leaving the port dead for the rest of the run
…itellm_/suspicious-jennings-5b6ef7
* ci: gate CircleCI jobs on changed paths Every CircleCI job used to run on every PR. Now each job starts with a lightweight `skip_if_unrelated_changes` step that inspects the PR diff and halts the job as successful when nothing relevant changed. Docs-only PRs (*.md, *.mdx, docs/) run nothing, UI-only PRs (ui/) run just the frontend jobs, and any backend change still runs both the backend and frontend jobs. The decision logic lives in .circleci/scripts/classify_changes.sh (pure, reads the changed-file list on stdin) so it can be unit tested, while path_filter.sh handles the git plumbing and fails open (runs the job) on any uncertainty such as a missing merge base or a non-PR pipeline. Halting via `circleci-agent step halt` keeps the job green, so required status checks are never left pending. The Windows smoke job is intentionally left ungated to avoid cross-platform shell fragility * fix(ci): keep path filter fail-open when classifier errors Guard the classify_changes.sh invocation with `|| run_full` so a broken or non-zero classifier runs the job instead of falling through to a silent halt, and mark the advisory logging pipe best-effort with `|| true`. Add path_filter.sh regression tests covering the docs-only halt, backend run, non-PR fail-open, and classifier-failure fail-open paths
…_model_registration fix(e2e): register batch + rust OCR deployments via /model/new
* fix: pass websearch tool params * fix: load db websearch tool params * fix: merge search tools in proxy * fix: satisfy websearch lint budget * fix: enforce websearch tool auth * fix: preserve search tools on empty sync * chore: rerun circleci
…nings-5b6ef7 test: de-flake langfuse callbacks-in-db e2e test
…31952) * feat(router): add separate ITPM/OTPM deployment rate limits Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): keep ITPM/OTPM diff minimal in router.py Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): make ITPM/OTPM limits separate and atomic Address Greptile review on separate ITPM/OTPM deployment rate limits. - OTPM is now reserved atomically pre-call with rollback, matching the ITPM path, so concurrent requests can no longer overshoot the configured output limit before reconciliation - ITPM counts input tokens only; it no longer accumulates completion tokens, so the input-token limit and x-ratelimit-limit-input-tokens header describe input usage as their names imply - _read_reservation_from_kwargs only falls back to litellm_params.metadata when the top-level metadata channel is absent, so production requests carrying a litellm_params.metadata dict still reconcile and refund their reservation Adds regression tests for OTPM atomicity under concurrency, input-only ITPM enforcement, and reservation lookup when litellm_params.metadata is present. * fix(router): subtract input tokens only from remaining-input-tokens header The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total tokens (input + output) instead of input tokens only, so clients saw remaining input quota understated by the completion token count on every response. Now consistent with the input-only ITPM counter. * fix(router): make itpm/otpm vs tpm/rpm precedence explicit When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path takes over and the tpm/rpm limits are not enforced. Log a warning the first time such a conflicting deployment is seen so the supersession is not silent, and document the mutual exclusivity. Post-call reconciliation now only trues up a counter that was actually reserved against, so the itpm/otpm keys are no longer incremented for deployments that never configured that limit. * fix(router): track actual io-token usage on the reservation-minute key Post-call reconciliation now keys off the exact cache key stashed at pre-call time rather than one recomputed from the response-time minute. This fixes two issues: a request whose pre-call estimate was 0 now still writes its actual billable input to the ITPM counter (previously it was skipped, leaving the limit unenforceable for that request), and a call that finishes in a later minute reconciles against the minute it reserved against instead of pushing a negative delta into the next minute. Counters are only touched when their limit is configured. * fix(router): run io-token reconciliation before the model_id guard async_log_success_event gated IO reconciliation behind the model_id guard that only the TPM tracking path needs. Since reconciliation works entirely from the cache keys stashed in kwargs, a success event whose standard_logging_object lacks model_id would skip reconciliation and leave the reservation on the counter until the TTL expired, wasting quota. Route the IO path first. * fix(router): don't replay in-flight delta for itpm/otpm headers For ITPM/OTPM model groups the counter is incremented at reservation time (pre-call), so the remaining values returned by get_remaining_model_group_usage already account for the current request. Replaying the in-flight delta on top double-counted it and understated x-ratelimit-remaining-input/output-tokens by up to max_tokens on every response. Skip the delta for io-token groups; the legacy TPM/RPM replay path is unchanged. * fix(router): clear io-token reservation after reconcile/refund async_io_token_refund_failure and async_io_token_reconcile_success now clear the stashed reservation keys from the request metadata once done. Otherwise, on a model group mixing IO-limited and non-IO deployments, a failed IO call that retries on a non-IO fallback left the stale sentinel in the shared request metadata; the fallback's success handler would divert into IO reconciliation against the already-refunded key, driving the ITPM counter negative and skipping the non-IO deployment's TPM tracking. * fix(router): tidy reservation channel lookup and header guard Consolidate the reservation channel lookup into a single ordered helper shared by read and clear, so top-level metadata always wins over litellm_params metadata without the tangled per-iteration fallback. Also stop gating the router rate-limit header block on the presence of x-ratelimit-remaining-input/output-tokens. That block only emits those headers for ITPM/OTPM groups; for a non-IO group backed by a provider that natively returns input/output token headers, the extra conditions suppressed the router's own remaining-tokens/requests headers. * fix(router): strip client-supplied io-token reservation keys The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key, and the otpm equivalents) are server-only, but metadata is caller-controlled on proxy requests. An authenticated caller could forge these fields with an arbitrary cache key so the post-call reconcile/refund path would decrement any deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs, which runs before the router stashes its own reservation, so only a genuine server-side reservation is ever read post-call. * fix(router): track TPM routing load for io-limited deployments deployment_callback_on_success early-returned for any deployment with itpm/otpm set, so its total-token usage never landed in the router's TPM routing counter. TPM-aware routing strategies then saw 0 load for IO deployments and over-routed to them in mixed model groups. Only skip tracking when neither tpm/rpm nor itpm/otpm are configured; itpm/otpm enforcement still runs separately in ModelRateLimitingCheck, so the routing counter and the enforcement counters stay independent. * fix(router): expose standard tpm/rpm headers for io-limited groups get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests; clients and prometheus gauges reading those saw no data. Build both header sets instead of returning early. Also simplify the in-flight header replay: only the tpm/rpm counters are incremented post-response, so the delta now adjusts just those. The itpm/otpm counters are incremented at reservation time (pre-call), so the input/output token headers already reflect the request and are left untouched - which removes the need for the separate io-group special case. * fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance Two follow-ups from review. The pre-call OTPM reservation only rolled back the ITPM reservation on a RateLimitError, so a transient cache error while reserving OTPM left the ITPM counter inflated until the TTL expired; catch any exception, release the ITPM reservation, then re-raise. Replace the module-level lru_cache warn-once (caching a logging side effect, which never re-warns in a long-lived process) with an instance-scoped set of already-warned deployment ids on ModelRateLimitingCheck. * fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup Clear the reservation in a finally block so a mid-reconciliation cache error still removes the stash and a duplicate success event can't re-process it. Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a deployment with no id no longer collapses every id-less deployment onto the str(None) key (which would suppress all but the first warning). * fix(router): skip io reservation when deployment can't be keyed _get_cache_keys returned a shared 'global_router:None:None:...' key when a deployment was missing model_info.id or litellm_params.model, so misconfigured deployments could share one rate-limit bucket. Return None in that case and skip io reservation for the request. * fix(router): honor explicit max_tokens=0 in io reservation _resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit max_tokens=0 fell through to the model default. Only fall back to max_completion_tokens when max_tokens is absent. * fix(ci): satisfy lint budget, router coverage, and dashboard schema sync - Modernize the new itpm/otpm module's type hints to PEP 585 lowercase generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006 violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match. - Replace three try/except Exception blocks that must stay broad by design (token_counter and litellm.get_model_info raise untyped exceptions, and an io-token refund failure must never break the logging pipeline) with contextlib.suppress(Exception), matching the codebase's existing resolution for this exact BLE001 pattern. - Add direct unit tests for get_model_group_io_token_usage (multi-deployment aggregation and the empty-model-list case) in test_router_helper_utils.py, satisfying the router function-coverage check. - Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types. * fix: enforce io token rate limits consistently * fix: honor zero max tokens in otpm reservation * fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10 floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span alias. The previously committed ruff-strict-budget.json ratcheted UP006 down from a stale base; litellm_internal_staging has since tightened that same ceiling further on its own. Reset the file to the current base's committed values and re-ratchet from there so the budget only ever moves down relative to the actual merge-base, never against a stale snapshot. * fix(router): attach ITPM/OTPM headers on dict responses and harden reservation Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit headers through /v1/messages dict responses via _hidden_params. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so set_response_headers can attach rate-limit headers to streaming Anthropic messages responses that lack a _hidden_params slot. Co-authored-by: Cursor <cursoragent@cursor.com> * style: ruff format add_retry_fallback_headers.py Fix CI ruff format check failure on get_hidden_params_dict call site. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): extract set_response_headers helpers to fix C901 budget Move header-attachment logic into add_retry_fallback_headers helpers so set_response_headers stays under the strict complexity ceiling. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: keep IO token reservation when response usage is missing Missing usage was reconciled as zero and fully refunded the pre-call reservation, allowing limit bypass on repeated successful calls. Only adjust counters when usage is resolved from the response or standard logging fields; otherwise keep the reservation until TTL expires. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: enforce RPM/TPM alongside IO-token limits on mixed deployments Deployments with both itpm/otpm and tpm/rpm previously returned after the IO reservation and skipped RPM/TPM checks. Run both paths and refund the IO reservation only when RPM/TPM rejects after a successful reservation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: track TPM usage on success for mixed IO+TPM deployments The early return after IO-token reconciliation in log_success_event and async_log_success_event skipped the TPM counter increment, so the tpm_key the pre-call check reads was never written and tpm_limit was never actually enforced on deployments that also configure itpm/otpm. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: treat total-only usage as unresolved in IO-token reconcile usage/standard_logging_object entries carrying only total_tokens (no prompt/completion or input/output breakdown) were treated as resolved usage, resolving to (0, 0) and refunding the full reservation. Both _usage_is_present and the standard_logging_object fallback now require an actual input/output breakdown before reconciling, keeping the reservation otherwise. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: reserve minimal token when input/output estimation fails _reservation_value(0, limit) reserved the entire limit whenever token estimation failed (empty/unsupported input, tokenizer error), letting one such request claim the whole bucket and 429 every concurrent request to the deployment until it completed. Reserve 1 token instead so estimation failures no longer serialize traffic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: refund IO reservation synchronously before retry deployment pick On retry, set_io_token_rate_limit_request_kwargs clears reservation sentinels from the shared kwargs dict before a background failure handler can refund them, stranding the counter until TTL. Refund and clear any stale reservation in _update_kwargs_with_deployment before stripping sentinels for the next attempt. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling Pass the deployment litellm_params.model to token_counter so it uses the model's native tokenizer instead of the generic fallback, narrowing the reservation over/under-estimate window between pre-call and post-call reconcile. Add a ponytail: comment to refund_stale_reservation_before_retry explaining the known ceiling: the synchronous DualCache.increment_cache issues a blocking Redis INCR when a Redis backend is configured. This only fires on streaming mid-stream retries (non-streaming failures await their failure handler before the retry picks a new deployment, leaving no sentinels to refund). Upgrade path: make _update_kwargs_with_deployment async. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream sync