BFE v1.8.9

Added

• Add ai_zero_cost_reason access-log marker (requires bfe-access-pb >= v0.3.16): why a 2xx request holding an RMB quota plan was charged 0, together with the BillingZeroCost and AbortBillingSkipped module counters (issue #1414). Requests that never became a model call (batch/files management endpoints) and AI cache hits are excluded from BillingZeroCost, since their zero cost is by design rather than an anomaly
• Support condition primitive time_weekday_in(weekday_list, zone): judge whether the weekday of the current time is in the given list (0=Sunday, 1=Monday, …, 6=Saturday, the same convention as the weekdays field of the cluster pricing tiers). zone is a single military time zone letter (see bfe_util.TimeZoneMap, e.g. H for UTC+8) and defaults to Z (UTC) when omitted — the weekday depends on the time zone, so deployments in UTC+8 should pass "H". weekday_list is a |-separated enumeration (no range notation), validated at config load time; the current time is read from the system clock, or from the X-Bfe-Debug-Time header when present
• Add mod_ai_batch, an OpenAI Batch API (/v1/files, /v1/batches) passthrough module: dedicated file/batch request modes so batch traffic stops falling into the default chat billing mode, operation classification, upload line/byte counting with large-file streaming (no retry buffering), batch-level key affinity bindings (SETNX claim-first, 404 delete plus penalty failover, create chains bound via input_file_id parsed from the buffered body before key selection), post-paid quota (reserve at create with balance pre-check, settle per-model usage at batch prices on result-file download, release on cancel/terminal status, with idempotent BATCH_RESERVE / BATCH_SETTLED Redis markers), batch rate-limit dimensions (batch_limits: create RPM, active batches, file bytes/lines; batch-only policies are now valid) and access-log batch fields (bfe-access-pb >= v0.3.13, fields 901-907) with ownership checks on file download (404 / allow_log policy). Requires the new mod_ai_batch module and its mod_ai_batch.data config
• Add mod_ai_cache, an exact-match Redis AI cache: rule matching at HandleFoundProduct, before route rule evaluation, so a cache hit short-circuits routing; skip header x-bfe-skip-ai-cache, question extraction (last/all questions), tenant-isolated keys, template-based hit responses (JSON and SSE with [DONE]), response capture with a complete flag (aborted responses are never cached), size limits and fail-open Redis access; cache hits skip RMB deduction and log ai_cache_status / ai_cache_key (bfe-access-pb >= v0.3.7, fields 789-790)
• Extend mod_ai_cache with a semantic cache phase: OpenAI-compatible embeddings client plus Chroma vector lookup (cosine space, tenant- and TTL-enforced queries), hybrid exact-first / semantic-fallback with dual write-back, per-rule enableSemanticCache (hot-reloadable, backward compatible with phase-1 rule files); a provider init failure degrades semantic caching globally with a WARN and never fails startup. Adds the hit_semantic status, ai_cache_semantic / ai_cache_similarity log fields (bfe-access-pb >= v0.3.10, fields 791-792) and SEMANTIC_HIT / EMBEDDING_ERR / VECTOR_ERR / SEMANTIC_SKIPPED counters; compatible with the Chroma 0.6.x UUID-only collections API
• Add mod_ai_intent, a semantic-routing intent module: lazy intent resolve triggered by condition evaluation (zero overhead for rules without intent conditions), System One decision-service client (Laya/Kev/Jev interchangeable via config) with timeout and circuit breaker, hot-reloadable intent_questions.data (question set with choice/score and per-question/global MinConfidence), explicit X-AI-Intent header precedence, in-process LRU+TTL cache, and the new req_ai_intent_in(String, String [, Float]) condition primitive; the consumed intent is logged in ai_intent_* access-log fields (bfe-access-pb >= v0.3.9, fields 803-809) from resolved-intent context only, never triggering a resolve. An empty question set disables intent classification
• Add mod_ai_context, an AI context compression/pruning module (phase 1, OpenAI chat format, fail-open): a lossless trim layer (tool-result truncation, latest-N inline images, thinking-block removal) plus a rule-based rewrite layer with tombstone preservation and a fidelity gate, registered at HandleAfterAITargetModel so the token budget is computed against the resolved target model and cache hits never pay for compression; adds access-log fields 793-796 (bfe-access-pb >= v0.3.11)
• Add mod_traffic_mirror, a traffic mirroring module for AI gateway shadow traffic: matched AI requests (method/path/headers/body, with model rewrite, path rewrite and header sanitize rules) are snapshotted at HandleForward and mirrored asynchronously to a shadow cluster via the new side-effect-free BalanceGslb.PickBackend(); the async side drains SSE responses to [DONE] while parsing usage/error/finish_reason and reports via a module-private Prometheus registry — the main request is never blocked, retried or double-mirrored on AI fallback (once-marker). Adds mirror_hit / mirror_cluster access-log fields (bfe-access-pb >= v0.3.8, fields 842-843); mirrored traffic is billing-isolated
• Add upstream error normalization behind AIConf.NormalizeUpstreamError (Enabled / StreamEnabled / UnrecognizedAction / MaxBodyBytes / RedactSecrets): per-protocol ErrorParser / StreamErrorParser adapters (openai/anthropic/gemini) map upstream errors into a unified UPSTREAM_* catalog; the interception point normalizes upstream failures after the fallback loop, with an SSE error-event filter, truncation detection and credential redaction covering raw/base64/URL/JSON-escaped forms; gateway-generated responses are skipped via the X-Bfe-Gw-Error marker. Adds ai_upstream_status / ai_upstream_err_code / ai_err_normalized / ai_err_normalize_miss / ai_stream_error_rewritten / ai_stream_truncated access-log fields (bfe-access-pb >= v0.3.12, fields 810-815); see ai_error_codes.md §2.5
• Add field-level enc$v1$ encryption at rest for distributed config files: sensitive fields in token_rule.data (Tokens outer keys) and cluster_conf.data (AIConf.Keys[].Key) may carry enc$v1$ AES-256-GCM ciphertext produced by the control plane at export, decrypted in memory at load time via a keyring configured in the new bfe.conf [Security] section (multiple keyIDs coexist, fail-closed parsing, error messages never carry ciphertext); hot reload re-reads the keyring on every data-file load, an unreadable or wrong keyring fails the reload and keeps the old config (fail-fast on first startup); plaintext without the marker passes through, keeping canary/rollback compatible
• Make the "usage unrecognized and not estimatable" billing bypass observable (issue #1410): the request-finish guard keeps its yardstick behavior (billing fields zeroed, no RMB deduction) but now increments the UsageUnrecognized module counter (exported via monitorHandlers, alertable) and logs a warn with cluster/key_id/model and the response_completed/allow_estimate sub-criteria. The access log gains ai_usage_source (bfe-access-pb >= v0.3.15, field 764) distinguishing parsed (final usage from the upstream response) / estimated (approved EstimateToken fallback) / unrecognized (billing skipped), orthogonal to the non-2xx suppression, so billing bypasses become directly queryable instead of requiring out-of-band reconciliation

Changed

• X-Bfe-Log-Id and the access-log logid are now a decimal string denoting a value in [1, 2^63-1] instead of a 32-char hex string, so the id converts losslessly into the uint64 logid field of the access log; the 63-bit bound keeps it inside the signed BIGINT of the report databases. Any consumer that parses or joins on the 32-hex form (X-Bfe-Log-Id, the text access log log_id, %bfe_log_id, WAF/mirroring logs) must be updated (#1417)
• Sessions/requests whose log id cannot be converted (mod_logid not loaded, malformed value, or a legacy 32-char hex header during a rolling upgrade) now carry a locally regenerated non-zero id and are counted by the new LogidRegenerated / SessionIdRegenerated module counters, instead of silently writing 0 (#1417)
• AI key session affinity now uses a server-owned Redis connection configured via the new [AIKeyAffinity] section in bfe.conf, instead of implicitly borrowing the mod_ai_rate_limit module's Redis. Config migration required: deployments that enabled affinity (AIConf.KeyPolicy.SessionAffinity=true) must add the [AIKeyAffinity] section (with Disabled=false and the Redis ServiceConf) before upgrading; otherwise affinity silently turns off after the upgrade (fail-open, no error, bindings lost)
• Server.EstimateToken now applies only when the upstream returned no usage at all, decided by the presence of the usage object: a response whose usage object reports all zeros is billed at 0 instead of being estimated, and is reported as ai_usage_source=parsed. Only that edge case changes amount (downward); requests whose response carries no usage keep the previous estimate behaviour (issue #1414)
• ai_cost_value is now written for every observed 2xx row, 0 included, so "charged nothing" is distinguishable from "field absent" (non-2xx / never billed). Reconciliation that tests ai_cost_value IS NULL for "no cost" should switch to ai_cost_value = 0 / IS NULL (issue #1414)
• mod_ai_token_auth.UsageUnrecognized no longer counts client aborts — they are counted by the new AbortBillingSkipped counter, so the two causes (upstream/parsing problem vs client-side disconnect) stay separable; existing thresholds on UsageUnrecognized need to be realigned. ai_usage_source gained the fourth value aborted (no new protobuf field: the value is added to the existing ai_usage_source) (issue #1414)

Fixed

• Populate the access-log logid with the request's real log id: strconv.ParseUint(req.LogId, 10, 64) parsed the hex log id as decimal and discarded the error, so logid was written as 0 for every request. Together with the report table's second-granularity dedup key and its full-row ON DUPLICATE KEY UPDATE, one API key and model firing n concurrent requests inside one second kept only 1 row — the measured loss was exactly (n-1)/n, 13.5% of a production hour. The session id column (session_id, and the session log's logid) had the same defect (#1417)
• Make usage recognition presence-based instead of value-based: an upstream that explicitly reports all-zero usage is no longer misclassified as "usage unrecognized" nor over-billed by EstimateToken (issue #1414)
• Report every zero-cost 2xx request that holds an RMB plan through the new BillingZeroCost counter and ai_zero_cost_reason marker, covering unrecognized usage, client aborts, batch releases, price-table gaps and pricing configuration loss at deduction time (issue #1414)
• Count failed quota deductions (DeductFailed) instead of leaving them as a log line only (issue #1414)
• Fix the token-triple consistency check (AiUsageInconsistent) returning early on ai_total_tokens == 0, which disabled it for exactly the input > 0 while total == 0 rows it exists to catch (issue #1414)
• Batch settlement now marks price misses (ai_price_miss), warns when the model table is missing and records a release without billing instead of staying silent (issue #1414)
• Measure the batch single-file limits (batch_limits.max_file_bytes / max_file_lines) on the multipart file part content instead of the request envelope: a 1-line/203-byte upload was rejected as BATCH_FILE_TOO_LARGE because the envelope (boundaries, part headers, purpose field) read as 11 lines / 509 bytes, and even a pre-accepted upload was aborted mid-stream by the same envelope count. The per-policy Content-Length pre-check now applies to raw uploads only, a pre-checked multipart upload no longer carries the streaming gate, and the gate parses the multipart envelope incrementally so only the file part is measured (also making chunked upload bindings exact instead of approximate) (#1434)
• Fix zeroed token usage in access logs and billing for requests without a final usage event (issue #1398): the request-finish reset now applies to a billing copy only, so the shared TokenUsage (read by mod_access_pb3) keeps observed/estimated values instead of silently logging total_tokens=0; final usage events with zero output tokens (input-only / pure cache-hit responses) are recognized instead of being dropped into the reset/estimate path; Server.InjectStreamUsage (default on) injects stream_options.include_usage=true into OpenAI streaming chat-completion requests when the client did not set it, so the upstream returns the real final usage chunk
• Parse the final usage embedded in the streaming finish chunk's choice (choices[N].usage, cache field cached_tokens) used by kimi-for-coding and other OpenAI-compatible upstreams that ignore the injected stream_options.include_usage (issue #1401): the real usage is no longer demoted to a guess, so EstimateToken=true no longer logs/bills a forged ContentLength/4 estimate and EstimateToken=false no longer logs (0, -1, 0) and bills 0; the negative CompletionTokens sentinel (auth-time "unknown" marker) is clamped to 0 at the access log boundary. EstimateToken and InjectStreamUsage now default to true as program defaults (ConfigBasic.SetDefaultConf), so the fixes no longer silently disappear when a binary is upgraded without the new conf keys
• Fix token usage collection for non-streaming gzip responses (Content-Encoding: gzip with Transfer-Encoding: chunked) (issue #1406): the compressed body is now decompressed on a parse-only copy (forwarding bytes untouched), the whole-body parse gate accepts chunked non-SSE JSON bodies, and the content-type decoder tolerates media type parameters (e.g. application/json; charset=utf-8), so ai_output_tokens / ai_total_tokens / ai_cache_read_tokens and RMB billing reflect the real usage instead of 0
• Fall back to the model's chat-mode price when the (model, mode) price entry is missing but a chat entry exists (product decision: an unpriced mode bills the same as chat), and mark such requests with the ai_price_miss access-log field (bfe-access-pb >= v0.3.14, field 763) counted by PriceFallbackUsed, so under-priced rows stay visible in reconciliation (issue #1408)
• Fix token-priced image_generation/video_generation rules billing 0 (issue #1420): the billing formula was picked by request mode alone, so an entry whose prices only carry token unit keys was billed with the per-image/per-video formula and skipped (costUnits > 0 gate). The billing method is now entry-driven — calcImageGenerationCost / calcVideoGenerationCost fall back to the token formula when the matched entry lacks the per-unit price key; discrimination is key presence, not value, so a mixed entry bills per unit and an explicit output_cost_per_image = 0 stays per-image billing at zero cost
• Drop the EstimateToken residue (auth-time prompt seeding / response-byte completion guessing) from the shared TokenUsage and suppress the token fields in the access log for non-2xx responses (new usage disposition, UsageDispositionNone): a failed request no longer logs request-body/4 as ai_input_tokens while ai_total_tokens stays 0 (issue #1409)
• Bill streaming (SSE) responses that end at a clean HTTP EOF without the protocol termination event ([DONE] / message_stop) and without any usage, instead of silently deducting 0 (issue #1415): the response is marked completed once the body has been fully delivered (200 SSE, successful copy) — a chunked response only EOFs at its terminating chunk, so client aborts and mid-stream upstream drops still surface as copy errors and never satisfy the billing gate
• Keep the client connection reusable after internally constructed AI responses (issue #1416): the no-route 404 (AI route not found), the findProduct 500, the protocol-mismatch rejection, the HandleAfterAITargetModel rejection and the no-cluster 500 no longer set the server-level closeAfterReply — BFE was closing the connection without advertising Connection: close, so the next request on a pooled keep-alive connection failed with EOF; module-contract finish/close paths and the protocol-level close shapes (HTTP/1.0, Connection: close, undrainable bodies) still close with advertisement
• Return a decidable synthetic 429 when no AI key can be selected for the target cluster (all keys penalized / Weight=0 / dead) instead of (res=nil, err=nil) (issue #1440): aiClusterInvoke now surfaces the result so the fallback decision can act on it — a configured fallback cluster is actually tried, and without one the client gets a retryable unified-error 429 instead of an opaque empty-body 500; shouldTriggerFallback treats (nil, nil) as fallback-eligible as a hard invariant
• Exclude Weight=0 keys from the session-affinity penalty filter and replace the "fallback to the original list" branch with a circuit breaker (issue #1439): when every usable key is penalized the filter returns no candidates — key selection fails, the synthetic 429 triggers cluster fallback, and the cluster makes zero upstream interaction until the penalty TTL expires (new 429s extend the window). Previously one upstream 429 against the only usable key was amplified into a 60s cluster-wide outage by a Weight=0 sibling key
• Emit the unified error contract on gateway-synthesized AI failures (issue #1441): the key-exhaustion 429 now carries error.code = NO_AVAILABLE_KEY (429, rate_limit_error), error.details.retry_after_seconds (equal to the 429 penalty TTL, single-sourced) and a standard Retry-After header, and sets err_code = NO_AVAILABLE_KEY for access-log attribution without suppressing cluster fallback; gateway-side failures in ServeHTTPForAI (findProduct failure, no cluster, backend connect/transport errors) return BACKEND_UNAVAILABLE (502) with the unified body instead of a bare empty-body 500, and client-abort paths keep the legacy empty-body record. Gateway-generated error responses now also carry request correlation identifiers: X-Bfe-Log-Id is always written (fallback chain request logid → request header → generated, with the latter two written back so the access-log logid stays same-source) and a client-supplied X-Request-ID is echoed back, so clients can correlate an error response with the access log and the key selection/penalty trail
• Make the outbound credential headers mutually exclusive, so a client-supplied credential can no longer override the gateway-injected upstream key (issue #1447): once BFE takes ownership of the upstream credential, the outbound request keeps exactly the one credential header the request's model_protocol adapter injects and drops the others (Authorization / x-api-key / x-goog-api-key, the sanitize set derived from the adapter registry via the new ProtocolAdapter.CredentialHeader()); the deletion runs on a cloned header map so the inbound client credential stays available to authentication, billing, access-log masking and the key-affinity session id. Previously InjectAuth only overwrote its own header, so a client sending both Authorization and x-api-key left its gateway-level x-api-key on the outbound request — kimi authenticated with the client token, answered 401, and that 401 was charged to the selected key: two requests marked both kimi keys dead for 3600s and every user was silently degraded to the fallback cluster while the client still saw 200. Clusters without configured keys (BYOK passthrough) are unaffected
• Release old backends when a subcluster disappears from cluster_table on reload (issue #1436): a K8s instance pool scaled to 0 exports an empty cluster entry, and the reload previously skipped the missing subcluster and kept the stale backends, so requests got HTTP 200 against released instances instead of the required 500 BK_NO_BACKEND; the empty pool is now loaded with an empty backend list (releasing old instances), and a subcluster with no weight>0 backends no longer fails the whole table load (warning + accept, the empty pool is covered by alerting)
• Keep ai_apikeytags in the access log for rejected requests (issue #1418): quota pre-check rejections (429 QUOTA_EXHAUSTED) and the other post-lookup rejection paths (KEY_DISABLED / KEY_EXPIRED / QUOTA_EXPIRED / INTERNAL_QUOTA_ERROR / SUBNET_NOT_ALLOWED) returned before SetTokenAuthContext, the only writer of AiBasicInfo.ApikeyTags, so the rejected log carried the key id but missed the tags and lost entity attribution; ApikeyTags is now written together with ClientKeyId right after the token is found
• Rebuild the EPP runtime on hot reload when non-scheduling parameters change (issue #218): EPPTLS, EPPTimeout.Connect, EPPCheck.Disabled and the EPP address list now form a runtime signature — a changed signature rebuilds the runtime (reusing the graceful retire path and inheriting active/health state when the address table is unchanged), an unchanged one updates in place without resetting the circuit-breaker sliding window; the EPPTLS default flips from {Insecure: true} (TLS with verification skipped) to {Plaintext: true} (plaintext dial), while explicit {"Insecure": true} / {"CAFile": ...} still use TLS
• Fix batch settlement and terminal-state persistence for completed passthrough batches (rainway-ai-gateway/ai-gateway-api#254): batch result rows nest model/usage under response.body (OpenAI contract), which the usage scanner now parses, so downloading the output file actually settles per-model usage at batch prices (BATCH_SETTLED idempotent claim, and SETTLE_USAGE_PARSE_ERR is now counted instead of silent); the finish handlers persist the terminal facts the control plane needs (reserved_units, created_at, endpoint, request_counts, settle_units, settle_status), and the recorded output-file size describes the file rather than the HTTP envelope
• Skip the request-body model rewrite for batch/file passthrough traffic (#1426): with a non-empty AI route target model, the rewrite replaced the whole multipart body of POST /v1/files with {"model":...} (providers failed multipart parsing with 400 multipart: NextPart: EOF, breaking the batch upload/create chain) and injected a model field into POST /v1/batches; the rewrite now skips ModeFile/ModeBatch traffic and any request whose Content-Type explicitly says non-JSON (missing/malformed headers and */*+json still allow it)
• Apply the global batch_limits file byte/line hard caps to API keys bound to no rate-limit policy: the limit resolution previously ran only on the policy-bound path, so an unbound key bypassed the global hard top (a 1.5MB upload passed under a 1MB cap); the unbound branch now resolves the global hard top only, matching the "policy without batch section ⇒ global hard cap only" design