Skip to content

test: keep tracing callsites enabled so log-capture tests stop flaking - #909

Merged
jarvis9443 merged 1 commit into
mainfrom
fix/auth-denial-log-capture
Aug 7, 2026
Merged

test: keep tracing callsites enabled so log-capture tests stop flaking#909
jarvis9443 merged 1 commit into
mainfrom
fix/auth-denial-log-capture

Conversation

@jarvis9443

@jarvis9443 jarvis9443 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Problem

auth::tests::a_missing_credential_denial_names_the_caller_and_the_route fails intermittently on main — it took down the rust unit + coverage job on the push run for #904 (run 31137791884):

panicked at crates/aisix-proxy/src/auth.rs:367:9:
got: 

The captured buffer is empty even though the subscriber is installed and the denial is emitted. Adding a marker event inside the same capture scope shows the marker but not the denial, so the subscriber and the thread are fine — the specific callsite is disabled.

Root cause

A callsite's Interest is cached process-wide the first time that callsite is hit, and it is computed from whatever dispatcher the hitting thread has (tracing_core::callsite::rebuild_callsite_interest, which falls through to dispatcher::get_default on that thread while only one dispatcher is registered).

capture_logs installs its subscriber with set_default, which is thread-local. When a sibling test reaches the same unauthenticated request path on another thread, that thread resolves to NoSubscriber, the callsite is cached as Interest::never(), and the event is skipped everywhere from then on — including inside the capture. Dozens of router tests hit the unauthenticated path, so under the parallel harness this is routine rather than exotic.

Fix

Install a bare Registry as the global default before capturing. It formats and writes nothing, but it means no thread ever falls back to NoSubscriber, so no callsite is ever cached as disabled. Registering a dispatcher also re-evaluates the callsites seen so far, so the lazy install repairs a cache already poisoned earlier in the run.

The same helper shape in aisix-guardrails (aliyun.rs, aliyun_ai_guardrail.rs) has the same defect and gets the same guard — its existing TRACING_CAPTURE_LOCK only orders capture tests against each other and does nothing about callsites registered by unrelated tests. A permanently registered dispatcher also pins the global max-level hint, which is the symptom that lock was originally added for.

aisix-obs's three with_default sites are audited and not exposed: the access-log callsite is only ever emitted from inside those captures, so every thread that can register it already has a subscriber. Left unchanged.

Verification

Same command, same machine, --test-threads=64 to raise the interleaving pressure:

before after
aisix-proxy --lib 4 of 6 runs failed 0 of 8 failed (plus 3/3 at default threads)
aisix-guardrails --lib 5 of 6 runs failed 0 of 16 failed

Suite wall-clock is unchanged (aisix-proxy 12.69s → 12.66s at default threads), which is why the fix uses a bare registry rather than a discarding fmt subscriber — the latter formats every event and cost ~1s on the guardrails suite.

Workspace cargo clippy --all-targets -- -D warnings and cargo fmt --check are clean.

No new test

The trigger is a process-global, order-dependent race in tracing's callsite cache: a callsite is registered at most once per process and cannot be un-registered, so the poisoning window cannot be forced deterministically from inside the same test binary. Verified statistically instead, as above.

Summary by CodeRabbit

  • Bug Fixes
    • Improved test log capture so denial and guardrail-related tracing messages are consistently retained.
    • Prevented previously cached tracing settings from suppressing captured logs during authentication and guardrail tests.

`auth::tests::a_missing_credential_denial_names_the_caller_and_the_route`
fails intermittently on main: the capture returns an empty buffer even
though the subscriber is installed and the denial is emitted.

A callsite's `Interest` is cached process-wide the first time that
callsite is hit, computed from whatever dispatcher the *hitting* thread
has. `capture_logs` installs its subscriber with `set_default`, which is
thread-local, so when a sibling test reaches the same unauthenticated
path on another thread the callsite resolves against `NoSubscriber`,
caches `Interest::never()`, and the event is skipped everywhere from then
on -- including inside the capture. Dozens of router tests hit that path,
so under the parallel harness the race is routine.

Install a bare `Registry` as the global default before capturing. It
formats nothing, but it means no thread ever falls back to
`NoSubscriber`, so no callsite is cached as disabled; registering it also
re-evaluates the callsites seen so far, so the lazy install repairs a
cache poisoned earlier in the run.

The same helper shape in aisix-guardrails has the same defect and gets
the same guard. `aisix-obs`'s `with_default` sites are not exposed: the
access-log callsite is only ever emitted from inside those captures.
Copilot AI balanced review requested due to automatic review settings August 7, 2026 07:44
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f4ef3410-7f77-4830-82ff-3eb26c50da28

📥 Commits

Reviewing files that changed from the base of the PR and between 8cc0359 and 6b80eee.

📒 Files selected for processing (4)
  • crates/aisix-guardrails/src/aliyun.rs
  • crates/aisix-guardrails/src/aliyun_ai_guardrail.rs
  • crates/aisix-guardrails/src/lib.rs
  • crates/aisix-proxy/src/auth.rs

📝 Walkthrough

Walkthrough

The changes add test-only tracing initialization that keeps callsites enabled and repairs cached interests. Guardrails and proxy log-capture helpers invoke this initialization before installing capture subscribers.

Changes

Tracing callsite retention

Layer / File(s) Summary
Global callsite initialization
crates/aisix-guardrails/src/lib.rs, crates/aisix-proxy/src/auth.rs
Test helpers install one-time permissive tracing subscribers and repair cached callsite interests.
Capture helper integration
crates/aisix-guardrails/src/aliyun.rs, crates/aisix-guardrails/src/aliyun_ai_guardrail.rs, crates/aisix-proxy/src/auth.rs
Log-capture helpers initialize tracing callsite state before installing scoped capture subscribers.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • api7/aisix#781: Modifies Aliyun guardrail test log-capture behavior.
  • api7/aisix#799: Modifies tracing and log-capture test infrastructure.

Suggested reviewers: copilot, moonming

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
E2e Test Quality Review ⚠️ Warning Both new helpers discard set_global_default(...) via let _ =; if registration fails, callsites remain vulnerable and the test fix silently provides no protection. Handle the Result explicitly. Fail the test setup or report a clear error when the permissive global subscriber cannot be registered.
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: preventing flaky log-capture tests by keeping tracing callsites enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Security Check ✅ Passed All seven categories have no findings: the 56 added lines are #[cfg(test)] tracing setup only and add no logs, storage, authorization, ownership, TLS, isolation, or secret resolution.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/auth-denial-log-capture

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Stabilizes tracing log-capture tests by preventing process-wide callsite disabling.

Changes:

  • Registers a silent global tracing registry in test binaries.
  • Applies the guard to proxy authentication and guardrail log-capture helpers.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
crates/aisix-proxy/src/auth.rs Enables tracing callsites before authentication log capture.
crates/aisix-guardrails/src/lib.rs Adds the shared test-only tracing initializer.
crates/aisix-guardrails/src/aliyun.rs Initializes tracing before Aliyun log capture.
crates/aisix-guardrails/src/aliyun_ai_guardrail.rs Initializes tracing before AI Guardrail log capture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@jarvis9443
jarvis9443 merged commit 2d7132e into main Aug 7, 2026
11 checks passed
@jarvis9443
jarvis9443 deleted the fix/auth-denial-log-capture branch August 7, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants