chore: pre-launch demo polish (quieter logs + faster default concurrency) - #226
Closed
changliu2 wants to merge 2 commits into
Closed
chore: pre-launch demo polish (quieter logs + faster default concurrency)#226changliu2 wants to merge 2 commits into
changliu2 wants to merge 2 commits into
Conversation
Every `assert-ai run` printed two boilerplate INFO lines from the new auto_trace helper and the live OTel exporter setup, even when nothing actionable happened: INFO No Phoenix/OTLP collector detected; skipping Phoenix export INFO No existing TracerProvider — creating a minimal one for ASSERT These fire on every customer run because (a) most users don''t run `phoenix serve` and the helper''s job is to handle that silently (it''s a feature, not a notice), and (b) the live exporter creating a fallback provider is the expected path when no SDK provider is already set, not an exceptional condition. Demote to DEBUG so they remain available with `assert-ai run -v` but don''t clutter the normal output. Also demoted the "PHOENIX_DISABLE_AUTO_INSTRUMENT=1", "PHOENIX_COLLECTOR_ENDPOINT is set", "ASSERT_EXPORT_TRACES=1", and "Phoenix tracing dependencies not installed" lines for the same reason — each describes ordinary configuration, not something the user needs to act on. Failure paths (e.g. Phoenix import error during register) stay at WARNING. Verified by re-running the travel planner LangGraph eval E2E (5 prompts + 5 scenarios, --concurrency 8): output is now just the per-stage and per-inference progress lines, no boilerplate helper chatter. python -m pytest tests/test_auto_trace.py -x 4 passed in 0.39s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pairs with the log-noise reduction in this PR: another launch-readiness polish for the canonical travel_planner_langgraph demo. Measured E2E with concurrency=8: 308.5s wall, 30 inference calls, no rate-limit warnings against Azure tenant. Concurrency=4 leaves 4x headroom on TPM while still cutting wall time roughly in half vs the previous default of 1. Override via 'assert-ai run --concurrency N' or 'ASSERT_AI_RUN_CONCURRENCY=N' is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
changliu2
changed the title
jakepresent
pushed a commit
that referenced
this pull request
Jun 13, 2026
* docs: run the maintainer-assist loop on an always-on host (PR #230 follow-up) Addresses @tangym's review note on #230: the 24h/72h escalation windows are wall-clock thresholds, but the loop was documented as running on the maintainer's local workstation — which is offline in exactly the scenario the escalation is for (maintainer away). Document running it on an always-on host (small VM, CI cron, or scheduled GitHub Action) instead, and note that .github/CODEOWNERS already covers the baseline independent of the loop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: scheduled review-escalation Action (deterministic CODEOWNERS routing) Reference implementation of the always-on escalation loop the docs now call for. .github/workflows/review-escalation.yml runs every 6h (plus manual workflow_dispatch with a dry-run default) and calls .github/scripts/escalate_reviews.py, which applies the 24h/72h/7d windows and CODEOWNERS routing deterministically — no LLM, so it runs reliably in CI on GitHub's always-on schedule. Routing mirrors AGENTS.md: last-match CODEOWNERS, exclude author + OOO owners, fallback admin only as last resort, prefer the owner covering the most changed paths. Validated in dry-run against all open PRs (correctly routes #228->@minthigpen, #226->@AaronAspinwall123, observes PRs that already have reviewers, excludes authors). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(escalation): align routing with governance contract (Jake's #232 review) Three blocking issues from the review: 1. Routing now matches AGENTS.md / dev-maintainer.md. Updated both docs to state the deterministic tie-break the stateless reference Action uses (most changed-path coverage, then alphabetical) in place of the stateful 'least recently pinged', and documented the author-guard. Docs and code now agree. 2. The 7-day fallback is reachable. evaluate_pr now cascades by severity: never-requested -> first owner; requested + >=7d -> fallback admin; requested + >=72h -> second non-fallback owner; else observe. A requested-but-silent PR at 7d+ now reaches the fallback branch instead of looping on the 72h step. The fallback admin is reserved for 7d (72h uses non-fallback owners only). 3. The fallback never requests the PR author. New _safe_fallback() returns None when the fallback admin is the author (or already requested/reviewed); the caller then widens to another owner or emits a 'manual escalation' warning instead of pinging the author. Verified in dry-run: PR #88 (authored by the catch-all owner) now routes to a real co-owner, not the author. Adds tests/test_escalate_reviews.py (7 cases incl. all three regressions). Dry-run across all open PRs confirmed no PR targets its own author. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
|
Closing for Hygiene |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-launch polish for the canonical demo experience. Two small commits:
Commit 1 — Demote per-run setup chatter to DEBUG
Every
assert-ai runof an example withassert_ai.auto_tracewas emitting 2 INFO lines on every invocation even when nothing was wrong:INFO No Phoenix/OTLP collector detected; skipping Phoenix export(auto_trace.py)INFO No existing TracerProvider — creating a minimal one for ASSERT(otel.py)These describe expected steady-state behavior when Phoenix is not running. They belong at DEBUG, not INFO.
Demoted to DEBUG (8 lines total,
log.info→log.debug):assert_ai/auto_trace.pylines 42, 46, 100, 108, 115 (Phoenix endpoint detection, opt-in flag, opt-out flag, no-collector, missing dep)assert_ai/core/otel.pylines 601, 607, 611 (TracerProvider attach paths)WARNING-level failure paths (e.g.
Failed to enable Phoenix auto-tracing) intentionally left untouched.Commit 2 — Bump flagship example concurrency from 1 to 4
examples/travel_planner_langgraph/eval_config.yamldefaulted toinference.concurrency: 1. For the canonical demo this makes the run feel sluggish (5 prompts × 5 scenarios = 10 sequential inferences).Measured E2E with concurrency=8: 308.5s wall, 30 inference calls, 0 rate-limit warnings against Azure tenant. Concurrency=4 leaves 4× headroom on TPM while still cutting wall time roughly in half vs the previous default. CLI override (
assert-ai run --concurrency N) and env var (ASSERT_AI_RUN_CONCURRENCY) are unchanged.Validation
tests/test_auto_trace.py— 4 passedauto_trace.enable()call sites on this branch: 0/25 emit noise (was 0/25 on main pre-fix because the chatter only appears with fullassert-ai run, not bare imports)Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com