Skip to content

fix(diagnostics): classify core error markers case-insensitively#880

Merged
bingran-you merged 1 commit into
mainfrom
codex/fix-541-scoring-case
Jul 2, 2026
Merged

fix(diagnostics): classify core error markers case-insensitively#880
bingran-you merged 1 commit into
mainfrom
codex/fix-541-scoring-case

Conversation

@bingran-you

Copy link
Copy Markdown
Collaborator

Summary

  • make classify_error() use the already-normalized lowercase error string for install failed and ACP error markers
  • add regression coverage for capitalized install failures and lowercase ACP errors

Fixes #541.

Verification

  • uv sync --extra dev --locked
  • uv run python -m pytest tests/test_scoring.py -q
  • uv run ruff check src/benchflow/_utils/scoring.py tests/test_scoring.py
  • uv run ruff format --check src/benchflow/_utils/scoring.py tests/test_scoring.py
  • uv run ty check src/benchflow/_utils/scoring.py

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 2, 2026 16:28 — with GitHub Actions Inactive
@bingran-you
bingran-you force-pushed the codex/fix-541-scoring-case branch from 7a01e4e to 768f54e Compare July 2, 2026 16:29
@bingran-you
bingran-you temporarily deployed to pypi-internal-preview July 2, 2026 16:29 — with GitHub Actions Inactive
@bingran-you

Copy link
Copy Markdown
Collaborator Author

Automation validation (2026-07-02): focused fix for #541 is ready for review.

Validated on head 768f54e1:

  • uv sync --extra dev --locked
  • uv run python -m pytest tests/test_scoring.py -q (48 passed)
  • uv run ruff check src/benchflow/_utils/scoring.py tests/test_scoring.py
  • uv run ruff format --check src/benchflow/_utils/scoring.py tests/test_scoring.py
  • uv run ty check src/benchflow/_utils/scoring.py

Merge gate: wait for GitHub CI plus normal non-author review before squash merge.

@bingran-you bingran-you added bug Something isn't working P2 Anti-pattern / type safety / docs precision / minor schema drift / non-deterministic but contained. status:ready Triaged, unassigned, available to claim. review:pending PR is ready-for-review, no reviewer engagement yet. area:diagnostics Issue / PR lives primarily in the "diagnostics" subsystem. labels Jul 2, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a01e4e6a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tests/test_scoring.py Outdated
assert classify_error("installing dependencies") == "other"

def test_error_classification_is_case_insensitive_for_core_markers(self):
"""Guards issue #541: core diagnostic markers should ignore case."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Name the guarded commit in the regression docstring

The repo-level AGENTS.md requires regression tests to name the PR or commit they guard, but this new regression docstring only cites issue #541. Please include the guarding PR or commit identifier here so the test follows the documented convention and remains traceable to the exact fix.

Useful? React with 👍 / 👎.

@bingran-you

Copy link
Copy Markdown
Collaborator Author

Users Simulation follow-up (2026-07-02T16:45Z): current head 768f54e1 is ready for this simulation scope. Focused local checks passed (tests/test_scoring.py, ruff check/format, ty on the scoring utility), and GitHub checks are now green: test, pip-audit, manifest-parity, integration-scope detect-scope, and integration-light detect-scope + rollout-smoke. Labels remain status:ready + review:pending; normal non-author review gate still applies.

@bingran-you
bingran-you merged commit df8c308 into main Jul 2, 2026
15 of 19 checks passed
@bingran-you
bingran-you deleted the codex/fix-541-scoring-case branch July 2, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:diagnostics Issue / PR lives primarily in the "diagnostics" subsystem. bug Something isn't working P2 Anti-pattern / type safety / docs precision / minor schema drift / non-deterministic but contained. review:pending PR is ready-for-review, no reviewer engagement yet. status:ready Triaged, unassigned, available to claim.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Guide §16 pytest -k selector and string-vs-constant drift silently hide failure-path coverage

1 participant