Skip to content

test: let LLM budget failures fail CI#446

Closed
ling-senpeng13 wants to merge 6 commits into
mainfrom
test/skip-out-of-budget-llm-providers
Closed

test: let LLM budget failures fail CI#446
ling-senpeng13 wants to merge 6 commits into
mainfrom
test/skip-out-of-budget-llm-providers

Conversation

@ling-senpeng13

@ling-senpeng13 ling-senpeng13 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Behavior

  • missing provider credentials may still skip tests that cannot run
  • an exhausted provider budget is a CI failure
  • unrelated provider and test failures continue to propagate normally

Validation

  • targeted Agent E2E test collection: 3 tests collected
  • python -m compileall -q e2e/test_suite1_basic_validation.py e2e/test_suite25_media_input.py
  • git diff --check

This supersedes the budget-skip behavior added by #445.

@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ling-senpeng13 ling-senpeng13 self-assigned this Jul 24, 2026
@ling-senpeng13 ling-senpeng13 changed the title test: skip LLM providers marked out of budget test: let LLM budget failures fail CI Jul 24, 2026
@ling-senpeng13

Copy link
Copy Markdown
Contributor Author

Closing because LLM budget exhaustion should be fixed manually rather than handled by test logic.

@ling-senpeng13
ling-senpeng13 deleted the test/skip-out-of-budget-llm-providers branch July 24, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant