fix(test): pin consolidator fallback test to the config model - #374
Merged
Conversation
`_build_real_client_consolidator` patched the generation site var to a fake `"gpt-test"` model, so the `LiteLLMConfig(model=...)` each test passed never reached the request — `PlaybookConsolidator` resolves its request model from the site var, not from the client config. The structured-output fallback guard added in #372 resolves a transport strategy per model. `"gpt-test"` is unknown to LiteLLM, so it resolved to `pydantic_passthrough` while the `gpt-5.4-mini` fallback resolved to `native_json_schema`, and the guard rejected the ladder: ValueError: Structured-output fallback models must use the same transport strategy as 'gpt-test' (pydantic_passthrough); incompatible fallbacks: ['gpt-5.4-mini'] `test_consolidator_calls_litellm_with_fallback_configured` has been failing on main since #372. Pin the site var to `config.model` so the test exercises the model it declares. The fallback-configured case now runs a realistic production pairing (`minimax/MiniMax-M3` primary + `gpt-5.4-mini` fallback, both `native_json_schema`) instead of an artificially incompatible one.
`_wait_for_operation`'s skew condition was committed hand-wrapped and is the only file in the package that `ruff format --check` flags. No behavior change.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe client’s stale-operation condition is reformatted without behavioral change. The playbook consolidator integration helper now derives its default generation model from the supplied ChangesPolling and fallback alignment
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
mainis red.test_consolidator_calls_litellm_with_fallback_configuredhas beenfailing since #372 landed:
Root cause
_build_real_client_consolidatorpatched the generation site var to a fake"gpt-test", so theLiteLLMConfig(model=...)each test passed never reached therequest —
PlaybookConsolidatorreads its request model from the site var(
deduplication_utils.py:191), not from the client config, and passes it asmodel=self.model_name(components/consolidator.py:743).#372 added
_validate_structured_fallback_strategies, which resolves a transportstrategy per model and rejects a ladder whose fallbacks disagree with the primary.
Measured strategies:
gpt-testpydantic_passthroughgpt-5.4-mininative_json_schemaminimax/MiniMax-M3native_json_schemaThe fake model made the pairing artificially incompatible. The guard itself is
correct — the test was asserting against a model it never intended to use.
Fix
Pin the patched site var to
config.model. The fallback-configured case nowexercises a realistic production pairing (MiniMax primary + OpenAI fallback, both
native_json_schema). The second test (claude-code/..., no fallbacks) neverreaches the guard and is unaffected.
Also included: a
style:commit applyingruff formattoclient/client.py,the only file in the package
ruff format --checkflagged. No behavior change.Verification
pytest tests/server/services/playbook/test_playbook_consolidator_integration.py— 14 passed (1 failed before)ruff format --check reflexio tests— 821 files already formattedruff check+pyrightclean on both touched filesSummary by CodeRabbit
Bug Fixes
Tests