PR #393 fixed tests/e2e/test_add_flush_agent_pipeline_e2e.py so it actually exercises the agent-skill chain — before that fix it structurally could not, and that is why a defect making the chain fail 4/4 reached a release.
The original failure had two layers:
tests/conftest.py:68-88 is an autouse fixture forcing EmbeddingCapability(provider=None) for every test (hermeticity — correct in itself). trigger_skill_clustering and extract_agent_skill both body-guard on get_embedding_capability().available, so both returned early. Measured log counts from an actual run: agent_case_extracted=5, skill_cluster_updated=0, agent_skills_extracted=0, strategy_gated_off_embedding_unavailable=10.- Its three skill assertions were
assert len(...) >= 0 — always true. The inline comment rationalized this as LLM-dependent flakiness; the count was in fact necessarily zero.
#393 opts the test in to a real embedding capability and gives it a meaningful floor plus a dead-letter assertion. But it keeps the slow + live_llm markers, and CI injects no provider credentials — so this remains the only end-to-end coverage of the chain, and it still never runs automatically.
Worth deciding how to close that gap. Options: a scheduled (nightly / weekly) job with credentials in secrets, running just the live_llm set; a recorded-fixture replay so the chain can be exercised without credentials (the examples/langfuse trace-replay work may be reusable); or accept it and add a release-checklist step that runs it manually.
Related: the same class of gap likely applies to other live_llm tests — worth auditing which of them, if any, are the sole coverage of their path.
PR #393 fixed
tests/e2e/test_add_flush_agent_pipeline_e2e.pyso it actually exercises the agent-skill chain — before that fix it structurally could not, and that is why a defect making the chain fail 4/4 reached a release.The original failure had two layers:
tests/conftest.py:68-88is anautousefixture forcingEmbeddingCapability(provider=None)for every test (hermeticity — correct in itself).trigger_skill_clusteringandextract_agent_skillboth body-guard onget_embedding_capability().available, so both returned early. Measured log counts from an actual run:agent_case_extracted=5,skill_cluster_updated=0,agent_skills_extracted=0,strategy_gated_off_embedding_unavailable=10.assert len(...) >= 0— always true. The inline comment rationalized this as LLM-dependent flakiness; the count was in fact necessarily zero.#393 opts the test in to a real embedding capability and gives it a meaningful floor plus a dead-letter assertion. But it keeps the
slow+live_llmmarkers, and CI injects no provider credentials — so this remains the only end-to-end coverage of the chain, and it still never runs automatically.Worth deciding how to close that gap. Options: a scheduled (nightly / weekly) job with credentials in secrets, running just the
live_llmset; a recorded-fixture replay so the chain can be exercised without credentials (theexamples/langfusetrace-replay work may be reusable); or accept it and add a release-checklist step that runs it manually.Related: the same class of gap likely applies to other
live_llmtests — worth auditing which of them, if any, are the sole coverage of their path.