Skip to content

Release gate cell P2 is misconfigured and can never pass #5614

Description

@mmabrouk

Found during the 0.106.2 release gate run. Not a product bug. The P2 cell in resources/qa_product.py (agent-release-gate skill) asks for a combination the product rejects by design, so every turn-based journey in that cell fails on any deployment.

Two separate problems

1. The connection mode contradicts the slug. The cell declared:

"connection": {"mode": "self_managed", "slug": None}  # slug filled from --custom-slug

The API rejects that pair outright:

HTTP 500: 1 validation error for AgentTemplate
connection: mode 'self_managed' must not carry a 'slug'

That is correct behaviour. Per sdks/python/agenta/sdk/agents/connections/models.py, agenta mode uses a vault connection and slug selects which one, while self_managed injects nothing so a slug has nothing to resolve against. P2's whole point is a vault-backed custom provider, so the mode must be agenta.

I changed it to agenta locally, which moved the failure on to the second problem.

2. The model family is incompatible with a custom provider on Pi. With the mode fixed:

HTTP 422: provider 'custom' is not supported by harness 'pi_core'

sdks/python/agenta/sdk/agents/capabilities.py documents the allowed triples:

pi_core/pi_agenta + openai + direct or custom  -> allowed
pi_core/pi_agenta + any other family + custom  -> rejected

P2 pins deepseek/deepseek-v4-flash, a deepseek-family model, with provider: custom. Rejected by design.

This is not a regression

Both rules are unchanged in 0.106.2. I diffed v0.106.1...release/v0.106.2: neither connections/models.py's validator nor capabilities.py appears. So the cell has been failing for as long as those rules have existed, on every deployment.

Why it matters

P2 covers "the path every self-hoster with a proxy or local vLLM uses, and the least-travelled one", per coverage.md. That path currently has no working coverage in the gate, and the failure looks like a product problem in every run, which costs triage time on every release.

What a fix needs

Set mode: agenta (done locally), and pin a model whose family the rule allows with a custom provider on Pi, which means an openai-family model id reached through the custom OpenAI-compatible endpoint rather than a deepseek id. Whoever owns the gate should choose the exact fixture, since coverage.md says the pinned models are deliberate.

Current state

Cells C2, C3, C4 and P1 all pass every journey on 0.106.2. P2 is the only red cell and it is red for this reason.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions