feat(mcp_registry): #201 PR 4 of 5. HTTPMCPServerRegistryBackend read paths + tier negotiation

[dep0we]

Summary

Ships the HTTP-catalog reference implementation for the MCPServerRegistryBackend Protocol (#201 PR 4 of 5):

• atomic_agents/mcp_registry/http.py NEW (~1050 lines): HTTPMCPServerRegistryBackend with read paths (list/load/load_all/validate), tier-1/2/3 capability negotiation via the Decision 4 five-step probe, threading-locked capability cache, per-request bearer auth headers, full httpx exception mapping, response-body shape validation against catalog-server injection, and the bulk ?expand=spec endpoint that eliminates N+1 round trips at agent construction (MUST 10). Install/uninstall stubs raise NotImplementedError until PR 5.
• MCPServerSpec.to_dict() / from_dict() promoted to public methods on the dataclass in atomic_agents/mcp.py so the HTTP backend can deserialize wire JSON without depending on profile/types.py private helpers. The helpers in profile/types.py now delegate to the class methods (backward-compat preserved).
• Factory http branch added to get_default_mcp_server_registry_backend in mcp_registry/__init__.py. Operators set ATOMIC_AGENTS_MCP_SERVER_REGISTRY_BACKEND=http plus ATOMIC_AGENTS_MCP_SERVER_REGISTRY_BACKEND_URL=https://catalog/?agent_scope=... and optionally ATOMIC_AGENTS_MCP_SERVER_REGISTRY_AUTH_TOKEN=... to wire the HTTP backend without touching code.
• [http] optional extra (httpx>=0.27) added to pyproject.toml. Filesystem-default operators pay zero httpx import cost (lazy import inside the http branch).
• spec/36 PR 4 amendments (still DRAFT until PR 5 LOCK): four new subsections (§HTTP wire format with Required/Optional field tables for catalog server authors, §Tier negotiation, §Capability handshake static-vs-runtime, §Per-scope filtering) plus the httpx exception mapping table and the http factory env-var documentation.

Workflow validation: 5-stream Sonnet prep pass caught 36 findings (8 P0, 18 P1, 10 P2) before any code shipped; the pre-landing /ship review army (5 specialists + Claude adversarial + Step 9 checklist) surfaced 34 follow-up findings of which 8 P0 / P1 were applied inline before push. See the Pre-Landing Review section for the full list.

Test Coverage

Coverage: 90% (subagent-assessed) across 6 audited code surfaces in http.py
Tests: 3232 → 3307 (+75 net new)
Quality breakdown across http.py surfaces:
  ★★★ behavior + edge + error:  MCPServerSpec round-trip, validate_server_name,
                                  _redact_url_for_error, _parse_mcp_server_spec_from_dict,
                                  _parse_capabilities_response, _materialize_spec,
                                  _ensure_probed (cache hit + failure + success),
                                  capabilities property + refresh, _auth_headers
  ★★  happy path:                Factory http branch (kwarg path via conformance),
                                  _parse_servers_list_to_refs, _parse_validation_result,
                                  _get_client, close, backend_id
  ★   smoke check:               install/uninstall stubs via conformance gate
Remaining gaps (4 P3 follow-ups, captured in spec/36 reserved at lock):
  - mid-session tier regression handling (PR 5 install path)
  - HTTP backend's own command-shape validation (deferred to spec/19)
  - wire_version forward-compat warning (deferred to v1.1)
  - cross-backend Postgres / SaaS adapter coverage (post-LOCK)

Pre-Landing Review

Pass 1 (CRITICAL): No findings. PR 4 has no SQL changes, no LLM prompt changes, no schema breaks, no audit-trail shape changes.

Pass 2 (INFORMATIONAL): 34 findings from the review army; 8 P0 / P1 fixed inline before push, 10 P2 cleanup applied, ~12 P3 deferred as documented in spec/36 §"Reserved at lock" + new follow-up tracking.

Cross-confirmed P0 fixes applied:

1. Adv-F2 RuntimeError on closed client race → mapped to MCPRegistryUnavailable in all 5 except blocks. The adversarial reviewer cited this as "the most exploitable finding" because a reader holding self._real_client outside _client_lock against a concurrent close() would surface a raw RuntimeError bypassing the MCPRegistry contract.
2. T-F1 + Adv-F3 OPTIONS probe non-404 / 405 silent fallback → 401 now raises MCPRegistryAuthRequired; 5xx and other 4xx raise MCPRegistryUnavailable. A misconfigured token would have silently downgraded a tier-3 server to tier-1.
3. A-F1 MCPServerRef.source uses redacted URL instead of raw → switched to raw catalog_url per spec/36 line 228 (the redactor aggressively strips after :// even when no credentials present, breaking downstream navigation). Recommended operator pattern is auth_token env var, not URL-embedded credentials.
4. Adv-F4 catalog_url with query string corrupts request URLs → normalized once at __init__ (strip trailing slashes AND query string). Maintainability M4 (six .rstrip('/') sites collapsed into one).
5. S-F1 factory ValueError leaks raw URL with credentials → uses _redact_url_for_error per the PR 1 P0 redaction discipline.

P1 test additions:

• T-F4: MCPServerSpec.to_dict / from_dict public round-trip + extra-key forward-compat + required-key KeyError
• T-F2: httpx.InvalidURL strict assertion via injected MockTransport (was accepting both ValueError and MCPRegistryUnavailable; mapping not verified)
• T-F3: MUST 10 full field equality with non-default args / env / description
• T-F6 / Adv-F7: Concurrent first-call probe verification (locks D-PR4-3 design)
• T-F5: httpx.DecodingError mapping test
• T-F7: agent_scope query-param forwarding verification
• T-F8: Successful-probe-cached test
• T-F9: Factory function tests including credential-redacting ValueError and env-var auth-token read

P2 mechanical cleanup: from dataclasses import replace to top-level (M5), redundant urlencode import removed (M6), validate() 404 message rewording to honestly reflect spec ambiguity (A-F2), MCPServerRef.from_dict version="" normalization (A-F5), _ensure_probed docstring concurrent-first-call note (R-F6), README + CLAUDE.md test counts (3232 → 3307).

P3 deferred (file-tracked or naturally rolled into PR 5):

• Performance thundering-herd policy (intentional per D-PR4-3; documented + verified by test; sentinel optimization deferred)
• Command / args shell-metacharacter validation (deferred to spec/19 subprocess sandboxing)
• __all__ exports for HTTP backend (aligned with [redis] precedent; not exported)
• wire_version forward-compat warning (deferred to v1.1)
• Maintainability M1 / M2 / M3 / M7 / M8 / M9 / M10 (cross-file DRY refactors; not blocking)

Plan Completion

Plan Completion Audit: 25 / 25 DONE, 0 partial, 0 deferred, 0 unverifiable.

• atomic_agents/mcp_registry/http.py created with HTTPMCPServerRegistryBackend (read paths + write stubs)
• Constructor signature matches spec/36 with _http_client test seam
• Lazy httpx import via _get_httpx() sentinel; __init__ is side-effect-free (MUST 2)
• 5-branch capability probe (Decision 4 steps 1-5) with B-F8 non-404 4xx guard
• probe_failure_cache_s=60.0 distinct from request_timeout_s=10.0
• All 5 wire-format endpoints implemented
• Auth via Bearer header per-request; auth_token never in error messages
• URL credential redaction; MCPServerRef.source uses raw catalog URL per spec
• PR 4 static capability defaults: supports_install=False, supports_uninstall=False, supports_capability_handshake=True
• install / uninstall raise NotImplementedError (PR 5 TODO comment)
• make_http_mcp_server_registry_backend_from_url factory
• register_mcp_server_registry_backend("http", ...) at module bottom
• Factory http branch in mcp_registry/__init__.py
• [http] = ["httpx>=0.27"] in pyproject.toml
• MCPServerSpec.to_dict() / from_dict() promoted to public methods
• profile/types.py helpers delegate to class methods (backward compat preserved)
• spec/36 PR 4 amendments: §HTTP wire format, §Tier negotiation, §Capability handshake, §Per-scope filtering, §Exception surface (httpx mapping table), §Default factory env vars
• tests/test_mcp_server_registry_http_backend.py NEW (54 tests; spec target was ~31)
• Conformance suite params flipped to ["filesystem", "http"] on both fixtures
• populated_backend fixture HTTP branch with 3-server MockTransport catalog
• CHANGELOG.md [Unreleased] entry for PR 4

Documentation

README.md: Backend protocols table row for MCPServerRegistryBackend updated from Planned to 🟡 In progress (PR 4 of 5) with the new HTTP reference impl listed alongside filesystem. Test count refreshed (3,232 → 3,307).

CHANGELOG.md: [Unreleased] entry covers HTTPMCPServerRegistryBackend, tier negotiation, [http] extra (httpx>=0.27), MCPServerSpec.to_dict / from_dict promotion, test delta +75, and /ship streak extension to 11.

CLAUDE.md: Test count refreshed to 3,307 collected. Architecture diagram shows MCPServerRegistry 🟡 matching PR 4 of 5 in-progress state.

docs/spec/36-mcp-server-registry-backend.md: Four new PR 4 subsections all present (§HTTP wire format with Required/Optional field tables, §Tier negotiation, §Capability handshake, §Per-scope filtering) plus updated §Exception surface (httpx mapping table) and §Default factory (env-var documentation).

Test plan

• Full pytest suite: 3248 passed, 59 skipped, 0 failures, 0 regressions
• tests/test_mcp_server_registry_*.py: 216 passed, 9 skipped
• ruff check + ruff format --check clean on all PR 4 files
• CI matrix runs on Python 3.11 + 3.12 (workflow .github/workflows/test.yml)
• MUST 2 side-effect-free construction verified by tests asserting zero network calls during __init__
• MUST 10 bulk-vs-per-name equivalence verified with non-default field values
• Threading: test_concurrent_first_probes_both_succeed verifies D-PR4-3 design

🤖 Generated with Claude Code