Add HTTP transport, OAuth, Redis, GCS, and MCP App widgets#172
Closed
Add HTTP transport, OAuth, Redis, GCS, and MCP App widgets#172
Conversation
Part 2 of the remote MCP server split: adds Streamable HTTP transport with OAuth 2.1 (Supabase JWKS), Redis-backed state for multi-pod deployments, GCS result caching, paginated inline results, and MCP App widget templates (progress, results, session UIs). New modules: auth.py, http_config.py, routes.py, redis_utils.py, gcs_storage.py, gcs_results.py, state.py, settings.py, templates.py. Updated: models.py (_SingleSourceInput base, input_data/input_json, SingleAgentInput, optional output_path with pagination), tools.py (HTTP client, GCS upload, inline pagination, single_agent), server.py (--http flag, argparse, re-exports). Ref: feat/remote-mcp-server branch (PR #165) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
Add _with_ui helper that prepends MCP App widget JSON only in HTTP mode. In stdio mode, tools return a single human-readable TextContent instead of [widget_json, human_text], avoiding wasted context tokens. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n, schema validation - auth.py: Delete old refresh token AFTER successful Supabase refresh, not before — prevents irrecoverable session loss on API failure - routes.py: Add offset/page_size pagination to /api/results JSON endpoint — prevents unbounded payloads for large result sets - models.py: Add missing response_schema validator to SingleAgentInput — rejects invalid schemas at input validation instead of KeyError Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move _get_client, _with_ui, _submission_text, _submission_ui_json, _write_task_state, TaskNotReady, _fetch_task_result, _recommend_page_size, _build_inline_response to tool_helpers.py - Make _TOKEN_BUDGET configurable via EVERYROW_TOKEN_BUDGET env var (default 20k) - Simplify everyrow_results docstring for stdio mode - Make load_csv keyword-only Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move preview_size and token_budget to _BaseSettings (env-configurable) - Remove _TOKEN_BUDGET env var, read from settings.token_budget - Add signed_url_expiry_minutes to HttpSettings, pass to GCSResultStore - Add _blob_path() helper for GCS blob path construction - Add CachedResult dataclass to replace raw tuple in result_cache - Remove hardcoded API URL fallback in routes.py - Add docstring explaining why ServerState is a dataclass not BaseModel Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…her security issues - CRITICAL: Hardcode algorithms=["RS256"] in JWT verification to prevent algorithm confusion attacks - HIGH: Use atomic GETDEL for refresh token consumption to prevent concurrent refresh race condition - HIGH: Add 10s timeout to httpx.AsyncClient to prevent DoS via Supabase hangs - MEDIUM: Reject JWTs missing 'sub' claim instead of defaulting to "unknown" - MEDIUM: Add global rate limit (10/min) on client registration endpoint - MEDIUM: Add close() method to EveryRowAuthProvider, wire into HTTP lifespan - LOW: Sanitize Redis key parts to prevent key-injection via embedded colons - LOW: Add safety comments on unverified JWT decode calls Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Simplify everyrow_results: stdio saves to file, HTTP goes through GCS. Remove _build_inline_response, _recommend_page_size, /api/results endpoint, and all in-memory cache machinery (result_cache, CachedResult, download tokens). Make gcs_results_bucket required in HttpSettings, remove _BaseSettings, drop JSON from GCS storage (CSV only). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… scoping, error leak - Validate redirect_uri against registered client URIs in authorize() and handle_callback() - Enable DNS rebinding protection in transport security settings - Scope mcp_auth_state cookie to /auth/callback path and delete after use - Replace leaked exception messages with generic "Internal server error" on progress endpoint Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests the full JSON-RPC → FastMCP tool dispatch → tool function → response pipeline via in-memory transport. 7 mocked tests (always run) + 2 real API tests (gated by RUN_INTEGRATION_TESTS=1). Also fixes _get_client patch target in test_http_real.py to use everyrow_mcp.tools (where it's imported) instead of everyrow_mcp.tool_helpers (where it's defined). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix rate-limit INCR/EXPIRE race: use pipelined pipeline() to avoid orphan keys if process crashes between the two Redis calls - Add httpx connection limits (20 max, 10 keepalive) to bound outbound connections to Supabase under concurrent load - Extract _decode_trusted_supabase_jwt() helper with safety docstring to consolidate verify_signature=False usage and reduce copy-paste risk - Document Redis plaintext storage stance with threat-model rationale Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…er-compose RESULT_STORAGE: memory no longer exists in HttpSettings. GCS_RESULTS_BUCKET is now required for HTTP mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Supabase's OAuth 2.1 Server now handles the full authorization flow (DCR, PKCE, token exchange, refresh). The MCP server becomes a pure resource server: it only verifies Supabase JWTs via JWKS. This removes ~460 lines of custom auth code (provider, models, Redis token storage, httpx client, custom routes) and the supabase_anon_key config field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- server.py: add --no-auth flag with dedicated no-auth lifespan - tool_helpers.py: _get_client returns singleton client in no-auth mode - tools.py: GCS upload failure falls back to inline paginated results instead of returning an error - test_server.py: update test to expect inline fallback behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fakeredis is no longer needed — the remaining tests only use basic get/set/setex/delete/ping through state.* methods. A 30-line MockRedis class in conftest.py replaces the external dependency entirely. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Store full CSV in Redis (1h TTL) instead of uploading to GCS. Add
/api/results/{task_id}/download endpoint for CSV retrieval authenticated
via poll token. This eliminates the google-cloud-storage dependency and
GCP service account requirement.
Key changes:
- state.py: add store_result_csv/get_result_csv, keep poll token on
task completion (needed for download auth)
- result_store.py: replaces gcs_results.py with Redis-backed storage
- routes.py: new api_download endpoint
- config.py: remove gcs_results_bucket/signed_url_expiry_minutes
- http_config.py: remove GCS setup, wire download route
- pyproject.toml: drop google-cloud-storage dependency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Simpler and faster — spawns redis-server on port 16379 directly, no Docker overhead. Flushes between tests for isolation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test suite spawns an in-memory redis-server process instead of using fakeredis. CI needs redis-server available on PATH. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Exercise the real streamablehttp_client → JSON-RPC → tool dispatch path that was previously untested. Covers health endpoint, tool listing, and the full agent lifecycle (submit → poll → paginated preview → CSV download). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Token revocation via Redis deny-list with SHA-256 fingerprints (fail-open) - Rate-limit middleware (fixed-window per IP, 429 + Retry-After, auth mode only) - JWKS cache bounds (lifespan=300, max_cached_keys=16) to mitigate cache-bust - --no-auth guardrail requiring ALLOW_NO_AUTH=1 env var - Explicit required JWT claims (exp, sub, iss, aud) in pyjwt.decode() - asyncio.Lock on JWKS refresh to prevent thundering herd Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Supabase's OAuth 2.1 Server is not enabled on our project (returns 404), so our MCP server acts as the full authorization server, delegating user login to Supabase Google OAuth behind the scenes. This restores the previously tested EveryRowAuthProvider with PKCE flow, dynamic client registration, token exchange, refresh rotation, and revocation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Supabase JWKS serves EC keys (ES256), not RSA. The hardcoded RS256-only algorithm list caused every token to fail verification (401 on POST /mcp). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop the single-agent tool, its input model, manifest entry, and all associated tests. The multi-row everyrow_agent tool covers this use case with a single-row input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The MCP server lifespan runs per session, not per server. Closing the auth provider's httpx client on first session disconnect broke all subsequent auth flows (RuntimeError: client has been closed). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename "Copy CSV" to "Copy link" (was confusing: copied URL, not CSV data) - Remove "Copy JSON" export button (JSON already available via copy format setting) - Simplify copy button label: "Copy (N)" with format in tooltip - Add CSP connectDomains to tool meta (host may read CSP from tool, not resource) - Fix everyrow_progress: don't suggest output_path in HTTP mode - Fix everyrow_results docstring: explain HTTP vs stdio behavior - Update manifest.json description to match Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # everyrow-mcp/src/everyrow_mcp/server.py # everyrow-mcp/src/everyrow_mcp/tools.py # everyrow-mcp/tests/test_integration.py
…ent dir - Update result_store summary text to tell Claude to display the CSV download URL as a clickable link (both single-page and paginated results) - Add parent directory existence check to ResultsInput.output_path validator to fail early instead of hitting FileNotFoundError after processing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ng, renames - Atomic auth-code consumption via GETDEL to prevent replay attacks - Restrict refresh-token scope widening per OAuth 2.1 §6.3 - Rename _decode_trusted_supabase_jwt → _decode_server_issued_supabase_jwt - Add configurable JWT audience param to SupabaseTokenVerifier - Rename revoke_token → deny_token, _is_revoked → _is_denied with clearer docstrings - Rename supabase_jwt field → supabase_access_token for consistency - Add auth-flow diagram comment and client_id=sub design note - Add TestAuthProvider tests for atomic code exchange and scope handling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ensure the model never sees HTTP-only content (widget JSON, download URLs, auth tokens, HTML) in stdio mode. Introduces test_stdio_content.py with 22 unit tests + 3 real-API integration tests that assert cleanliness at every step of the submit → progress → results pipeline through the MCP protocol. Adds tool_descriptions.py with per-transport descriptions patched onto FastMCP Tool objects at startup. Fixes misleading stdio instructions: placeholder path, self-referential tip text, and mixed-mode docstrings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… middleware - _load_settings_and_redis: single helper for settings + Redis creation - _configure_mcp_auth: isolated OAuth/JWT wiring into FastMCP - _ui_csp: CSP dict built once, reused for tools and resources - _add_middleware: extracted middleware construction - configure_http_mode: single entry point that branches only where needed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo
force-pushed
the
feat/http-transport
branch
from
e56f179 to
47c4b6e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 2 of the remote MCP server split (see original PR #165, Part 1: #171).
Adds Streamable HTTP transport mode with:
auth.py,http_config.py)state.py,redis_utils.py)gcs_storage.py,gcs_results.py)routes.py)templates.py)everyrow_single_agenttool for single research queries_SingleSourceInputbase class supportinginput_csv,input_data, andinput_jsondeploy/)New modules
auth.py,http_config.py,routes.py,redis_utils.py,gcs_storage.py,gcs_results.py,state.py,settings.py,templates.pyUpdated modules
models.py,tools.py,server.py,app.py,utils.py,conftest.py,manifest.jsonTest plan
everyrow_single_agenttool registered)🤖 Generated with Claude Code