Skip to content

Add HTTP transport with OAuth, Redis state, and deploy config#183

Merged
RafaelPo merged 39 commits intomainfrom
feat/http-transport-clean
Feb 24, 2026
Merged

Add HTTP transport with OAuth, Redis state, and deploy config#183
RafaelPo merged 39 commits intomainfrom
feat/http-transport-clean

Conversation

@RafaelPo
Copy link
Contributor

@RafaelPo RafaelPo commented Feb 21, 2026

Summary

Adds HTTP (Streamable HTTP) transport mode to the everyrow MCP server, enabling remote deployment with OAuth 2.1 authentication, Redis-backed state, and a Docker deploy stack.

New modules

  • auth.py — OAuth 2.1 provider (3-leg PKCE: Google → Supabase → MCP Server), JWKS verification, refresh token rotation
  • http_config.py — HTTP mode setup: OAuth wiring, custom routes, CSP, middleware
  • routes.py — REST endpoints for progress polling and CSV download (bypasses MCP protocol for widgets)
  • middleware.py — Redis-based fixed-window rate limiter per client IP
  • result_store.py — Redis-backed result caching with pagination and CSV download URLs
  • redis_utils.py — Redis client factory with Sentinel support
  • config.pyHttpSettings, DevHttpSettings, StdioSettings with _BaseHttpSettings base class
  • state.pyServerState with Transport enum, RedisStore, validate_assignment=True
  • tool_helpers.py — Transport-aware helpers (_get_client, create_tool_response, _write_task_state)
  • tool_descriptions.py — Per-transport tool description patching
  • templates.py — MCP App widget HTML (progress, results, session)

Other changes

  • everyrow_single_agent — New MCP tool for single-question research (no CSV required)
  • input_data / input_json — All submit tools now accept inline data (not just input_csv)
  • create_tool_response() — Deduplicated submission return pattern across all 6 submit tools
  • Deploy stackDockerfile, docker-compose.yaml, .env.example
  • Docs/README — Clinical trials case study edits, classification guide CSV fix, README updates

Stdio impact

  • No breaking changes — all existing stdio tool signatures and behavior preserved
  • Field descriptions restored where shortened (equivalence_relation, merge fields, output_path)
  • New hard dependencies (redis, httpx, PyJWT, pydantic-settings) loaded at module level in both modes

Architecture decisions

  • Settings are lazy singletons (@lru_cache getters), not on ServerState
  • ServerState has validate_assignment=True — rejects invalid transport values
  • Dead fields removed: auth_provider (write-only), dev_mode, settings property
  • is_http uses closed-form == Transport.HTTP check
  • Middleware ordering: logging wraps rate limiting (sees all requests including 429s)
  • _submission_ui_json() only called in HTTP mode (avoids wasted Redis writes in stdio)

Test plan

  • 192 unit tests pass (uv run pytest tests/ -v)
  • 7 integration tests pass with real API (RUN_INTEGRATION_TESTS=1)
  • Ruff lint + format clean
  • Pyright typecheck clean
  • Manifest sync test passes (docstrings match manifest.json)
  • Manual stdio smoke test (Claude Desktop)
  • Manual HTTP smoke test (--http --no-auth)

🤖 Generated with Claude Code

RafaelPo and others added 2 commits February 21, 2026 20:15
- http_config.py: HTTP mode setup (auth + no-auth) with OAuth, CSP, middleware
- tool_helpers.py: transport-aware helpers (_with_ui, _submission_ui_json,
  _get_client with per-request JWT, _fetch_task_result)
- tool_descriptions.py: transport-specific tool descriptions patched at startup
- routes.py: REST endpoints for progress polling and CSV download
- result_store.py: Redis-backed result caching with pagination
- middleware.py: rate limiting middleware
- tools.py: all tools use _with_ui for widget JSON, token storage in Redis
- models.py: add input_data/input_json alternatives to input_csv
- app.py: separate lifespans for stdio/http/no-auth-http
- server.py: CLI args for --http, --no-auth, --host, --port
- deploy/: Dockerfile, docker-compose, env example

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@RafaelPo RafaelPo changed the title Feat/http transport clean Add HTTP transport layer for remote MCP server Feb 21, 2026
RafaelPo and others added 6 commits February 21, 2026 20:48
… http_config

- Remove `settings` field from ServerState; add `everyrow_api_url` and
  `preview_size` as direct fields with defaults
- Add lazy `settings` property that returns the right settings via
  `_get_http_settings()` / `_get_dev_http_settings()` / `_get_stdio_settings()`
- Extract `_BaseHttpSettings` base class for shared Redis and HTTP fields
- Remove eager module-level settings singletons from config.py
- Inline Redis creation in `configure_http_mode`, extract `_patch_tool_csp`
- Update auth.py to use `_get_http_settings()` instead of `http_settings`

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ering

- Fix EveryRowAuthProvider constructor to pass (redis, token_verifier)
  instead of raw settings fields
- Use Transport.HTTP/STDIO enum values instead of string literals
- Remove duplicate results.html/session.html resource registrations
  from app.py (only registered in http_config.py for HTTP mode)
- Swap middleware order so logging wraps rate limiting (sees all requests)
- Merge identical progress descriptions in tool_descriptions.py
- Add override notes above everyrow_progress and everyrow_results
- Restore manifest.json everyrow_results description for stdio accuracy

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each tool return site now builds the base TextContent list and
conditionally prepends the widget JSON only in HTTP mode. This
avoids eagerly computing _submission_ui_json (which stores tokens
in Redis) in stdio mode where the result was discarded.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 6 submission tools now call create_tool_response() instead of
repeating the stdio/http branching inline. The helper lives in
tool_helpers.py alongside _submission_text and _submission_ui_json.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove dead fields: auth_provider (write-only), dev_mode (only read
  by dead settings property), settings property (never called)
- Remove dead _build_csv_url from result_store.py
- Add validate_assignment=True to catch invalid transport values
- Fix is_http to use == Transport.HTTP (closed-form check)
- Fix tests to use Transport.HTTP instead of "streamable-http" string
- Use spec=AuthenticatedClient on mock clients for Pydantic compat
- Add docstring note about singleton client in no-auth HTTP mode (C1)
- Restore field descriptions shortened in models.py (equivalence_relation,
  merge fields) and fix misleading output_path description for stdio

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The everyrow_client fixture was setting the old app._client attribute
which no longer exists after the refactor. Use state.client instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@RafaelPo RafaelPo changed the title Add HTTP transport layer for remote MCP server Add HTTP transport with OAuth, Redis state, and deploy config Feb 21, 2026
RafaelPo and others added 2 commits February 21, 2026 22:13
Export assert_stdio_clean from test_stdio_content (rename private helpers
to public) and apply it at every submit/progress/results step in
test_integration.py. Also assert len(result)==1 and normalize to
result[0].text, fixing a stale comment about widget JSON.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the global state.client singleton with a ClientFactory yielded
from lifespans and accessed via ctx.request_context.lifespan_context.
Tools now receive the client through FastMCP's Context parameter
injection instead of reading mutable global state, eliminating 22
patch.object(state, "client", ...) patterns in tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 2 commits February 21, 2026 23:23
Replace bare Context with Context[ServerSession, SessionContext] alias,
giving type checkers visibility into lifespan_context fields and making
the context contract explicit via a dataclass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace make_singleton_client_factory with lambda: client and inline
make_http_auth_client_factory into _http_lifespan. Removes indirection
that added no value over the inline forms.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 3 commits February 22, 2026 00:06
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The TOKEN_BUDGET setting (default 20k) was defined but unused. This wires
it in: after slicing a result page, clamp_page_to_budget binary-searches
for the largest subset that fits within the budget (~4 chars/token
heuristic). Applied in all three result paths (Redis cache hit, Redis
store, inline fallback). The progress completed message now recommends
page_size=preview_size and notes the auto-adjustment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…actories

ServerState no longer stores everyrow_api_url, preview_size, or
token_budget as flat fields.  A new `settings` property dispatches to
the correct @lru_cache-d factory (_get_stdio_settings,
_get_dev_http_settings, or _get_http_settings) based on transport mode.

Extracted _CommonSettings base class in config.py so preview_size,
token_budget, and everyrow_api_url are defined once and inherited by
both StdioSettings and _BaseHttpSettings.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@RafaelPo RafaelPo requested a review from rgambee February 22, 2026 12:21
@RafaelPo
Copy link
Contributor Author

@claude code review

RafaelPo and others added 2 commits February 22, 2026 18:03
- Convert TaskState from stdlib dataclass to Pydantic BaseModel with
  PrivateAttr, computed_field properties, and model_dump serialisation
- Add write_initial_task_state for submission tools, consolidate file
  writing into _write_task_state_file
- Move progress message building into TaskState.progress_message
- Replace ui_dict with model_dump(exclude=_UI_EXCLUDE) at call sites
- Remove dead inline-results fallback from everyrow_results (Redis is
  always available in HTTP mode)
- Extract _register_widgets and _register_routes in http_config
- Clean up dead re-exports in server.py
- Use dedent for multiline tool response strings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… harden edges

- Move Redis client creation and state.mcp_server_url assignment to server.py
  (before configure_http_mode) so state is fully initialised before consumers read it
- Pass redis_client explicitly to configure_http_mode, fixing RedisStore/Redis
  type mismatch in EveryRowAuthProvider and RateLimitMiddleware
- Add missing lifespan assignment in auth branch of configure_http_mode
- Add ctx argument to all tool calls in test_http_real.py (was missing after
  the lifespan context refactor)
- Add everyrow_single_agent to expected tools in test_http_transport.py
- Add meta={"ui": {"resourceUri": ...}} to everyrow_progress tool decorator
- Guard state.store.ping() with None check in both HTTP lifespans
- Reject FAILED/REVOKED tasks in _fetch_task_result before fetching results
- Use get_tool() instead of _tools.get() in _patch_tool_csp
- Use `is not None` in load_csv source counting
- Extract override_state() context manager in conftest.py, deduplicate 4 test fixtures
- Inline _cors_preflight, add return type annotations to route handlers
- Remove dead _validate_redis model validator (lifespan ping catches misconfigured Redis)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 3 commits February 22, 2026 18:56
- Remove `assert progress["total"] > 0` from screen integration test
  (screen tasks don't report row-level totals)
- Rename _get_http_settings → get_http_settings, _get_dev_http_settings →
  get_dev_http_settings, _get_stdio_settings → get_stdio_settings (drop
  leading underscore — these are used across modules)
- Extract local `transport` variable in server.py main()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… external URLs

- Fix state.transport never being set (local var instead of state assignment),
  which caused HTTP mode to run as stdio and exit immediately
- Remove progress bar MCP App widget (PROGRESS_HTML, ui://everyrow/progress.html
  resource, widget JSON from everyrow_progress responses)
- Replace "Copy link" button with "Download CSV" button using app.openLink()
- Use app.openLink() for all session/download links in results and session widgets
  so they trigger the host's "Open external link" confirmation dialog
- Fix research popover hover: mouseenter → mouseover (mouseenter doesn't bubble)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…collapse

- Add linkify() helper to auto-link URLs in cell text and research popovers
- Move Download CSV link next to session link as styled anchor
- Fix expand/collapse using innerHTML with linkify instead of textContent

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@RafaelPo RafaelPo force-pushed the feat/http-transport-clean branch 2 times, most recently from 8488aeb to 56529b4 Compare February 22, 2026 20:57
…de_settings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 4 commits February 22, 2026 22:24
request.client.host returns the proxy IP (e.g. Cloudflare tunnel),
causing all users to share a single rate limit bucket. Now checks
CF-Connecting-IP and X-Forwarded-For before falling back.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…_size in hints

- try_cached_result returns None (triggering API re-fetch) when CSV is
  expired or unreadable, instead of returning an empty preview
- Next-page hints use the user's original page_size, not the clamped
  effective size, so the server re-clamps independently per page
- Bump default preview_size from 50 to 100 (token budget auto-clamps)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hide fields that don't apply in each mode:
- HTTP: remove input_csv/left_csv/right_csv (server can't read client filesystem)
- HTTP: remove output_path from everyrow_results
- Stdio: remove offset/page_size from everyrow_results, make output_path required

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ata field

Collapse 3 mutually exclusive input fields (input_csv, input_data, input_json)
down to 2 (input_csv, data) where data accepts str|list[dict] with auto-detection.
MergeInput goes from 6 input fields to 4 (left_data/right_data replace
left_input_data/left_input_json/right_input_data/right_input_json).

SingleAgentInput.input_data (dict context, not tabular) is unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 2 commits February 23, 2026 21:27
…riptions.py

Replace the single ResultsInput model with StdioResultsInput (output_path
required) and HttpResultsInput (output_path optional, pagination fields).
Each mode gets its own function — everyrow_results_stdio and
everyrow_results_http — eliminating runtime schema patching and description
overrides. The stdio variant is registered by default; server.py re-registers
the HTTP variant when --http is used.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove ARG002 from ruff ignore list. Fix violations: add noqa for
auth.py protocol override, remove redundant everyrow_client fixture
params from test_integration.py (already pulled in via real_ctx).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When the poll token TTL (24h) elapses, get_poll_token() returns None
and _get_csv_url() silently interpolated it into the URL, producing a
broken download link that 403s. Now _get_csv_url() returns None on
expiry, and both callers fall back to a fresh API fetch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts to integrate forecast and cancel tools from main with
the split results functions on this branch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 2 commits February 24, 2026 09:44
Use Redis pipeline with EXPIRE NX to avoid race where TTL is reset on
every request, and increase default preview_size to 1000 rows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ForecastInput now inherits from _SingleSourceInput, gaining the data
field so forecast works in HTTP mode where file paths are unavailable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 3 commits February 24, 2026 09:54
Instead of returning a misleading error when try_store_result fails,
return the results inline as a preview with a summary message.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ToolManager.add_tool() is a no-op for existing names, so the HTTP
override of everyrow_results was silently ignored, leaving the stdio
schema (with required output_path) exposed in HTTP mode.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Verifies that remove_tool + re-register replaces the stdio schema
with HttpResultsInput, preventing regression of the silent no-op bug.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- get_client_ip returns None instead of "unknown" when IP cannot be
  determined; rate limiter skips these requests instead of grouping
  them into a shared bucket.
- HttpResultsInput no longer validates that output_path's parent
  directory exists on the server, since in HTTP mode the path comes
  from a remote client's filesystem.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RafaelPo and others added 2 commits February 24, 2026 10:23
get_client_ip now returns None instead of "unknown", but auth's
_client_ip passed it directly to build_key which calls .replace().
Fall back to "unknown" in auth — we want to rate limit unidentified
OAuth clients, unlike the middleware where we skip them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pandas to_dict(orient="records") preserves float('nan') which json.dumps
serializes as NaN — invalid JSON that breaks widget rendering in the client.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment on lines +134 to +138
else:
summary = (
f"Results: showing rows {offset + 1}-{min(offset + page_size, total)} "
f"of {total} (final page)."
)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: When offset equals total rows, the pagination summary message becomes nonsensical, e.g., 'showing rows 101-100 of 100'.
Severity: MEDIUM

Suggested Fix

Add a conditional check at the beginning of the else block in _build_result_response. If offset >= total, return a specific message indicating that there are no more rows to show, rather than attempting to calculate and display a row range.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: everyrow-mcp/src/everyrow_mcp/result_store.py#L134-L138

Potential issue: When a user requests a page with an `offset` equal to or greater than
the `total` number of rows, the `_build_result_response` function generates a
nonsensical summary message. For example, if `offset` and `total` are both 100, the
message becomes "showing rows 101-100 of 100 (final page)". While the function correctly
returns an empty set of rows, the accompanying text is confusing for the user. This can
occur because the `offset` parameter in `HttpResultsInput` is only validated to be
non-negative, allowing users to paginate past the end of the dataset.

@RafaelPo RafaelPo merged commit 450f211 into main Feb 24, 2026
8 checks passed
@RafaelPo RafaelPo deleted the feat/http-transport-clean branch February 24, 2026 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants