Skip to content

Add POST support for subset snapshots to avoid URL length limits#3777

Merged
KyleAMathews merged 9 commits intomainfrom
claude/load-subsets-post-get-eOPIF
Jan 27, 2026
Merged

Add POST support for subset snapshots to avoid URL length limits#3777
KyleAMathews merged 9 commits intomainfrom
claude/load-subsets-post-get-eOPIF

Conversation

@KyleAMathews
Copy link
Copy Markdown
Contributor

@KyleAMathews KyleAMathews commented Jan 27, 2026

Summary

Adds HTTP POST support for subset snapshots, allowing clients to send subset parameters (WHERE clauses, ordering, pagination) in the request body instead of URL query parameters. This prevents HTTP 414 Request-URI Too Long errors that occur with complex queries or large IN lists (common in join-loaded queries with hundreds of IDs).

Root Cause

When using fetchSnapshot with large parameter lists (e.g., WHERE id = ANY($1) with hundreds of IDs), the encoded query parameters can exceed browser and CDN URL length limits (~2KB for many CDNs, ~8KB for most browsers), causing 414 errors. There's no way to work around this with GET requests.

Approach

Backend (Elixir):

  • Added POST route for /v1/shape alongside existing GET
  • New parse_body/2 plug parses JSON request bodies with proper error handling (400 for invalid JSON with details, 413 for oversized bodies)
  • merge_body_params/2 merges subset params from body with query params, body taking precedence
  • Supports both nested ({"subset": {...}}) and flat subset parameter formats

Frontend (TypeScript):

  • Added subsetMethod option on ShapeStream for default method selection
  • Added method option on SubsetParams for per-request override
  • Defaults to GET for backwards compatibility
  • Extracted #buildSubsetBody helper to construct POST body with column mapper encoding

Key Invariants

  1. Body params override query params when both present (allows query string defaults with body overrides)
  2. Column mapper encoding applies to both GET and POST paths
  3. Server error messages are preserved and surfaced to client via FetchError.fromResponse()
  4. CDN infinite loop fix preserved: lastSeenCursor cleared after first suppression

Non-goals

  • Changing default to POST (breaking change, deferred to Electric 2.0)
  • Supporting POST for regular shape streaming (only affects fetchSnapshot/requestSnapshot)

Trade-offs

GET as default vs POST as default:
Chose GET as default for backwards compatibility. Documentation recommends POST for large queries. Will deprecate GET in Electric 2.0.

Flat vs nested body format:
Server accepts both {"where": "..."} and {"subset": {"where": "..."}} to match flexibility of query string format.

Verification

# TypeScript client tests
cd packages/typescript-client && pnpm test

# Type checking
pnpm tsc --noEmit

Files Changed

  • packages/sync-service/lib/electric/plug/router.ex - Added POST route, updated CORS
  • packages/sync-service/lib/electric/plug/serve_shape_plug.ex - Added parse_body, merge_body_params, error logging
  • packages/typescript-client/src/client.ts - Added POST support in fetchSnapshot, #buildSubsetBody helper, fixed error handling
  • packages/typescript-client/src/types.ts - Added method to SubsetParams
  • website/docs/api/http.md - POST endpoint documentation
  • website/docs/api/clients/typescript.md - Client usage documentation
  • website/docs/guides/troubleshooting.md - 414 error troubleshooting guide
  • website/electric-api.yaml - OpenAPI spec for POST endpoint

🤖 Generated with Claude Code

https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK

Adds support for HTTP POST requests when loading subset snapshots, sending
subset parameters in the request body as JSON instead of URL query parameters.
This avoids HTTP 414 "Request-URI Too Long" errors that occur when queries
involve many parameters (e.g., `WHERE id = ANY($1)` with hundreds of IDs).

Server changes:
- Add POST route for /v1/shape endpoint in router.ex
- Update ServeShapePlug to parse JSON body and merge with query params
- Update CORS headers to include POST method

Client changes:
- Update TypeScript client's fetchSnapshot() to use POST with JSON body
- Subset params (where, params, limit, offset, order_by) sent in body

Documentation:
- Update HTTP API docs to recommend POST over GET for subsets
- Add deprecation notice: GET for subsets will be removed in Electric 2.0
- Update TypeScript client docs explaining POST is used automatically
- Add POST endpoint to OpenAPI specification

https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Jan 27, 2026

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

  • 🔍 Trigger a full review

Comment @coderabbitai help to get the list of available commands and usage tips.

@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented Jan 27, 2026

Open in StackBlitz

npm i https://pkg.pr.new/@electric-sql/react@3777
npm i https://pkg.pr.new/@electric-sql/client@3777
npm i https://pkg.pr.new/@electric-sql/y-electric@3777

commit: 97fd2cb

@netlify
Copy link
Copy Markdown

netlify Bot commented Jan 27, 2026

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit c373fc4
🔍 Latest deploy log https://app.netlify.com/projects/electric-next/deploys/6978ee1fcde4780008e41e1d
😎 Deploy Preview https://deploy-preview-3777--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 27, 2026

Codecov Report

❌ Patch coverage is 31.25000% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.56%. Comparing base (571ed07) to head (97fd2cb).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
packages/typescript-client/src/client.ts 31.25% 22 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3777      +/-   ##
==========================================
+ Coverage   84.15%   86.56%   +2.40%     
==========================================
  Files          44       23      -21     
  Lines        2834     2039     -795     
  Branches      535      543       +8     
==========================================
- Hits         2385     1765     -620     
+ Misses        447      272     -175     
  Partials        2        2              
Flag Coverage Δ
elixir ?
elixir-client ?
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/start 82.83% <ø> (ø)
packages/typescript-client 92.12% <31.25%> (-1.36%) ⬇️
packages/y-electric 56.05% <ø> (ø)
typescript 86.56% <31.25%> (-0.81%) ⬇️
unit-tests 86.56% <31.25%> (+2.40%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blacksmith-sh

This comment has been minimized.

claude and others added 8 commits January 27, 2026 16:29
Adds a `method` option to `SubsetParams` allowing users to choose between:
- `POST` (default): Sends subset params in request body as JSON
- `GET` (deprecated): Sends subset params as query parameters

This provides backwards compatibility while defaulting to the safer POST method.

https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
- Default method is now GET (not POST) for backwards compatibility
- Add `subsetMethod` option to ShapeStreamOptions to set default for all
  subset requests on the stream
- Per-request `method` option overrides the stream-level setting
- Update documentation to reflect GET as default and show how to enable POST

https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
Documents the common issue where subset snapshot requests fail with
414 errors due to URL length limits, and explains how to use POST
requests to avoid this limitation.

https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
- Use FetchError.fromResponse() to preserve server error messages
- Add error details and logging for Elixir parse_body failures
- Restore CDN infinite loop fix (clear lastSeenCursor after suppression)
- Fix JSDoc to correctly state GET is the default method
- Refactor fetchSnapshot to reduce duplication with #buildSubsetBody helper
- Extract merge_body_params into separate functions with pattern matching

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@robacourt robacourt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. LGTM 👍

@KyleAMathews KyleAMathews merged commit 3f257aa into main Jan 27, 2026
52 of 53 checks passed
@KyleAMathews KyleAMathews deleted the claude/load-subsets-post-get-eOPIF branch January 27, 2026 17:19
@github-actions
Copy link
Copy Markdown
Contributor

This PR has been released! 🚀

The following packages include changes from this PR:

  • @core/sync-service@1.4.0
  • @electric-sql/client@1.5.0

Thanks for contributing to Electric!

KyleAMathews added a commit that referenced this pull request Jan 27, 2026
Tests the POST method support added in #3777 for subset snapshots:
- fetchSnapshot with method: 'POST' sends params in request body
- subsetMethod: 'POST' on ShapeStream defaults fetchSnapshot to POST
- Per-request method option overrides stream-level subsetMethod
- Parametrized WHERE clauses work correctly with POST

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
KyleAMathews added a commit that referenced this pull request Jan 28, 2026
## Summary

Fixes a bug where subset snapshot responses were missing the
`electric-offset` header, causing POST subset requests to fail. Also
adds integration tests for the POST subset snapshot feature introduced
in #3777.

## Root Cause

The `put_resp_headers` function in `response.ex` had separate handling
for `:subset` response types. While regular responses called
`put_offset_header(response)`, subset responses did not, causing the
client's header validation middleware to throw `MissingHeadersError`.

## Approach

**Bug Fix (1 line)**
```elixir
# Added to put_resp_headers for :subset response type
|> put_offset_header(response)
```

**Integration Tests**
Four test cases using `fetchWrapper` to intercept requests and verify:
1. Per-request `method: 'POST'` sends subset params in body
2. Stream-level `subsetMethod: 'POST'` defaults requests to POST  
3. Per-request method overrides stream-level default
4. Parametrized WHERE clauses work correctly with POST

## Key Invariants

- Subset responses must include `electric-offset` header (same as
regular responses)
- Tests run with both long polling and SSE modes via
`describe.for(fetchAndSse)`

## Non-goals

- Testing server-side POST parsing (covered by backend tests)
- Testing error scenarios for POST (existing error tests cover this)

## Verification

```bash
cd packages/typescript-client && pnpm test
```

## Files Changed

- `packages/sync-service/lib/electric/shapes/api/response.ex` - Add
`put_offset_header` to subset responses
- `packages/typescript-client/test/client.test.ts` - Add 4 integration
tests for POST support
- `.changeset/fix-subset-offset-header.md` - Changeset for the bug fix

---

Follow-up to #3777

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants