Add POST support for subset snapshots to avoid URL length limits#3777
Add POST support for subset snapshots to avoid URL length limits#3777KyleAMathews merged 9 commits intomainfrom
Conversation
Adds support for HTTP POST requests when loading subset snapshots, sending subset parameters in the request body as JSON instead of URL query parameters. This avoids HTTP 414 "Request-URI Too Long" errors that occur when queries involve many parameters (e.g., `WHERE id = ANY($1)` with hundreds of IDs). Server changes: - Add POST route for /v1/shape endpoint in router.ex - Update ServeShapePlug to parse JSON body and merge with query params - Update CORS headers to include POST method Client changes: - Update TypeScript client's fetchSnapshot() to use POST with JSON body - Subset params (where, params, limit, offset, order_by) sent in body Documentation: - Update HTTP API docs to recommend POST over GET for subsets - Add deprecation notice: GET for subsets will be removed in Electric 2.0 - Update TypeScript client docs explaining POST is used automatically - Add POST endpoint to OpenAPI specification https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the
Comment |
commit: |
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3777 +/- ##
==========================================
+ Coverage 84.15% 86.56% +2.40%
==========================================
Files 44 23 -21
Lines 2834 2039 -795
Branches 535 543 +8
==========================================
- Hits 2385 1765 -620
+ Misses 447 272 -175
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
Adds a `method` option to `SubsetParams` allowing users to choose between: - `POST` (default): Sends subset params in request body as JSON - `GET` (deprecated): Sends subset params as query parameters This provides backwards compatibility while defaulting to the safer POST method. https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
- Default method is now GET (not POST) for backwards compatibility - Add `subsetMethod` option to ShapeStreamOptions to set default for all subset requests on the stream - Per-request `method` option overrides the stream-level setting - Update documentation to reflect GET as default and show how to enable POST https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
Documents the common issue where subset snapshot requests fail with 414 errors due to URL length limits, and explains how to use POST requests to avoid this limitation. https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK
- Use FetchError.fromResponse() to preserve server error messages - Add error details and logging for Elixir parse_body failures - Restore CDN infinite loop fix (clear lastSeenCursor after suppression) - Fix JSDoc to correctly state GET is the default method - Refactor fetchSnapshot to reduce duplication with #buildSubsetBody helper - Extract merge_body_params into separate functions with pattern matching Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
This PR has been released! 🚀 The following packages include changes from this PR:
Thanks for contributing to Electric! |
Tests the POST method support added in #3777 for subset snapshots: - fetchSnapshot with method: 'POST' sends params in request body - subsetMethod: 'POST' on ShapeStream defaults fetchSnapshot to POST - Per-request method option overrides stream-level subsetMethod - Parametrized WHERE clauses work correctly with POST Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
## Summary Fixes a bug where subset snapshot responses were missing the `electric-offset` header, causing POST subset requests to fail. Also adds integration tests for the POST subset snapshot feature introduced in #3777. ## Root Cause The `put_resp_headers` function in `response.ex` had separate handling for `:subset` response types. While regular responses called `put_offset_header(response)`, subset responses did not, causing the client's header validation middleware to throw `MissingHeadersError`. ## Approach **Bug Fix (1 line)** ```elixir # Added to put_resp_headers for :subset response type |> put_offset_header(response) ``` **Integration Tests** Four test cases using `fetchWrapper` to intercept requests and verify: 1. Per-request `method: 'POST'` sends subset params in body 2. Stream-level `subsetMethod: 'POST'` defaults requests to POST 3. Per-request method overrides stream-level default 4. Parametrized WHERE clauses work correctly with POST ## Key Invariants - Subset responses must include `electric-offset` header (same as regular responses) - Tests run with both long polling and SSE modes via `describe.for(fetchAndSse)` ## Non-goals - Testing server-side POST parsing (covered by backend tests) - Testing error scenarios for POST (existing error tests cover this) ## Verification ```bash cd packages/typescript-client && pnpm test ``` ## Files Changed - `packages/sync-service/lib/electric/shapes/api/response.ex` - Add `put_offset_header` to subset responses - `packages/typescript-client/test/client.test.ts` - Add 4 integration tests for POST support - `.changeset/fix-subset-offset-header.md` - Changeset for the bug fix --- Follow-up to #3777 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Summary
Adds HTTP POST support for subset snapshots, allowing clients to send subset parameters (WHERE clauses, ordering, pagination) in the request body instead of URL query parameters. This prevents
HTTP 414 Request-URI Too Longerrors that occur with complex queries or large IN lists (common in join-loaded queries with hundreds of IDs).Root Cause
When using
fetchSnapshotwith large parameter lists (e.g.,WHERE id = ANY($1)with hundreds of IDs), the encoded query parameters can exceed browser and CDN URL length limits (~2KB for many CDNs, ~8KB for most browsers), causing 414 errors. There's no way to work around this with GET requests.Approach
Backend (Elixir):
/v1/shapealongside existing GETparse_body/2plug parses JSON request bodies with proper error handling (400 for invalid JSON with details, 413 for oversized bodies)merge_body_params/2merges subset params from body with query params, body taking precedence{"subset": {...}}) and flat subset parameter formatsFrontend (TypeScript):
subsetMethodoption on ShapeStream for default method selectionmethodoption on SubsetParams for per-request override#buildSubsetBodyhelper to construct POST body with column mapper encodingKey Invariants
FetchError.fromResponse()lastSeenCursorcleared after first suppressionNon-goals
fetchSnapshot/requestSnapshot)Trade-offs
GET as default vs POST as default:
Chose GET as default for backwards compatibility. Documentation recommends POST for large queries. Will deprecate GET in Electric 2.0.
Flat vs nested body format:
Server accepts both
{"where": "..."}and{"subset": {"where": "..."}}to match flexibility of query string format.Verification
Files Changed
packages/sync-service/lib/electric/plug/router.ex- Added POST route, updated CORSpackages/sync-service/lib/electric/plug/serve_shape_plug.ex- Addedparse_body,merge_body_params, error loggingpackages/typescript-client/src/client.ts- Added POST support infetchSnapshot,#buildSubsetBodyhelper, fixed error handlingpackages/typescript-client/src/types.ts- Addedmethodto SubsetParamswebsite/docs/api/http.md- POST endpoint documentationwebsite/docs/api/clients/typescript.md- Client usage documentationwebsite/docs/guides/troubleshooting.md- 414 error troubleshooting guidewebsite/electric-api.yaml- OpenAPI spec for POST endpoint🤖 Generated with Claude Code
https://claude.ai/code/session_016ij4YWznxum2ytc8a8aQPK