Skip to content

fix(openapi_fdw): handle RFC 8288 Link header pagination#599

Merged
burmecia merged 4 commits into
supabase:mainfrom
codybrom:fix/openapi-fdw-link-header-pagination
May 19, 2026
Merged

fix(openapi_fdw): handle RFC 8288 Link header pagination#599
burmecia merged 4 commits into
supabase:mainfrom
codybrom:fix/openapi-fdw-link-header-pagination

Conversation

@codybrom
Copy link
Copy Markdown
Contributor

@codybrom codybrom commented May 5, 2026

Closes #598

Summary

  • handle_pagination only inspected JSON-body pagination paths (/links/next, /meta/pagination/next, etc.) and never read the HTTP Link response header. APIs that paginate exclusively via Link — GitHub, GitLab, and most REST services that follow RFC 5988 / RFC 8288 — never auto-paginated past the first page.
  • Thread response headers into handle_pagination and parse rel="next" entries before the existing JSON-body lookups. Cross-origin protection is preserved automatically by the existing resolve_pagination_url path.
  • Precedence: configured cursor_pathLink header → JSON-body next-URL → has_more + cursor.

Verification

Tested end-to-end against the real GitHub API:

SELECT count(*) FROM repo_pulls
WHERE owner = 'supabase' AND repo = 'wrappers' AND state = 'all';
-- → 468  (vs 30 without the fix, with page_size=30)

Debug trace shows exactly 16 chained HTTP requests, each one following the previous response's Link: <...>; rel="next" header (GitHub even rewrites the path to /repositories/558211706/pulls?...&page=N — the FDW transparently follows it).

Test plan

  • 12 new unit tests in response_tests.rs cover: GitHub-style multi-rel headers, case-insensitive header name, unquoted rel values, multi-rel values (rel="next prev"), extra params, no-match returns None, array bodies + Link header, Link beats body autodetect, configured cursor_path beats Link, fallback to body when no rel="next", multiple separate Link headers, malformed input doesn't panic.
  • All 89 pre-existing pagination tests still pass.
  • Verified end-to-end against real GitHub API (468 PRs, 16 chained requests via Link header).

Docs

  • docs/catalog/openapi.md — added Link header to the supported pagination styles list.
  • docs/catalog/wasm/index.md — bumped OpenAPI tag/source links from v0.1.4 → v0.2.0 (was stale; main openapi.md already lists 0.2.0 as current).
  • wasm-wrappers/fdw/openapi_fdw/examples/github/README.md — added a "Pagination across pages" subsection in §6 with the count(*) demo as a real-world test.

handle_pagination only read JSON-body pagination paths (/links/next,
/meta/pagination/next, etc.) and never inspected the HTTP Link response
header. APIs that paginate exclusively via Link (GitHub, GitLab, most
RFC 5988 / RFC 8288 REST services) never auto-paginated beyond the
first page, regardless of max_pages, page_size, or page_size_param.

Thread response headers into handle_pagination and parse rel="next"
entries before the existing JSON-body lookups. Cross-origin protection
is preserved by the existing resolve_pagination_url path.

Precedence (first match wins):
  1. Configured cursor_path
  2. RFC 8288 Link header (rel="next")
  3. JSON-body next-URL paths
  4. has_more flag + cursor paths

Verified end-to-end against the real GitHub API: count(*) on repo_pulls
for supabase/wrappers fetches 468 PRs across 16 chained HTTP requests,
each following the previous response's Link header.

Closes supabase#598
Copilot AI review requested due to automatic review settings May 5, 2026 20:16
@codybrom
Copy link
Copy Markdown
Contributor Author

codybrom commented May 5, 2026

Note: I did not bump the version on this PR, but can if the change is approved.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds RFC 8288 Link header support to openapi_fdw pagination so APIs that paginate exclusively via Link: <...>; rel="next" (e.g., GitHub/GitLab) can automatically fetch subsequent pages, while preserving existing pagination behaviors and cross-origin protections.

Changes:

  • Thread HTTP response headers into handle_pagination and add Link header parsing (with precedence between cursor_path / header / body heuristics).
  • Add unit tests covering a variety of Link header formats and precedence rules.
  • Update OpenAPI FDW docs and GitHub example README to document Link header pagination support.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
wasm-wrappers/fdw/openapi_fdw/src/response.rs Adds Link header parsing helpers and updates pagination precedence to consider RFC 8288 headers.
wasm-wrappers/fdw/openapi_fdw/src/response_tests.rs Adds/updates unit tests for Link header pagination and new handle_pagination signature.
wasm-wrappers/fdw/openapi_fdw/src/request.rs Passes response headers into pagination handling at request time.
wasm-wrappers/fdw/openapi_fdw/examples/github/README.md Documents GitHub pagination behavior and provides a multi-page count(*) example.
docs/catalog/wasm/index.md Updates OpenAPI Wasm wrapper tag/source links to v0.2.0.
docs/catalog/openapi.md Documents Link header pagination as a supported pagination style.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread wasm-wrappers/fdw/openapi_fdw/src/response.rs
codybrom added 2 commits May 5, 2026 15:29
split_link_entries toggled in_quotes on every `"` character without
accounting for backslash-escaped quotes inside an HTTP quoted-string
(e.g., title="a \"quoted, comma\" page"). With escapes unhandled the
quote state could desync and a comma inside a quoted parameter value
could be treated as an entry separator, causing rel="next" to be
missed.

Track quoted-pair escapes when scanning, so `\"` inside a quoted
parameter is consumed as a literal and does not flip in_quotes.

Addresses Copilot review feedback on supabase#599.
apply_headers now always adds a default user-agent when none is provided.
Six tests still asserted the old header counts and indices from before
that change was introduced.
@burmecia burmecia added the wasm label May 19, 2026
Copy link
Copy Markdown
Member

@burmecia burmecia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

@burmecia burmecia merged commit 31c86a5 into supabase:main May 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openapi_fdw: pagination breaks for APIs that use HTTP Link header (e.g. GitHub)

3 participants