Fix HTTP envelope substitution and body classification - #310
Merged
Conversation
added 3 commits
July 30, 2026 14:56
Two follow-ups to the binary response work in #304. `$name.field` on an HTTP envelope left the pattern in place when the field was missing. In SQL that is deliberate -- PostgreSQL reports it with its own diagnostics -- but a URL, a header or a multipart field has no such parser, so a typo travelled over the wire verbatim: `Bearer $auth.token` would be sent as a literal and the request would fail somewhere far less obvious. Raw contexts now fail and list the available fields; the SQL and null-safe paths are unchanged. Body classification no longer trusts `Content-Type` alone. A declared textual type is still decoded by reqwest so the `charset` parameter is honoured, but an unrecognised or absent type is decided by its bytes: valid UTF-8 without NUL is text, anything else is base64. This stops an untyped binary download being mangled by a UTF-8 decode, and stops textual types no allowlist will ever cover (`application/jwt`, `application/x-ndjson`) being base64-encoded for no reason. NUL is excluded because PostgreSQL's `text` cannot hold it, so a mostly-zero body would otherwise fail on the way into the result row rather than at the decision that caused it. Also split `Content-Type` on a comma before classifying, so a proxy folding duplicate headers into `text/html, application/octet-stream` is handled, and name the unit in the error-body preview rather than reporting a base64 length as bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two follow-ups to the binary response work in #304, plus a written record of
what is still wrong with the HTTP activities.
Changes
Fail loudly on missing envelope fields
$name.fieldon an HTTP envelope previously left the pattern in place when thefield did not exist. In SQL that is deliberate — PostgreSQL reports it with its
own diagnostics — but a URL, a header or a multipart field has no such parser,
so a typo travelled over the wire verbatim:
Bearer $auth.tokenwas sent as aliteral and the request failed somewhere far less obvious. Raw contexts now fail
and list the available fields; the SQL and null-safe paths are unchanged.
Sniff unlabelled bodies
Body classification no longer trusts
Content-Typealone. A declared textualtype is still decoded by reqwest so the
charsetparameter is honoured, but anunrecognised or absent type is decided by its bytes: valid UTF-8 without NUL is
text, anything else is base64. This stops an untyped binary download being
mangled by a UTF-8 decode, and stops textual types no allowlist will ever cover
(
application/jwt,application/x-ndjson) being base64-encoded for no reason.NUL is excluded because PostgreSQL's
textcannot hold it, so a mostly-zerobody would otherwise fail on the way into the result row rather than at the
decision that caused it.
Also splits
Content-Typeon a comma before classifying, so a proxy foldingduplicate headers into
text/html, application/octet-streamis handled, andnames the unit in the error-body preview rather than reporting a base64 length
as bytes.
Fix HTTP test failure diagnostic
tests/e2e/sql/06_http_and_ssrf.sqlreported the wrong value on failure.Document outstanding problems
docs/http_problems.mdrecords 18 open issues found while reviewing the HTTPwork — unbounded payloads, unvalidated JSONB inputs, a missing upgrade grant for
df.http_multipart(), panics on malformed node config, unvalidatedContent-Dispositionfilenames, tests that pass without asserting, anddocumentation that misdescribes 5xx/retry behaviour, among others. Each entry
has evidence and a proposed fix. No behaviour change; the document is a work
queue, not a claim that any of it is fixed.
Testing
cargo fmt -p pg_durable -- --checkcargo clippy --features pg17./scripts/test-unit.sh./scripts/test-e2e-local.sh