Capture binary HTTP response bodies, and address envelope fields directly - #304
Merged
Conversation
df.http() and df.http_multipart() read every response with `.text()`, which does a lossy UTF-8 conversion. Any non-textual body — an image, a PDF, an MP3 — was silently replaced with U+FFFD characters, so a binary response could not be stored or forwarded. The bytes were gone by the time the envelope was built. Responses are now classified by Content-Type. A textual body is carried through unchanged; a non-textual one is base64-encoded and the envelope records which happened in a new `encoding` field (`text` or `base64`). A body counts as text when Content-Type is absent, `text/*`, a `+json`, `+xml` or `+yaml` suffix type, or one of the common textual `application/*` types. Everything else is treated as binary, which is the safe default: mislabelling text as base64 is recoverable, mislabelling binary as text destroys it. The classification and envelope construction moved into a new activities::http_response module so both activities share one definition rather than maintaining parallel copies. Also makes HTTP envelope fields addressable with dot notation. Previously dot notation only resolved against SQL results, which carry a `rows` array, so reading a status or body required an intervening SQL node just to destructure JSON. `$resp.status`, `$resp.body`, `$resp.ok` and `$resp.encoding` now resolve directly. `rows` still takes precedence, so SQL results behave exactly as before. Adds examples/audio-roundtrip/, which exercises the whole path end to end: it sends text to Azure OpenAI text-to-speech, pipes the returned MP3 straight into a Whisper transcription upload with no intermediate table, and verifies the transcript against the original phrase. Provisioning and teardown scripts are included so the example can be run and cleaned up without leaving billable resources behind.
pinodeca
added a commit
that referenced
this pull request
Jul 30, 2026
* Fail loudly on missing envelope fields and sniff unlabelled bodies Two follow-ups to the binary response work in #304. `$name.field` on an HTTP envelope left the pattern in place when the field was missing. In SQL that is deliberate -- PostgreSQL reports it with its own diagnostics -- but a URL, a header or a multipart field has no such parser, so a typo travelled over the wire verbatim: `Bearer $auth.token` would be sent as a literal and the request would fail somewhere far less obvious. Raw contexts now fail and list the available fields; the SQL and null-safe paths are unchanged. Body classification no longer trusts `Content-Type` alone. A declared textual type is still decoded by reqwest so the `charset` parameter is honoured, but an unrecognised or absent type is decided by its bytes: valid UTF-8 without NUL is text, anything else is base64. This stops an untyped binary download being mangled by a UTF-8 decode, and stops textual types no allowlist will ever cover (`application/jwt`, `application/x-ndjson`) being base64-encoded for no reason. NUL is excluded because PostgreSQL's `text` cannot hold it, so a mostly-zero body would otherwise fail on the way into the result row rather than at the decision that caused it. Also split `Content-Type` on a comma before classifying, so a proxy folding duplicate headers into `text/html, application/octet-stream` is handled, and name the unit in the error-body preview rather than reporting a base64 length as bytes. * Fix HTTP test failure diagnostic * Document outstanding problems with http and http_multipart --------- Co-authored-by: GitHub Copilot <copilot@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #303. That PR made
df.http_multipart()able to send real payloads; this one makesdf.http()able to receive them, and removes the ceremony around reading a response.Binary response bodies were being destroyed
Both HTTP activities read the response with
reqwest's.text(), which performs a lossy UTF-8 conversion. Any non-textual body — an image, a PDF, an MP3 — came back with every invalid sequence replaced by U+FFFD. The bytes were already gone by the time the envelope was built, so no amount of downstream casting could recover them. A binary response simply could not be stored or forwarded.Responses are now classified by
Content-Type:The envelope records which happened in a new
encodingfield. A body counts as text whenContent-Typeis absent,text/*, a+json/+xml/+yamlsuffix type, or one of the common textualapplication/*types. Everything else is treated as binary.That default is deliberate: mislabelling text as base64 is recoverable by the user, whereas mislabelling binary as text destroys the data irreversibly. When in doubt, preserve.
Classification and envelope construction moved into a new
activities::http_responsemodule so both activities share a single definition instead of maintaining parallel copies.Envelope fields are now addressable directly
Dot notation previously resolved only against SQL results, which carry a
rowsarray. Reading anything out of an HTTP response therefore required an intervening SQL node purely to destructure JSON:$resp.status,$resp.body,$resp.okand$resp.encodingresolve directly.rowsstill takes precedence, so SQL results behave exactly as before — this is additive, not a change to existing resolution.Worked example
examples/audio-roundtrip/exercises the whole path: it sends text to Azure OpenAI text-to-speech, pipes the returned MP3 straight into a Whisper transcription upload with no intermediate table, then verifies the transcript against the original phrase.It depends on both halves of this work — the MP3 survives only because of binary capture, and it reaches the upload only because
data_b64accepts$speech.body(#303). The form also carries apromptpart whose base64 wraps across lines, which exercises #303's decode fix.Provisioning and teardown scripts are included; the teardown purges soft-deleted Azure OpenAI accounts, which otherwise keep holding quota.
Testing
tests/e2e/sql/06_http_and_ssrf.sql: a binary response captured and re-uploaded losslessly, envelope field access, androwsprecedenceencoding = base64Locally:
cargo fmt --checkclean, clippy adds no new warnings (4 pre-existing insrc/lib.rsremain), unit tests 229 passed / 0 failed. I did not complete a local E2E run on this branch — relying on CI for that.Note on the diff
Docs examples that #303 had to phrase defensively (using a SQL-sourced reference and
($resp::jsonb->>'ok')) revert here to the direct form, since dot notation now exists to support them.