Skip to content

Capture binary HTTP response bodies, and address envelope fields directly - #304

Merged
pinodeca merged 1 commit into
mainfrom
feat/http-binary-responses
Jul 28, 2026
Merged

Capture binary HTTP response bodies, and address envelope fields directly#304
pinodeca merged 1 commit into
mainfrom
feat/http-binary-responses

Conversation

@pinodeca

Copy link
Copy Markdown
Contributor

Follow-up to #303. That PR made df.http_multipart() able to send real payloads; this one makes df.http() able to receive them, and removes the ceremony around reading a response.

Binary response bodies were being destroyed

Both HTTP activities read the response with reqwest's .text(), which performs a lossy UTF-8 conversion. Any non-textual body — an image, a PDF, an MP3 — came back with every invalid sequence replaced by U+FFFD. The bytes were already gone by the time the envelope was built, so no amount of downstream casting could recover them. A binary response simply could not be stored or forwarded.

Responses are now classified by Content-Type:

  • text — carried through unchanged, exactly as before
  • base64 — the raw bytes, base64-encoded

The envelope records which happened in a new encoding field. A body counts as text when Content-Type is absent, text/*, a +json / +xml / +yaml suffix type, or one of the common textual application/* types. Everything else is treated as binary.

That default is deliberate: mislabelling text as base64 is recoverable by the user, whereas mislabelling binary as text destroys the data irreversibly. When in doubt, preserve.

Classification and envelope construction moved into a new activities::http_response module so both activities share a single definition instead of maintaining parallel copies.

Envelope fields are now addressable directly

Dot notation previously resolved only against SQL results, which carry a rows array. Reading anything out of an HTTP response therefore required an intervening SQL node purely to destructure JSON:

-- before
df.http(url) |=> 'resp'
~> 'SELECT ($resp::jsonb->>''ok'')::boolean AS ok' |=> 'checked'
~> df.if('$checked.ok', ...)

-- now
df.http(url) |=> 'resp'
~> df.if('$resp.ok', ...)

$resp.status, $resp.body, $resp.ok and $resp.encoding resolve directly. rows still takes precedence, so SQL results behave exactly as before — this is additive, not a change to existing resolution.

Worked example

examples/audio-roundtrip/ exercises the whole path: it sends text to Azure OpenAI text-to-speech, pipes the returned MP3 straight into a Whisper transcription upload with no intermediate table, then verifies the transcript against the original phrase.

It depends on both halves of this work — the MP3 survives only because of binary capture, and it reaches the upload only because data_b64 accepts $speech.body (#303). The form also carries a prompt part whose base64 wraps across lines, which exercises #303's decode fix.

Provisioning and teardown scripts are included; the teardown purges soft-deleted Azure OpenAI accounts, which otherwise keep holding quota.

Testing

  • 15 new unit tests covering content-type classification, envelope construction, and envelope field resolution
  • E2E tests 2e/2f/2g in tests/e2e/sql/06_http_and_ssrf.sql: a binary response captured and re-uploaded losslessly, envelope field access, and rows precedence
  • The example was run end-to-end against live Azure OpenAI: 3/3 phrases round-tripped, all with encoding = base64

Locally: cargo fmt --check clean, clippy adds no new warnings (4 pre-existing in src/lib.rs remain), unit tests 229 passed / 0 failed. I did not complete a local E2E run on this branch — relying on CI for that.

Note on the diff

Docs examples that #303 had to phrase defensively (using a SQL-sourced reference and ($resp::jsonb->>'ok')) revert here to the direct form, since dot notation now exists to support them.

df.http() and df.http_multipart() read every response with `.text()`, which
does a lossy UTF-8 conversion. Any non-textual body — an image, a PDF, an
MP3 — was silently replaced with U+FFFD characters, so a binary response
could not be stored or forwarded. The bytes were gone by the time the
envelope was built.

Responses are now classified by Content-Type. A textual body is carried
through unchanged; a non-textual one is base64-encoded and the envelope
records which happened in a new `encoding` field (`text` or `base64`). A
body counts as text when Content-Type is absent, `text/*`, a `+json`,
`+xml` or `+yaml` suffix type, or one of the common textual `application/*`
types. Everything else is treated as binary, which is the safe default:
mislabelling text as base64 is recoverable, mislabelling binary as text
destroys it.

The classification and envelope construction moved into a new
activities::http_response module so both activities share one definition
rather than maintaining parallel copies.

Also makes HTTP envelope fields addressable with dot notation. Previously
dot notation only resolved against SQL results, which carry a `rows` array,
so reading a status or body required an intervening SQL node just to
destructure JSON. `$resp.status`, `$resp.body`, `$resp.ok` and
`$resp.encoding` now resolve directly. `rows` still takes precedence, so
SQL results behave exactly as before.

Adds examples/audio-roundtrip/, which exercises the whole path end to end:
it sends text to Azure OpenAI text-to-speech, pipes the returned MP3
straight into a Whisper transcription upload with no intermediate table,
and verifies the transcript against the original phrase. Provisioning and
teardown scripts are included so the example can be run and cleaned up
without leaving billable resources behind.
@pinodeca
pinodeca merged commit 0c95154 into main Jul 28, 2026
5 checks passed
@pinodeca
pinodeca deleted the feat/http-binary-responses branch July 28, 2026 23:19
pinodeca added a commit that referenced this pull request Jul 30, 2026
* Fail loudly on missing envelope fields and sniff unlabelled bodies

Two follow-ups to the binary response work in #304.

`$name.field` on an HTTP envelope left the pattern in place when the field
was missing. In SQL that is deliberate -- PostgreSQL reports it with its own
diagnostics -- but a URL, a header or a multipart field has no such parser,
so a typo travelled over the wire verbatim: `Bearer $auth.token` would be
sent as a literal and the request would fail somewhere far less obvious.
Raw contexts now fail and list the available fields; the SQL and null-safe
paths are unchanged.

Body classification no longer trusts `Content-Type` alone. A declared
textual type is still decoded by reqwest so the `charset` parameter is
honoured, but an unrecognised or absent type is decided by its bytes: valid
UTF-8 without NUL is text, anything else is base64. This stops an untyped
binary download being mangled by a UTF-8 decode, and stops textual types no
allowlist will ever cover (`application/jwt`, `application/x-ndjson`) being
base64-encoded for no reason. NUL is excluded because PostgreSQL's `text`
cannot hold it, so a mostly-zero body would otherwise fail on the way into
the result row rather than at the decision that caused it.

Also split `Content-Type` on a comma before classifying, so a proxy folding
duplicate headers into `text/html, application/octet-stream` is handled, and
name the unit in the error-body preview rather than reporting a base64
length as bytes.

* Fix HTTP test failure diagnostic

* Document outstanding problems with http and http_multipart

---------

Co-authored-by: GitHub Copilot <copilot@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant