Reverse transport: run files-pull in the push direction (remote drives, local dials out)

[adamziel]

What it does

Lets the ordinary pull pipeline move a site from a machine that cannot accept inbound connections (laptop, NAT, dev box) to a public remote. The remote runs the regular importer and owns all state — cursors, index, diff, file writes; the local site only ever makes outbound requests. An ordinary files-pull completes end to end over this reversed channel in the test suite, mirroring a source tree byte-for-byte.

In industry terms: the importer becomes a task-queue server and the source polls it over a reverse connection (the same model as GitHub Actions runners or GitOps agents), with each result piggybacked on the source's next request.

Rationale

Push previously meant building a second transfer subsystem (staging store, upload client, receiver endpoints) that reimplements what pull already does well. But the only thing actually wrong with pull for this direction is who dials whom. Inverting the transport keeps the entire tested pull pipeline — resume, delta diff, batching — and adds no new state machine:

• Remote owns the state. There is no client-side belief about the target that can drift; the machine mutating the tree holds the cursor. A lost source just pauses the transfer.
• No remote liveness problem. There is no long-lived importer process and no held socket. Each outbound request from the source is the trigger: it runs one bounded importer step and returns.
• The local side is stateless. Export commands are read-only and cursor-driven, so an interrupted source loop is replaced by rerunning it.

Implementation

Three classes under packages/reprint-importer/src/lib/reverse-transport/ plus a two-line guard per choke point in ImportClient:

ReverseTransport — the importer-side transport. Holds the one source-delivered result as a stream and serves the importer's two fetch shapes from it: fetch_json() reads it whole (JSON endpoint responses are small); fetch_streaming() reads it in 64 KB chunks, inflating incrementally when gzipped (the way curl decodes Content-Encoding: gzip), and feeds the same MultipartStreamParser chunk handler the curl path uses — a multi-megabyte file_fetch result is never buffered whole. The request after the answered one throws TransportYield (extends Error so it slips past the importer's catch (Exception) wrapper), unwinding the importer back to the endpoint with the next command. No request/response matching: the protocol is strictly sequential and the importer resumes deterministically from its persisted cursor, so a single consumed flag is all the bookkeeping.

ReverseTransportEndpoint::handle_exchange — the remote's single endpoint. One exchange banks the delivered result, advances the importer by one request, and returns the source's next instruction:

request:  raw (possibly gzip) result bytes as the body — empty on the first
          exchange — with the reported HTTP status beside it (a header in
          production, an argument in-process)
response: { "status": "done" } | { "status": "command", "command": {...} }
        | { "status": "error", "message": string }

The body is deliberately not JSON-embedded: base64-in-JSON would force both sides to buffer it whole. Commands are small; results are not. The endpoint re-enters the real importer (fresh ImportClient per pass, like an exit-code-2 resume; the same ReverseTransport across passes so the result is consumed exactly once) until it yields or completes, and never throws — importer failures come back as status: "error" so the source can tell a remote failure from transport garbage.

ReverseTransportSource — the outbound-only loop on the source site: deliver the previous result, receive the next command, run it against the source's own export.php (a hostile remote can only ask for what the source already exposes), repeat. The export output ships exactly as the engine produced it (typically gzip), spooled to a temp file — never a PHP string. Any exchange failure throws; the source never re-sends a result. Recovery is the remote re-asking from its persisted cursor: a rerun starts with no result and the remote re-issues exactly the request it is missing. Re-sending could hand a stale result to a newer request (results carry no ids), so the exchange callable must not retry either.

ImportClient — gains set_reverse_transport() and a delegation guard at the two curl choke points:

if ($this->reverse_transport !== null) {
    return $this->reverse_transport->fetch_json($url);
}

The curl path stays inline and untouched: it is welded to client state (HMAC signing, tuner hooks, progress heartbeats, curl diagnostics), so one interface over both paths would relocate ~450 battle-tested lines for zero behavior change. The adaptive tuner is skipped entirely in reverse mode (as with --no-adaptive): it paces a dialing client with duty-cycle sleeps, and on the remote those sleeps would run inside the exchange handler. import.php PHPCS unchanged (504/74). No cursor-persistence changes were needed: the importer already checkpoints at exactly the per-request boundary exit-code-2 resume uses, so a yield lands on an existing checkpoint.

Failure contract (mirrors the curl path, message-for-message): a 200 body with no multipart boundary or no completion chunk throws the curl path's exact Invalid response: … errors (which also keeps sql_chunk's crash-recovery classifier working); fread/inflate_add failures throw instead of truncating; non-200 results carry the server's error body up via ReverseTransportHttpError, and the guards run it through the same diagnose_http_error block the curl tail uses (extracted into a shared throw_diagnosed_stream_error()), so fetch_json returns curl's exact non-200 shape — an HTTP 500 error body can no longer pass the preflight payload !== null gate. Two invariant guards refuse loudly what the checkpointing can't yet support: the endpoint whitelists commands that persist a checkpoint before every request (files-pull), and follow_symlinks is rejected over the reverse transport (symlink discovery re-scans without a sub-stage checkpoint). A re-entry pass cap turns any residual no-progress loop into an error.

Not in this PR: the HTTP glue (an HMAC-authenticated route + CLI carrying the exchange over curl), the source's real loopback into its own export.php (tests use a subprocess because the exporter tears down output buffering to stream), db-pull (same seam via sql_chunk/db_index; needs a live MySQL to verify), and the sub-stage checkpoints that would let follow_symlinks and the composite pull/preflight ride the reverse transport.

Testing instructions

cd tests && ../vendor/bin/phpunit --testsuite "Reverse Transport Tests"

ReverseTransportTest runs the real importer against the real exporter — no mocks of either:

1. Happy path — files-pull mirrors a 4-file source tree (incl. binary content) byte-for-byte in three exchanges (file_index → file_fetch → done).
2. Streaming — a 24 MB file (gzip-window-defeating content, so the wire really carries ~24 MB) transfers with peak memory growth under 8 MB; identity checked by hash.
3. Crash-resume — the source loop dies before delivering the fetch result; a fresh one starts with no result, the remote re-asks from persisted state, and finishes byte-for-byte.
4. Transport failure — a 502-style garbage response throws malformed exchange response instead of finishing clean.
5. Remote failure — an importer exception surfaces on the source with its original message via status: "error".
6. Garbage-200 body — a non-multipart 200 result aborts with missing multipart boundary instead of re-asking forever.
7. Truncation — a result cut before the completion chunk fails with missing completion chunk, not a clean partial.
8. Corrupt gzip — a flipped byte fails loudly instead of silently truncating the transfer.
9. Non-200 as data — a 500 JSON error body converts to curl's exact non-200 fetch_json shape (json: null, error_code set).
10. Guards — unsupported commands and follow_symlinks are rejected up front; a no-progress importer loop is capped, not spun.

[github-actions]

Pull pipeline performance — large-directory

Site: large-directory · 2,000+ plus targeted file-transfer scenarios files · 10,000 posts · 25,000 postmeta · PHP 8.5.8

Stage	PR	trunk	Δ	Status	Details
playground-sqlite-db-pull	9.30 s	9.12 s	⚪ +179 ms (+2.0%)	✓	condition=db-pull in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=lexer native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=selected trunk: condition=db-pull in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=lexer native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=selected
playground-sqlite-db-apply	3.60 s	3.52 s	⚪ +81 ms (+2.3%)	✓	condition=db-apply to SQLite in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=parser native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=verified native_ast=WP_MySQL_Native_Parser_Node sqlite_driver_parser=verified trunk: condition=db-apply to SQLite in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=parser native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=verified native_ast=WP_MySQL_Native_Parser_Node sqlite_driver_parser=verified
Total	12.90 s	12.64 s	⚪ +260 ms (+2.1%)

_{Numbers carry runner noise; treat single-run deltas as directional, not authoritative.}

_{📈 Trunk performance history — commit-by-commit timeline.}

[adamziel]

Closing after the design pivot in #315: push stays local-driven — the remote is a passive authenticated API, no polling loop, no remote-owned cursors. This reverse-transport experiment answered the question and lost the argument; the outbound-only constraint it solved is handled by ordinary outbound requests in the new flow.

I almost went through with this one, but I couldn't make myself like the remote site requesting paths from a local site. It's a privacy and security boundary I don't think Reprint should cross. We'll do a regular push, not inverted pull. The local site will choose what to communicate and the remote site will validate it and accept it.