Document the push sync design#315
Merged
Merged
Conversation
The agreed design for local-driven push: self-relative change detection against per-site baselines, envelope-only HMAC over TLS, the staged artifact store as transfer target, a seconds-long journaled apply window with a whitelisted maintenance file and a no-boot escape hatch, row-diff database sync as phase two, and the accepted limitations stated out loud. Ends with the PR stack that delivers it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Pull pipeline performance —
|
| Stage | PR | trunk | Δ | Status | Details |
|---|---|---|---|---|---|
playground-sqlite-db-pull |
9.49 s | 9.35 s | ⚪ +139 ms (+1.5%) | ✓ | condition=db-pull in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=lexer native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=selected trunk: condition=db-pull in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=lexer native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=selected |
playground-sqlite-db-apply |
3.52 s | 3.55 s | ⚪ -26 ms (-0.7%) | ✓ | condition=db-apply to SQLite in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=parser native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=verified native_ast=WP_MySQL_Native_Parser_Node sqlite_driver_parser=verified trunk: condition=db-apply to SQLite in PHP.wasm runtime=php.wasm 8.3 wp_mysql_parser=enabled mode=parser native_lexer=verified native_token_stream=WP_MySQL_Native_Token_Stream native_token_count=18 native_parser=verified native_ast=WP_MySQL_Native_Parser_Node sqlite_driver_parser=verified |
| Total | 13.01 s | 12.90 s | ⚪ +113 ms (+0.9%) |
Numbers carry runner noise; treat single-run deltas as directional, not authoritative.
📈 Trunk performance history — commit-by-commit timeline.
This was referenced Jul 7, 2026
Closed
This was referenced Jul 7, 2026
"Dead-man switch" and "lazy janitor" named mechanisms instead of explaining them. Now the doc says what actually happens: WordPress ignores a .maintenance file older than 10 minutes, and any later authenticated request finishes an unfinished apply journal before its own work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
adamziel
added a commit
that referenced
this pull request
Jul 7, 2026
adamziel
added a commit
that referenced
this pull request
Jul 7, 2026
Step 2 of the push stack (#315). ## What it does Adds `HMAC(nonce + timestamp + UNSIGNED-PAYLOAD + method + path?query)` authentication support for requests with large bodies. `UNSIGNED-PAYLOAD` means the literal string `UNSIGNED-PAYLOAD`. The previous way way of HMACing the entire request body is kept for now for backwards compatibility, and may eventually be removed in favor of this one. ## Why Push uploads are large. Hashing a 512MB on both sides just to sign it costs a full extra read of the data, and #293 already had to fight the buffering this causes. Over HTTPS the body hash also buys nothing: TLS already guarantees nobody modified the body in transit. With this PR, we lose full-body verification. If a request is somehow captured, it may be replayed with a different body for a limited time. This seems acceptable, since requests are transmitted between trusted sites over HTTPS. ## Implementation - **A body-signed request can never pass the envelope check, and envelope headers can never pass a body-signed check.** The two signatures are computed over strings that can never be equal: one contains a 64-character hex hash exactly where the other contains `UNSIGNED-PAYLOAD` plus the method and path. - **The server route picks which check runs.** The command endpoints (preflight, plan confirmation, and similar) keep calling `verify_control_request()`, so a client cannot talk them into the body-less check. `--force-http` (arriving with the first push networking, stack step 7) is the one situation where an unsigned body is a real loss — on plain HTTP someone in the middle could alter the body. Its help text will say exactly that. ## Testing instructions ```bash cd tests && ../vendor/bin/phpunit HmacServerTest.php composer analyze ``` Tests cover: a signed request verifies (method case does not matter); captured headers fail against a different route or verb; a body-signed request fails the envelope check and envelope headers fail the body-signed check; expired timestamps fail; URL normalization. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
adamziel
added a commit
that referenced
this pull request
Jul 7, 2026
Part of #276. Step 3 of the push plan (`markdown/PUSH-SYNC.md`, merged in #315). Continues the store built in #298, which was closed in favor of an earlier consolidation attempt; the design has moved since (see the doc). ## What it does Adds `Site_Export_Staged_Artifacts`, a class that moves push uploads into a staging directory on the target site before they are applied. The sender uploads a file ("artifact") in pieces with repeated `append()` calls, marks it finished with `finalize()`, can ask `status()` at any time, and can throw a half-done artifact away with `discard()`. Nothing in here touches the live site — that is the apply step's job, in a later PR. Later on, this handler will also be reused for pulls to avoid modifying the live site during the network transmission. ## Rationale **Why stage at all:** a transfer can die at any moment. Bytes accumulate in a separate staging directory while the site keeps running, so the web server can never serve half an uploaded file, and giving up costs nothing — discard the staged data and the site never knew. **Why the caller drives the loop:** `append()` does exactly one bounded step per call — take one buffer, write it, flush it, move the cursor, return. The caller reads its own source and decides buffer sizes, the same way readers drive `MySQLDumpProducer` and `FileTreeProducer` by calling `next_chunk()`. Because nothing is held between calls, the transfer can stop after any step and a later request — or a whole new process — resumes from `committed_bytes`. It also means a test can stop the loop at any chosen iteration and prove recovery, which internal read loops made impossible. **Why byte counts instead of checksums:** `finalize()` compares the assembled size against the size declared when the push was planned, and never reads the file back — so it costs the same for 1 KB and 50 GB. Hashing a 50 GB artifact inside one PHP request would blow the execution time limit. This is the same trust pull's writer places in its local disk: corruption that keeps the length the same is not detected. Protecting bytes in transit is TLS's job (#319). That last point is a trade-off. It may be revisited in the future, but I want to give it a shot. ## Implementation Everything lives under one staging directory: - `files/<artifact id>` — the bytes, at their plain target-relative paths. No suffixes and no bookkeeping files in this tree, so any file name a site can contain stages verbatim, including `index.php.part` or `state.json`. - `state.json` — the cursor: which artifact is currently being uploaded and how many bytes are committed. Written as a temp file and renamed into place, so a reader never sees a half-written record. There is only one cursor because transfers are sequential like pull: starting a different artifact forgets the unfinished one's progress, and coming back to it means re-uploading it from zero. - `verified/<artifact id>` — one small marker per finished artifact, holding its verified size. Markers instead of one shared list because every `append()` checks "is this artifact already finished?" — with markers that is reading one known path no matter how many files the push has completed; a shared list would be re-read and re-parsed on every call. - `lock` — every call takes one exclusive `flock` on this file for its single step. A retry racing a timed-out earlier request gets `busy` instead of interleaved writes. The lock needs its own file: `state.json` is replaced by rename, and renaming a locked file leaves the lock on the old, deleted copy while the next request happily locks the new one. The crash rule: bytes are flushed **before** the cursor moves. A kill between the two leaves a tail past the cursor, and the next `append()` truncates that tail away and re-accepts the buffer. Every response carries `committed_bytes`; already-committed bytes come back as `duplicate` (skip forward and continue) and a wrong position comes back as `offset_gap` (resume from the reported count). A `discard()` that did not return `true` must be retried until it does — each partial state a killed discard can leave behind is cleaned up by the next attempt. ```php $staged = new Site_Export_Staged_Artifacts($staging_dir); // The endpoint's loop: read the request body in bounded buffers. while (($buffer = fread($input, 262144)) !== false && $buffer !== '') { $result = $staged->append($artifact_id, $offset, $buffer); // => ['status' => 'accepted', 'reason' => null, 'detail' => null, 'committed_bytes' => 262144] $offset = $result['committed_bytes']; } $done = $staged->finalize($artifact_id, $total_bytes); // => ['status' => 'verified', 'path' => '/staging/files/...', ...] — ready for the apply step ``` Failures return typed results, never exceptions: `accepted | duplicate | busy | rejected`, with a machine-readable `reason` and, where one reason can come from several places, a `detail` naming the exact operation that failed (`persist_cursor`, `open_artifact_file`, ...). ## Testing instructions ```bash cd tests && ../vendor/bin/phpunit StagedArtifactsTest.php composer analyze ``` 38 tests. The heart of the suite builds the exact on-disk state a kill can leave behind and proves the next run recovers and continues: stopping after every possible step and resuming in a fresh process, a kill inside a step (bytes past the cursor), a kill inside the cursor write itself, both windows of a killed discard, and a finished artifact whose file went missing. The rest covers resume signals (`duplicate`, `offset_gap`), a corrupt `state.json`, artifact names that collide with the store's own files or with each other, hostile ids (`..`, absolute paths, backslashes, NUL bytes), lock contention, and zero-byte artifacts. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
adamziel
added a commit
that referenced
this pull request
Jul 7, 2026
Part of #276. Step 4 of the push stack (#315), on top of #317. ## What it does Reprint's own storage directory becomes invisible to file indexing. `storage_path` is the server's own setting, so the indexer knows it for every request — including a pulling peer's. The staging root also gets an `.htaccess` and a blank `index.php` for hosts where it must live inside the document root. ## Rationale Some hosts allow writing nowhere outside the document root. Two consequences the design (markdown/PUSH-SYNC.md) commits to: - **Never index it**: an index that contains reprint's own working data would sync or delete the transfer's records mid-transfer. - **Never index it**: an index that contains reprint's own working data would sync or delete the transfer's records mid-transfer. The exclusion is unconditional — `include_caches=1` re-admits junk, never storage. - **Never serve it**: anything under the document root is servable, so the store writes an `.htaccess` (`Require all denied` + `Deny from all` behind IfModule) and a blank `index.php` into the staging root. That is all that can be done from inside the directory — Apache honors the deny rules, nginx ignores both files and loses nothing by their presence. Do not keep the staging directory inside the document root unless the host offers nowhere else to write. ## Implementation - The endpoint trims a trailing slash off the setting and canonicalizes it with `realpath()` once per request, because the traversal walks realpath-canonical paths (the existing wp.com symlink handling). - The per-entry check reuses `path_is_within_root()` from the exporter's `utils.php` — one string comparison, placed right beside the existing default deny-list. It runs before `stat()`, so skipping an entry costs no filesystem call, and it inherits the deny-list's resume behavior: the cursor advances past excluded entries the same way. - `Site_Export_Staged_Artifacts::ensure_web_guards()` runs from `open_lock()` and writes the two guard files, skipping any that already exist. `FileIndexSkipDefaultsTest` is now registered in `tests/phpunit.xml` (it was silently never run by `composer test`) and covers the exclusion end to end; the store test asserts the guards land and that a site file named `.htaccess` still stages verbatim under `files/`. ## Testing instructions ```bash cd tests && ../vendor/bin/phpunit FileIndexSkipDefaultsTest.php StagedArtifactsTest.php composer analyze ``` --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #276.
The agreed design for push, written down before the stack lands: local-driven flow, per-site self-relative baselines (ctime/TSV as granted), envelope-only HMAC with TLS required and an explicit
--force-http, the staged artifact store (#298) as the transfer target, a journaled seconds-long apply window with a whitelisted.maintenancefile and a standalone no-boot escape hatch, row-diff database sync as phase two, accepted limitations stated explicitly, and the delivery plan as a 12-PR stack.Supersedes the relay approach in #277 (its apply primitives get harvested in stack step 9).