feat(activesync): stream Sync responses to prevent client read timeouts#84
Merged
Conversation
Flush WBXML to the client after the envelope status, after every exported message, and at each folder close when the new 'streaming' sync config is enabled, so clients with hard read timeouts (Gmail Android, ~30s without body bytes) no longer abort large syncs. - Encoder::flushOutput() pushes the WBXML stream and the SAPI buffer. - The count-based maxmessagesperresponse cap is folded into the effective window before Commands starts, so MOREAVAILABLE always precedes Commands via the window-exceeded path without buffering. - The legacy maxresponsetime budget and its Commands buffer are bypassed while streaming (log notice when both are configured). - New maxmessagetime / maxrequestduration guards stop a batch between messages; unsent changes stay in sync_pending. - Post-commit abort handling: exporter exceptions mid-stream are logged and the WBXML envelope is closed instead of rethrowing to the RPC layer (HTTP 500 is impossible after the first body byte). - README: new "Sync response streaming" section (architecture, config, error model, operator notes); doc/todo.md updated. Requires the companion streaming path in horde/rpc and the config passthrough in horde/horde (same branch name in both repos). Refs #83, #77.
maximumwindowsize (client WindowSize override) and the streaming maxmessagesperresponse cap (#83) both feed the same export-loop bound via min(). Record in the Horde 6 roadmap that the Sync options / configuration-builder refactor should collapse them into one batching policy; they stay separate on FRAMEWORK_6_0 for rollback semantics.
…en streaming Streaming (phases 1-4) fixed export-side stalls, but a large client upload batch (e.g. Gmail FullDraftsUpSync with 200 Modify commands) was still imported to the backend during request parsing - before any response byte - leaving the wire silent well past hard client read timeouts (~30s on Gmail Android). When streaming is enabled, client-sent ADD/MODIFY/REMOVE commands are now only queued during parsing and imported during response output, after the flushed envelope status, at the same logical position the inline imports occupied (before change detection and synckey generation). Between imports the encoder flushes a WBXML keep-alive token (redundant SWITCH_PAGE to the active code page, a no-op for conforming parsers) so response bytes keep flowing for the whole import phase. Import errors are recorded as per-command reply statuses instead of being thrown through the streamed body. The non-streaming flow is unchanged; the shared import logic is extracted into helpers used by both paths.
Split the monolithic README into a landing page plus doc/ guides for administrators (configuration.md), integrators (integration.md), and developers (architecture.md), with cross-cutting references for EAS protocol versions 2.5-16.1 (protocol-versions.md) and the streamed Sync delivery design (sync-streaming.md). Document all protocol versions equally instead of highlighting 16.x only. Add @see pointers from Request_Sync and Encoder::keepAlive() to the streaming design doc.
4 tasks
🔍 CI ResultsOverall: ❌ 12/12 lanes failed TL;DR: ❌ Quality issues: PHPUnit: 54 tests failed; PHPStan: 329 unique errors in 8 lanes; PHP-CS-Fixer: 22 files. Summary by PHP Version
Quality Metrics
❌ Failed Lanesphp8.0-dev
php8.0-stable
php8.1-dev
php8.1-stable
php8.2-dev
php8.2-stable
php8.3-dev
php8.3-stable
php8.4-dev
php8.4-stable
php8.5-dev
php8.5-stable
CI powered by horde-components • View full results |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in streaming mode for Sync responses so clients with strict “time-to-first/next-byte” read timeouts (notably Gmail on Android) keep receiving response bytes while the server is still exporting changes and/or importing large up-sync batches. It also reorganizes documentation into README.md + audience-targeted guides under doc/, and adds unit coverage for the new streaming behavior.
Changes:
- Add streamed
Syncdelivery: incremental WBXML flushing during export, plus deferred import of client commands with WBXML keep-alive tokens during long up-sync work. - Extend WBXML encoder with idempotent header emission,
flushOutput(), andkeepAlive(). - Add/refresh documentation and add unit tests validating streaming mechanics and bookkeeping.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
lib/Horde/ActiveSync/Request/Sync.php |
Implements streaming mode, deferred imports, keep-alives, and new streaming-related guards/caps. |
lib/Horde/ActiveSync/Wbxml/Encoder.php |
Adds streaming primitives (flushOutput(), keepAlive()) and idempotent header emission. |
test/unit/Horde/ActiveSync/Request/SyncStreamingTest.php |
Adds unit tests for streaming window sizing, header idempotence, keep-alive transparency, and deferred import behavior. |
README.md |
Refactors into a landing page with a documentation index and updated feature highlights. |
doc/sync-streaming.md |
Design document for streaming Sync responses, including error model and operator notes. |
doc/configuration.md |
Admin-facing configuration guide including streaming options and deployment prerequisites. |
doc/integration.md |
Integrator guide for embedding the library and handling streamed vs buffered responses. |
doc/architecture.md |
Developer-oriented internal architecture overview including streaming mechanics. |
doc/protocol-versions.md |
Protocol version negotiation and per-version behavior overview. |
doc/todo.md |
Updates roadmap notes, including streaming status and follow-ups. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Encoder::flushOutput() now also flushes an active PHP output buffer before flush(): the Horde RPC layer removes buffers when streaming, but output_buffering=On or non-Horde integrators may still have one active, which would silently hold the streamed bytes back. Reshape the delete branch of _importRemoves() to the same ['results' => ..., 'missing' => ...] structure importMessageMove() returns, replacing the confusing legacy self-assignment ($results['results'] = $results) that was moved here from the inline import path.
Round-3 reporter testing (issue #77) showed Gmail Android receiving a streamed Sync response without timing out - the transport fix works - but then retrying with the old synckey, i.e. discarding the response. Prime suspect is the volume of WBXML keep-alive tokens: one redundant SWITCH_PAGE per imported command (200 in the failing request) may exceed what Gmail's stricter-than-spec parser tolerates. Emit keep-alives at most once per interval instead of per command: new keepaliveinterval sync setting, default 15s - safely below the ~30s client read timeout while reducing 200 tokens to 3 for a 57s import phase. 0 restores per-command emission. The import summary log line now reports how many keep-alives were emitted.
5 tasks
The MODIFY import path dropped the conversationid/conversationindex returned by the backend, unlike the ADD path. For EAS 16 email collections the exporter skips SYNC_MODIFY replies for items without attachment or conversation data, so a Drafts up-sync (e.g. Gmail's FullDraftsUpSync) received a response with no Replies block at all. Gmail rejects such a response and re-sends the same command batch indefinitely, never advancing its synckey. Preserve the conversation data like the ADD path does so every successfully imported draft modify gets its reply. Affects both the streaming (deferred) and legacy inline import paths, which share _importSyncCommand().
This was referenced Jul 17, 2026
4 tasks
ralflang
approved these changes
Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Syncresponse delivery: WBXML is flushed to the client incrementally (chunked HTTP) instead of buffering the whole response behind aContent-Lengthheader. Implements feat(activesync): stream Sync responses to avoid client read timeouts (Gmail Android) #83, motivated by email sync quite erratic against both Gmail and Nine on Android #77.FullDraftsUpSync).README.mdplusdoc/guides for administrators, integrators, and developers, including a dedicated design document for streaming and equal coverage of all EAS protocol versions (2.5–16.1).Motivation
Some clients — notably Gmail on Android — abort a
Syncconnection after ~30 seconds without response body bytes (SocketTimeout), retry with the old sync key, and can end up in a permanently broken sync state (#77). The client timeout is on time to first/next byte, not total request duration, so keeping bytes flowing fixes the class of failures without artificial batch limits.Two silent periods could exceed that budget:
Changes
Request_Sync: streaming mode flushes after the envelope, after every exported message, and at folder close. IncomingAdd/Change/Deletecommands are queued during parsing and imported during response output — at the same logical position relative to change detection and sync-key generation — with a keep-alive flushed between imports. Import errors are reported in-protocol (per-commandSyncRepliesstatuses) instead of HTTP 500.Wbxml_Encoder: idempotent WBXML header; newkeepAlive()emitting a redundantSWITCH_PAGEto the active code page (semantic no-op for conforming parsers, verified against this package's own decoder).MoreAvailableordering without buffering: themaxmessagesperresponsecount cap is folded into the effective window size up front, so truncation is known before theCommandsblock starts; the legacymaxresponsetimebudget only applies when streaming is off.$conf['activesync']['sync']:streaming(defaultfalse),maxmessagesperresponse,maxmessagetime,maxrequestduration. Fully reversible: disablingstreamingrestores the exact buffered behaviour.doc/configuration.md,doc/integration.md,doc/architecture.md,doc/protocol-versions.md,doc/sync-streaming.md;README.mdrewritten as a landing page with a doc index.test/unit/Horde/ActiveSync/Request/SyncStreamingTest.php(keep-alive transparency to the decoder, header idempotence, deferred-import execution and bookkeeping, config plumbing).Companion changes:
horde/rpc—Horde_Rpc_ActiveSyncstreaming path forCmd=SyncPOST only (no output buffer, no zlib, noContent-Length,headers_sent()-guarded error paths): feat(rpc): stream ActiveSync Sync response bodies Rpc#5.horde/horde—rpc.phpconfig passthrough andconf.xmlkeys: already merged intoFRAMEWORK_6_0.Expected exception types in new catch blocks:
Horde_ActiveSync_Exception(and subtypes) around deferred command import and streamed export; failures are logged and converted to per-command/folder statuses.Test plan