feat(activesync): stream Sync responses to avoid client read timeouts (Gmail Android)

[TDannhauer]

Summary

Android Gmail (and possibly other clients) abort ActiveSync Sync requests after ~30 seconds of no response body bytes, then enter a confused state that does not self-heal. Horde currently buffers the entire Sync WBXML body before sending it to the client. The existing maxresponsetime time budget (introduced in 3.0.0-RC4) mitigates large batches but does not fix connection silence, especially for bidirectional Drafts sync and heavy initial mailbox sync.

This issue tracks incremental Sync response delivery so the connection stays active while messages are assembled.

Motivation

Related to #77.

• Observed: FullDraftsUpSync SocketTimeout at 30s while the server was still working; repeated on retry; client SSL errors afterward.
• Server logs show Sync time budget (25s) reached … MOREAVAILABLE after the client already timed out.
• Recovery today requires delete-account + server state reset — we should remove the primary server-side trigger.

Root cause

1. Horde_Rpc_ActiveSync captures the full response in ob_start(1MB) and sends it only in sendOutput() with Content-Length (horde/rpc ActiveSync.php) — no body bytes reach the client until the handler finishes.
2. Horde_ActiveSync_Request_Sync may buffer the entire Commands section in a temp stream when time budget is enabled (for correct MoreAvailable ordering before Commands).
3. Incoming client Commands (e.g. draft upload via FullDraftsUpSync) and IMAP change polling run before or outside the time budget’s effective protection.
4. Gmail’s timeout is on read blocking until HTTP response data arrives, not on total sync duration while bytes flow.

Proposed solution

Phase 1 — RPC streaming (critical)

• Sync-specific path in Horde_Rpc_ActiveSync: skip full-response output buffering for Cmd=Sync; flush WBXML incrementally (likely chunked transfer-encoding for Sync only).
• Keep Content-Length / buffered path for GetAttachment, ItemOperations, etc. (Bug #12486).

Phase 2 — Sync handler incremental export

• Encoder::flushOutput() after each successful sendNextChange().
• Count-based maxmessagesperresponse (e.g. default 10): if more changes than cap, emit MoreAvailable before Commands, then stream up to cap.
• Disable command temp-buffer when streaming is enabled (_useSyncCommandsBuffer).

Phase 3 — Guards

• maxmessagetime — per-message assembly cap (pathological single draft/MIME).
• maxrequestduration — whole-request wall clock including incoming import phase.
• maxresponsetime: keep as optional legacy fallback (0 default once streaming is validated); not the primary keep-alive mechanism after streaming.

Config (proposed)

Key	Purpose
activesync.sync.streaming	Master switch (rollback to legacy)
activesync.sync.maxmessagesperresponse	Count-based batch cap
activesync.sync.maxmessagetime	Per-message assembly cap
activesync.sync.maxrequestduration	Whole-request wall clock
activesync.sync.maxresponsetime	Legacy export-phase time budget

Non-goals

• Fixing Gmail’s slow follow-up after MOREAVAILABLE (client pacing).
• Fixing client state recovery after a broken relationship (operational workaround remains).
• Full H6 request/response pipeline rewrite (doc/todo.md) — align with that roadmap but ship a focused fix first.

Test plan

• PHPUnit: MoreAvailable byte order before Commands; flush per message; pending changes preserved.
• Integration: mock driver with delayed getMessage; assert TTFB < 5s with streaming.
• Manual: Gmail Android — large Inbox + Drafts, initial sync + FullDraftsUpSync.
• Regression: iOS Mail, Nine (if available).
• Verify GetAttachment / ItemOperations unchanged (Bug #12486).

Implementation order

1. Instrumentation (timing: import / poll / export / TTFB).
2. RPC streaming path + feature flag.
3. Sync per-message flush + count cap + MoreAvailable ordering.
4. Request-wide guards; deprecate time budget as primary mechanism.
5. Client validation — then ask email sync quite erratic against both Gmail and Nine on Android #77 reporters to retest.

References

• email sync quite erratic against both Gmail and Nine on Android #77
• 3.0.0-RC4: feat(activesync): adaptive Sync response time budget
• doc/todo.md — request/response pipeline, Sync data structures

[TDannhauer]

Design update: post-commit error handling

Follow-up from implementation planning (deployment design doc + doc/todo.md).

Constraint

Once Sync streaming has sent the first WBXML body byte, we cannot send HTTP 500 (or change the HTTP status). The client already has an implicit HTTP 200 and a partial body. This is the main architectural change vs today's buffered model, where Horde_Rpc_ActiveSync only sends the body in sendOutput() and the RPC catch block can still set HTTP/1.1 500.

Two phases per Sync request

Phase	When	HTTP errors?	Error channel
Pre-commit	Before first body flush()	Yes (rare)	_handleGlobalSyncError(), _handleError(), RPC 400/500
Post-commit	After first body byte	No	Folder Status (5, 8, 12, …), SyncReplies, MoreAvailable, streaming abort handler

Pre-commit must complete work that can fail without partial delivery: policy, decode, incoming import (or deliberate importfailures), change poll. Poll failures already become folder Status before Commands.

Post-commit rules:

1. Do not rethrow to Horde_Rpc_ActiveSync for Sync after streaming starts.
2. Recoverable export errors: existing sendNextChange() false → stop loop, preserve sync_pending.
3. Uncaught exceptions: new streaming abort handler — close WBXML tags, set Status=5 (or 111 where appropriate), flush final chunk, log — not HTTP 500.
4. State/synckey must match flushed messages only; never advance past truncated Commands.

RPC notes

• Cmd=Sync: chunked transfer, no Content-Length, skip full-buffer sendOutput().
• Other commands: keep buffered Content-Length path (Bug #12486).

Ship gate

Do not close this issue without README.md documentation (problem, architecture, config keys, error model, operator notes, client matrix). Tracked as Phase 6 in the design doc; doc/todo.md near-term entry references the same requirement.

Decision log

• Post-commit Sync errors are in-protocol only; HTTP 500 only before first body byte.
• README + todo cleanup required before closing feat(activesync): stream Sync responses to avoid client read timeouts (Gmail Android) #83.

[TDannhauer]

implemented by #84