Photon iMessage: persistent RST_STREAM code 2 (Internal server error) on free shared line — sidecar alive but stream dead

[cjboy007]

Hi,

Photon/iMessage has been completely non-functional since yesterday. The sidecar process stays alive and port 8789 is open, but the gRPC stream to Photon cloud keeps getting killed with RST_STREAM code 2 (Internal server error). Restarting the sidecar reconnects briefly (~90s) then the same error returns.

Environment

• hermes-agent: f53b184 (latest main)
• macOS 26.5.1, Apple Silicon
• Node v25.6.1
• @photon-ai/advanced-imessage: 0.12.0
• @spectrum-ts/imessage: 8.0.0
• @spectrum-ts/core: 8.0.0
• nice-grpc: 2.1.16
• Photon free shared line

Symptoms

1. Sidecar process is running, port 8789 is LISTEN + ESTABLISHED
2. gRPC stream to Photon cloud connects successfully ("Spectrum started", "connected — sidecar on 127.0.0.1:8789")
3. After ~60-90 seconds, stream dies with:

Received RST_STREAM with code 2 (Internal server error)
path: '/photon.imessage.v1.MessageService/SubscribeMessageEvents'
code: 13
grpcCode: 13

4. Sidecar's built-in reconnect loop fires ("stream interrupted; reconnecting", attempt 1, delay 500ms)
5. Reconnect hits the same RST_STREAM error again
6. This repeats indefinitely — 685 occurrences in my logs since 2026-06-29 10:53

What I tried

• Killed sidecar PID, adapter auto-restarted it → connected briefly, then same error after ~90s
• Verified @photon-ai/advanced-imessage is latest (0.12.0) on npm registry
• Verified no local network issues (other gRPC/HTTPS connections fine)
• No auth/credential errors in logs — the connection authenticates fine, then gets dropped by the server

Impact

• iMessage channel is completely dead (no inbound, likely no outbound either)
• Other platforms (Feishu, Telegram) unaffected
• Sidecar is not crashing — it's alive but the upstream stream is unusable

This looks like a Photon cloud-side issue on the free shared line. The server accepts the connection, then immediately resets the stream with an internal error.

Any workaround or ETA on a fix?

[mrgrumpy83]

Seeing the same issue here, latest docker container on Ubuntu 24.04 LTS host.

[thomasscottbeck-sudo]

Confirming the same failure class on a clean setup, with data points that isolate it server-side:

• macOS (Mac Mini), hermes-agent v0.18.0 (2026.7.1), photon sidecar on spectrum-ts 8.1.1, free shared-line tier.
• Recurring RST_STREAM code 2 (Internal server error) on both MessageService/SubscribeMessageEvents and PollService/SubscribePollEvents: ~102 reconnect events 20:43–21:12 UTC July 1, another at 23:50 UTC.
• Between storms the stream subscribes cleanly and reports healthy, but inbound messages are silently not delivered; outbound space.send() keeps working the whole time.
• Ruled out locally: reproduced after fresh sidecar processes, fresh gRPC subscriptions, project secret rotation, a brand-new Photon project with new creds, and eliminating a second competing subscriber. One roundtrip succeeded 23:31 UTC during a healthy window; the next inbound ~20 min later was lost to a new RST window.
• Net: inbound delivery loss during these windows with no client-visible error and no replay on resubscribe. A history-API polling fallback (as suggested in Photon iMessage shared-number live stream can stay cold while history API sees inbound messages #54036) may be the only client-side mitigation, but the current spectrum-ts 8.1.1 imessage provider doesn't expose a message-list call.

[adoreparler]

Same issue, NOT using docker.

version: 0.18.0 [b98baa3] (2026-07-02)
os: Darwin 25.5.0 arm64
python: 3.11.15
openai_sdk: 2.24.0

[alt-glitch]

This was generated by AI during triage.

Re-triaged: now independently corroborated by 3 distinct reporters (Docker/Ubuntu, non-Docker macOS, and a detailed server-side isolation run), so needs-repro is removed. Related to the Photon-inbound family — #54036 (cold live stream while the history API still sees inbound), #51897 (shared-line outbound blocked), #51254 (opt-in webhook inbound transport), #50755 (secret-rotation auth) — but a distinct mechanism (an upstream RST_STREAM code 2 storm that resets the gRPC stream ~60-90s after a clean subscribe, with no replay on resubscribe). The isolation data (reproduced after a brand-new Photon project, fresh sidecars, secret rotation, and removing competing subscribers) points to a Photon cloud-side issue on the free shared line rather than a Hermes bug. The opt-in webhook inbound transport (#51254) may be the most robust client-side mitigation since the degrading catchUpEvents/subscribe stream is the shared failure point.

[zhengyilab]

I got the same issue repeatedly per 1~2 days. The silent death of the gRPC connection caused it's unable to catch / recover by Hermes gateway adapter. Since it might be a common issue for Photon shared lines, hopefully we can merge the opt-in webhook solution soon.(#51254)

version: v0.17.0
with locally patched PR#51161 for gRPC error detection & recovery
spectrum-ts/core: 6.1.1

[zhengyilab]

Subscribing for updates on RST_STREAM stability.