Skip to content

Streaming API: multiplexed WebSocket connection (#3) - #141

Merged
milmazz merged 13 commits into
mainfrom
streaming-3
Jul 16, 2026
Merged

Streaming API: multiplexed WebSocket connection (#3)#141
milmazz merged 13 commits into
mainfrom
streaming-3

Conversation

@milmazz

@milmazz milmazz commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Closes #3. The last item of the Mastodon 4.6 API parity effort's user-facing backlog; builds on the Req/Mint stack from #103.

What

Real-time events over Mastodon's multiplexed streaming WebSocket (3.3+), WebSocket-only by design (SSE stays out of scope):

  • Hunter.Streamingconnect/2 (options: streams:, subscriber:, url: override for split streaming hosts, transport_opts:), runtime subscribe/3 / unsubscribe/3, graceful close/1, and health?/2 (plain-text endpoint, bypasses the JSON transformer). Requires Erlang/OTP 26+.
  • Hunter.Streaming.Connection — one GenServer over mint_web_socket (new runtime dep; mint was already in the tree) owning the Mint conn + WebSocket state. Handshake happens synchronously in init/1, so connect/2 returns {:error, reason} cleanly with no orphan process. Server pings are answered, never surfaced. No auto-reconnect: a single {:hunter_stream, pid, {:closed, reason}} (:local | {:remote, code} | {:error, term}) is delivered and the process exits :normal — callers supervise.
  • Hunter.Streaming.Event — pure frame parser; payloads decode through the existing transformer (update/status.updateStatus, notification, conversation, announcement, announcement.reaction → new :announcement_reaction target, deletes → id string). Unknown event types are delivered with the raw payload for forward compatibility.
  • Removed Hunter.EventStream (breaking): the 2017 SSE frame struct was never wired to anything.

Testing

  • Scripted in-process WebSocket server (bandit + websock_adapter, test-only): real handshakes through mint_web_socket — token on the upgrade request, control frames on the wire, event delivery, ping→pong, all three close paths monitor-verified. 213 unit tests, 0 failures; format/credo/dialyzer clean.
  • CI stack gains the mastodon-streaming container and nginx WebSocket proxying; new live integration test (connect to the user stream, post a status, assert the update event) passed 25/25 against Mastodon v4.3.8.

Notable: dialyzer cleanliness required routing around a mint_web_socket 1.0.5 typespec bug (its @opaque t declares fragment: tuple() while defaulting to nil, which erases new/4's ok-branch) — we call new/5 with the documented default mode: :active, with a load-bearing comment at the call site.

Spec: docs/superpowers/specs/2026-07-10-streaming-design.md · Plan: docs/superpowers/plans/2026-07-10-streaming.md

🤖 Generated with Claude Code

milmazz and others added 12 commits July 10, 2026 01:20
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add Mastodon's dedicated streaming container to the CI compose stack,
proxy /api/v1/streaming through nginx with WebSocket upgrade headers,
boot the new service from setup_mastodon.sh, and add a live integration
test covering Hunter.Streaming.health?/2, connect/2, and close/1 against
a real user-stream update.
Fix an http(s) url: override crashing Connection.init/1 (schemes/1
only matches ws/wss) instead of returning {:error, reason} as
documented, and note the OTP 26+ floor in the moduledoc. Also:
tidy connect/2 and handle_call(:close, ...), relocate the private
ws_uri/2 helper, drop a no-op map key, document close/1's :noproc
exit on repeat calls, switch the test WebSocket server to
start_supervised!, and add regression tests for the url normalization
and for events with no "stream" key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread lib/hunter/streaming/connection.ex Outdated
@milmazz

milmazz commented Jul 16, 2026

Copy link
Copy Markdown
Owner Author

@claude review once.

@claude

claude Bot commented Jul 16, 2026

Copy link
Copy Markdown

Claude finished @milmazz's task in 3m 11s —— View job


Review: Streaming API (multiplexed WebSocket) — PR #141

I read through Hunter.Streaming, Hunter.Streaming.Connection, Hunter.Streaming.Event, the transformer additions, the test-only WebSocket server, and the unit/integration tests. Overall this is a well-scoped, carefully-tested implementation — the design doc's constraints (no reconnect, synchronous handshake in init/1, ping answered but never surfaced) are all correctly reflected in the code and covered by tests.

Already addressed: your own review suggestion collapsing the redundant case subscribe_initial(...) do ... end into a direct return (lib/hunter/streaming/connection.ex) is in commit fc13b98init/1 now returns subscribe_initial(state, streams) directly, which is correct since that function already returns {:ok, state} | {:error, reason}.

Minor observations (no blocking issues found):

  1. lib/hunter/streaming.ex:118-149http_base_url/2 and ws_uri/2 both implement the same wss/wshttps/http scheme swap in opposite directions. Could be factored into one small private helper, but the duplication is small and each call site stays readable as-is — not worth churn.

  2. lib/hunter/streaming/connection.ex:61-75handle_call({:control, ...}) and handle_call(:close, ...) both reply :ok even when send_frame/2 fails, relying on the subsequent {:hunter_stream, pid, {:closed, reason}} message as the actual failure signal. That's consistent with the module's fire-and-forget design (and matches the moduledoc's description of the closed-notification contract), but it might be worth a one-line note on subscribe/3/unsubscribe/3/close/1 docs that a :ok return only means "enqueued," not "delivered" — easy to miss for a caller skimming just the @spec.

  3. @handshake_timeout 15_000 (connection.ex:12) is fixed and not exposed via connect/2 options. Probably fine for now given no other timeouts are configurable either, but could be a future opts addition if slow/flaky streaming hosts turn up in practice.

  4. Test coverage is thorough (three close paths monitor-verified, ping/pong, malformed-frame resilience, runtime subscribe/unsubscribe, url override normalization). One small gap: there's no test for init/1 failing after the handshake succeeds but during subscribe_initial (e.g., the control-frame send fails) — low value given how unlikely that path is right after a successful upgrade, but worth knowing it's untested if it ever changes.

Nothing here blocks merging — the connection lifecycle, event parsing (including the announcement.reactionHunter.Announcement.Reaction target and forward-compatible unknown-type passthrough), and close/error handling all look correct and match the stated design.

I did not run mix test/credo/dialyzer myself in this sandbox (no network access to fetch deps was available), so I'm relying on the PR description's stated clean run — happy to actually execute them if --allowedTools is expanded to permit git fetch/mix deps.get.

@milmazz
milmazz merged commit 808dfc1 into main Jul 16, 2026
5 checks passed
@milmazz
milmazz deleted the streaming-3 branch July 16, 2026 04:37
milmazz added a commit that referenced this pull request Jul 16, 2026
Adds Hunter.Streaming with a multiplexed WebSocket connection, event
frame parser, and health check. Removes Hunter.EventStream. Includes a
live CI integration test against a real Mastodon streaming service.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Streaming API support

1 participant