docs: add LLM.md for explorer architecture#12
Open
Pattermesh wants to merge 40 commits into
Open
Conversation
- Add docker.yml using hanzoai/.github/.github/workflows/docker-build.yml@main - Add workflow-sanity.yml to enforce canonical CI contract - Remove bespoke docker build steps from existing workflows Refs: hanzoai/.github canonical Docker CI contract.
Drop branch triggers, semver-only tags. Pin lux-build-linux-{amd64,arm64}
ARC runners. Eliminates the amd64-only manifest that crashed Apple Silicon
local k3d pulls.
Drop the _lqd._tcp / lqd / liquid daemon entry from the mDNS service discovery list. The lqd daemon is the Liquidity-side white-label node binary; the OSS Lux explorer must not advertise it.
ci: build multi-arch (amd64 + arm64) on native runners
Module renamed to github.com/luxfi/explorer (was -unified). Imports
luxfi/indexer/{daemon,evm,storage,explorer} and luxfi/graph/{engine,
indexer,storage} as libraries; embeds the luxfi/explore SPA via go:embed
all:static.
A new ChainSupervisor wraps the existing ChainRegistry: adding a chain
(via config / mDNS / admin API) spawns a per-chain indexer goroutine plus
one graph engine + indexer per enabled subgraph; removing cancels them
and frees the routes from a sync.Map dispatcher mounted on /v1/indexer/,
/v1/explorer/ (legacy alias), and /v1/graph/.
Config schema is fully per-chain: indexer poll interval, list of graph
subgraphs (each can pick a built-in schema like amm, securities, fhe…),
brand (name/coin/accent + on-disk icon/logo paths), token features, and
top-level networks list. /envs.js exposes the chain list and active
brand to the SPA at runtime, so a deploy can rebrand by remounting
chains.yaml + assets — no image rebuild.
Dockerfile clones luxfi/explore + luxfi/indexer + luxfi/graph as
siblings inside the build, populates ./static from the SPA build, and
produces one binary. CI checks out the three repos as siblings so the
local replace in go.mod works without tagged releases.
Mirrors the luxfi/indexer change: switch all workflows from the nonexistent hanzo-* labels to lux-build-linux-amd64, inline the multi- arch Docker build (arm64 via QEMU), drop the cross-org reusable, drop the workflow-sanity job that referenced cross-org actions.
Local go env had GONOSUMDB=...hanzoai/* which made `go mod tidy` use the GitHub-direct hash (YxjIynl…). CI fetches from proxy.golang.org which returns a different hash (5Rl3…) because the proxy cached a slightly different zip layout. Replace ours with the same value indexer/graph already use.
…setta Matches lux/graph + lux/indexer docker.yml. The unified explorer binary bundles SPA + indexer + graph in one image, so this is the canonical multi-arch artifact for the Liquidity local stack and devnet.
* chore: add BSD 3-Clause License * docs: add LICENSING.md pointer to canonical IP/licensing strategy
Imaginary runner label `lux-build-linux-amd64` does not exist. The live amd64 pool is `hanzo-build-linux-amd64`. - docker.yml: docker job
Cross-org runs-on dispatch failure: hanzo-build-linux-amd64 scale set is registered to github.com/hanzoai and silently rejects luxfi/* jobs. Switch to lux-build (githubConfigUrl=github.com/luxfi, max 20).
The bundled SPA opens EventSource (SSE) channels for live block / tx /
token streams and pre-flights each with HEAD:
fetch(url, {method: 'HEAD'}).then(r => r.ok ? openEventSource() : disable());
When the path doesn't match a real asset and falls through to index.html,
the HEAD returns 200 — so the SPA proceeds to open the EventSource. That
immediately aborts because the body is `text/html`, not
`text/event-stream`, and the browser logs:
EventSource's response has a MIME type ('text/html') that is not
'text/event-stream'. Aborting the connection.
…for every channel (blocks, internal-txs, tokens, token-transfers,
gas-tracker, stats, validators). Noisy; channel disabled anyway after
the abort.
Return 404 specifically for HEAD requests on the SPA-fallback path. Real
embedded assets still serve fine on HEAD; only the index.html fallback
gets the 404, which lets the SPA's HEAD pre-flight cleanly fail and the
channel disable itself silently. GET/POST/etc. on the same paths still
serve index.html for SPA client-side routing — no visible UX change.
* fix(frontend): 404 on HEAD for SPA-fallback paths
The bundled SPA opens EventSource (SSE) channels for live block / tx /
token streams and pre-flights each with HEAD:
fetch(url, {method: 'HEAD'}).then(r => r.ok ? openEventSource() : disable());
When the path doesn't match a real asset and falls through to index.html,
the HEAD returns 200 — so the SPA proceeds to open the EventSource. That
immediately aborts because the body is `text/html`, not
`text/event-stream`, and the browser logs:
EventSource's response has a MIME type ('text/html') that is not
'text/event-stream'. Aborting the connection.
…for every channel (blocks, internal-txs, tokens, token-transfers,
gas-tracker, stats, validators). Noisy; channel disabled anyway after
the abort.
Return 404 specifically for HEAD requests on the SPA-fallback path. Real
embedded assets still serve fine on HEAD; only the index.html fallback
gets the 404, which lets the SPA's HEAD pre-flight cleanly fail and the
channel disable itself silently. GET/POST/etc. on the same paths still
serve index.html for SPA client-side routing — no visible UX change.
* feat(realtime): add per-channel SSE endpoints for the bundled SPA
The bundled Vite SPA opens EventSource (Server-Sent Events) against
relative paths matching its channel list — `/blocks`,
`/transactions`, `/token-transfers`, `/internal-txs`, `/tokens`,
`/gas-tracker`, `/validators`, `/stats`. None of these existed on
the backend, so every channel pre-flight (`HEAD <path>`) fell
through to the SPA's static fallback, the SPA then opened the
EventSource, and the browser logged:
EventSource's response has a MIME type ("text/html") that is not
"text/event-stream". Aborting the connection.
…for each channel on every page load. PR #4 silenced this by 404-ing
the HEAD pre-flight for SPA-fallback paths. This PR goes further:
make the endpoints REAL so the SPA stays connected and receives
broadcasts from the existing RealtimeHub.
Adds:
* `realtime_sse.go` — `sseRegistry` (separate from the WebSocket
client map), `HandleSSE(channel)` factory returning a per-path
http.HandlerFunc. Each client gets a 64-deep bounded channel;
slow consumers drop messages rather than back-pressure the hub.
* `RealtimeHub.Broadcast` — now fans out to both WebSocket clients
(existing path, unchanged shape) AND SSE clients (new path).
Message payload is encoded once, wrapped in SSE framing only
when sending to the SSE registry.
* `main.go` — 8 path → channel mappings, registered on the mux with
explicit HEAD + GET so the SPA's HEAD pre-flight + EventSource
open both hit the same handler.
Special handling:
* HEAD requests return 200 + the SSE Content-Type, no body. The
SPA's `fetch(url, {method: HEAD}).then(r => r.ok)` check passes
and EventSource opens cleanly.
* `/stats` synthesizes an event every 5s from the hub's connection
count — useful liveness signal even before broadcasters are
wired up.
* Other channels stay open with 30s keep-alive comment lines
(`: ping\n\n`) and wait for hub.Broadcast() to fire. They emit
no events today (no production broadcasters), but the
infrastructure is in place — once the indexer is wired up to
push to the hub, live streaming "just works".
Verified locally:
* `go build .` → clean (34MB binary)
* `go test ./... -run 'Realtime|Hub|SSE'` → ok
* The existing WebSocket subscribe / broadcast flow is unchanged
(no breaking changes to RealtimeHub's public API or wire format).
---------
Co-authored-by: Woo Bin <woo.bin@satschel.com>
When ChainSupervisor.AttachRealtime is called (from main), each chain's indexer.Subscriber.OnBroadcast is wired to push block events into the RealtimeHub on the chain's slug. The hub then fans out to WebSocket clients (existing path) and SSE subscribers (new in #5). Requires luxfi/indexer v1.4.4 — pulls in the exported Subscriber() accessor. End-to-end chain: evm.Indexer.indexBlock → Subscriber.BroadcastBlock → OnBroadcast → RealtimeHub.Broadcast → ws.send + SSE fanout → SPA EventSource Now /blocks, /transactions, /token-transfers, etc. on the explorer host actually stream live events as blocks land instead of just returning HEAD 200 with no payload.
…ting /v1/indexer (#7) dispatchIndexer's splitSlug peels off the first segment as a "slug" if it matches the slug regex ([a-z0-9][a-z0-9-]{0,63}). When that segment turns out NOT to be a registered chain, the previous code forwarded only the REMAINDER to the default chain — so `/v1/indexer/main-page/blocks` silently became `/v1/indexer/evm/blocks` server-side, a completely different endpoint that returns a paginated envelope instead of the raw array the SPA expects. The SPA's home page widget does `data.slice(0, 8).map(...)` on the result and threw "o.slice is not a function" against the envelope shape. Fix: when the parsed slug isn't a registered chain, re-join it with the rest before delegating so the original path is preserved under the default chain (`/v1/indexer/main-page/blocks` → `/v1/indexer/evm/main-page/blocks`).
The bundled SPA opens a single EventSource at /v1/base/realtime and
routes incoming frames via msg.event off a handler map — see
realtime_sse.go for the wire shape. Per-channel /blocks /transactions
etc. stay for external consumers; the SPA only uses this one path.
client.es.onmessage = o => {
const c = JSON.parse(o.data);
const d = this.handlers.get(c.event);
if (d) for (const m of d) m(c.data);
};
Wire:
- HandleMultiplexedSSE() attaches a wildcard sseClient (channel=`*`)
- Broadcast() now builds two SSE payloads per event:
perChannel: existing `event:<type>\ndata:<msg>\n\n` framing for
/blocks /transactions /etc.
wildcard: `data:{"event":<type>,"chain":<slug>,"data":<...>}\n\n`
for /v1/base/realtime — no `event:` line so SPA's
onmessage gets the JSON envelope and dispatches.
- sseRegistry.fanout dispatches both payloads to matching clients
based on channel ("*" → wildcard, "blocks" → perChannel)
- main.go mounts GET + HEAD /v1/base/realtime (HEAD is the SPA's
pre-flight to decide whether to disable the live channel).
Without this path the SPA's HEAD pre-flight got a 404 (the explorer
SSE-HEAD-silencer from #4) and the client set `disabled=true`, so
no live updates ever reached the home page.
… content-type (#9) GCP HTTP/2 load balancers (and Traefik in some configs) return 502 to clients when a HEAD response carries `Connection: keep-alive` plus a streaming content-type — because they expect HEAD to be a closed-body response, not an opened SSE stream. Symptom: external `curl -I https://.../v1/base/realtime` → 502 even though in-cluster `curl -I http://localhost:8090/v1/base/realtime` → 200 OK with full SSE headers. SPA's `fetch(..., {method:'HEAD'}).then(r => r.ok)` gets false → live channel `disabled=true` → no live updates ever fire. Fix: HEAD pre-flight returns ONLY `Content-Type: text/event-stream` + 200, without the streaming-only headers. The full SSE header set still ships on the actual GET stream. Applies to both the per-channel HandleSSE() and the multiplexed HandleMultiplexedSSE() handlers.
) Two related fixes for running the explorer against a mainnet-fork devnet: 1. Honor the per-chain `indexer.start_block` field from chains.yaml when the indexer DB is fresh. Without this the indexer always backfills from block 0, which on a mainnet fork is 25M+ blocks of irrelevant pre-fork history. Now an operator can set: chains: - slug: evm rpc: http://anvil:8545 indexer: start_block: 25162000 and the indexer skips to that block on first run. Once any rows exist in evm_blocks, start_block is ignored and indexing resumes from MAX(number)+1 as before — backward-compatible for fresh-genesis chains. 2. Default the bundled SPA's NETWORK_ID to 8675312 (the Liquid devnet chain ID). The Dockerfile previously inherited luxfi/explore's default of 8675311 — a stale leftover from before the devnet chainId migration. Adds NETWORK_ID / NETWORK_NAME / CURRENCY_SYMBOL as build-args so the same Dockerfile produces correct images for devnet/testnet/mainnet. Requires luxfi/indexer #8 (StartBlock field on evm.Config) — bumped via `go mod edit` to the post-merge commit d5e65d4b50c1.
The previous matrix did linux/amd64,linux/arm64 with arm64 via docker/setup-qemu-action. QEMU-emulated builds of this image (Next.js SPA + Go cgo-sqlite) ran for ~2 hours per tag — fast enough to bottleneck every hotfix tag. Drop arm64. Cluster runners are amd64; Mac devs already run native arm64 under OrbStack with its own arm64 base story. If we ever need an arm64 multi-arch index, the right path is a second matrix entry on a native arm64 runner — not QEMU.
Pulls luxfi/indexer#9 — formatAddress + addrCounters now read both the legacy `transactions_count` / `contract_code` and the SQLite-backed `tx_count` / `code` column names, and coerce nil counters to int64(0) so the Blockscout-derived SPA can call .toLocaleString() without crashing. Repros against rpc.dev.satschel.com: GET /v1/indexer/evm/addresses/0x58a86aafb6cdd0989c799353c891e420fb530e0a returned transactions_count: null + is_contract: false for a real deployed AMM pool → now returns the actual tx count + is_contract: true.
Pulls luxfi/indexer#10 on top of #9. Address pages on the explorer SPA now show real tx counts and correctly flag contracts.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
luxfi/indexer,luxfi/graph,luxfi/explore), the route strategy on a single listener, and the WS + parallel SSE registry that shares the broadcast feed.README.mdfor the canonical routes table, fullchains.yamlexample, and curl examples.Test plan
main.go,supervisor.go,realtime.go,frontend.go,registry.go,README.md, andgo.mod.