Skip to content

Firehose phase 2: follows hose (kind 3) + relay-URL harvesting #9

Description

@melvincarvalho

Phase 2 of the firehose rework (after #7). Adds the follows hose (kind 3) and retires the legacy raw-upsert path for kind 3. First real exercise of the HOSES switch (HOSES=profiles,follows).

Background

The follows collection uses a derived shape — { pubkey, follows: [hex…], created_at, count } — not raw events. That matters: the in-degree query (followerCountcountDocuments({ follows: pubkey })) and enrichCounts both depend on the top-level follows array + count. The legacy upsertEvent fallback (current phase-1 path for kind 3) writes the raw event instead, which has no follows array — so follower counts silently don't accrue for events it writes. Moving kind 3 to a proper hose fixes that.

The old followshose.js also harvested relay URLs from the kind-3 content (legacy NIP-65 relay map, {"wss://…":{read,write}}) into the relays collection. We keep that discovery side-effect.

Scope

  • src/hoses/event.js: extract the shared structural + schnorr-signature check (eventId, verifySignature) used by every hose; refactor the profiles hose to use it (DRY).
  • src/hoses/follows.js (kind 3):
    • Verify the event signature (kind 3 content is optional — unlike kind 0, it is not required to be JSON).
    • Derive { pubkey, follows: [hex…], created_at, count } from p tags (validate 64-hex, dedupe).
    • Latest-wins upsert (the old followshose blindly replaced — an out-of-order older event could clobber a newer list; we gate on created_at).
    • Owns its indexes: pubkey, follows (reverse lookup), created_at.
    • Relay harvesting: parse content as a relay map, canonicalize URLs (trailing slash on origins; ws/wss only), $setOnInsert into the relays directory so discovery never overwrites health metrics. Best-effort — never fails ingest.
  • src/indexer.js: register the follows hose. The switch then removes kind 3 from the legacy set automatically; HOSES=profiles,follows becomes a meaningful selection.
  • Tests: verify/parse/harvest unit tests; planIngest two-hose cases already exist from Firehose phase 1: validated profiles hose (kind 0) + extensible hose seam #7.

Out of scope (later phases)

Relay-lists hose (kind 10002) · relay-health prober · deploy/retire legacy server hoses.

Acceptance

  • npm test green incl. new follows-hose tests.
  • Default indexer now routes kind 3 → follows hose (derived shape, in-degree works); kind 10002 still legacy until phase 3.
  • Forged/tampered kind-3 events rejected; relay URLs from content discovered into relays without clobbering existing health docs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions