Beacon

A FOSS search engine for the Reticulum mesh. Beacon discovers NomadNet nodes from RNS announces, crawls their micron (.mu) pages over encrypted Reticulum Links, and builds a searchable index. The code is MIT-licensed, keeps no visitor identifiers, and speaks MeshAPI.

If you want Beacon to leave your node alone, see Opting out. One line is enough and it is honoured permanently.

Prior art: Roogle proved a mesh search engine is viable but is closed-source with thin link-only ranking. Beacon is open, and ranks on text relevance and node trust.

Crawl policy

Crawling a mesh is not crawling the web. Pages can sit behind LoRa links where the handshake costs more than the page it carries, and the person paying for that airtime did not ask to be indexed. Every limit below is enforced in code rather than by convention, and the environment variable is given so you can check what any given deployment is actually running.

Limit	Default	Variable
Pages held per node	20	BEACON_MAX_PAGES_PER_NODE
Link depth from a node's index page	2	BEACON_MAX_DEPTH
Delay between page fetches	5s	BEACON_FETCH_DELAY
Re-crawl interval, per url	24h	BEACON_RECRAWL_HOURS
Pages fetched over one Link	up to 8	BEACON_NODE_BATCH
Worker threads	1	BEACON_CRAWL_WORKERS

In practice a node sees at most about 21 requests per day, being 20 pages plus a one time robots check, spread across roughly three Links.

One worker by default, which means Beacon holds at most one Link open anywhere on the mesh at any moment. The per-node cap is what makes that sufficient: a full daily pass over every reachable node is on the order of 1,600 page fetches, which one worker covers several times over. A worker pool only ever raised the ceiling on how hard Beacon could hit the network, never how complete the index was.

If a deployment does raise it, a node's queued pages are still leased to one worker as a block, so workers never converge on the same node, and that block rides a single established Link rather than one per page.

Things Beacon does not do, which are as much the policy as the numbers are:

• It never requests downloads. Anything under /file/ or ending in a binary extension is dropped before a Link is opened.
• It does not deep link into your node from someone else's page. A link from another node discovers your node and queues your index page, nothing further. Your deeper pages are only ever reached from your own index, which is what stops a crawl walking the mesh until it surfaces somewhere unrelated.
• It backs off unreachable nodes exponentially, and an announce does not reset that backoff. Announcing is not evidence a node will answer a Link.
• It identifies on every Link, with one stable identity rather than a rotating set, so you can see who is crawling you and block exactly that.

Opting out

Beacon reads three signals. Any one is enough, all are permanent, and an announce does not undo them. On seeing a refusal Beacon stops crawling that node, drops everything queued for it, and by default deletes what it had already indexed (BEACON_OPTOUT_PURGE).

1. A MeshData robots field, in any page's head block. On your index.mu it applies to the whole node.

# +robots: noindex

This is the cheapest signal available, because it rides a page the crawler was fetching anyway and costs you no extra request. See MeshData section 9 for the full token set.

2. A bare head comment, if you would rather not use MeshData:

# robots: noindex

3. A robots page at /page/robots.mu, which Beacon asks for once per node, ever, over a Link it already has open. Any non-empty Disallow is read as a refusal for the whole node. Beacon does not implement per path rules and over-complies deliberately: reading a partial rule as a total one costs Beacon some index, whereas reading a total rule as partial would cost you bandwidth.

User-agent: *
Disallow: /

Or block the identity. Beacon identifies on every Link, so one entry stops it. Each deployment persists its own identity, so ask the operator of the instance crawling you for theirs.

An operator can also be added to BEACON_OPTOUT_NODES directly, for anyone who asks out of band rather than by publishing a marker.

Architecture

• Announce listener. An RNS AnnounceHandler on nomadnetwork.node; every node's hash, name and first/last-seen lands in the nodes registry, and its index.mu is queued.
• Crawler. Leases a block of one node's queued pages, opens a single identified Link, fetches them within the limits above, extracts readable text, a title and outgoing micron links, and queues what it is still allowed to take.
• Store. Postgres: nodes, pages (full text tsvector plus a pgvector embedding), links, crawl_queue.
• Search. An RNS service answering queries, with optional federation to an rns-wiki instance. The search interface people use is a NomadNet node; this repository's HTTP surface is private operator dashboards, not a public site.
• /healthz + /stats. JSON, for monitoring.

Run

pip install -e .
# set BEACON_DB_* and point config/rns-config at a reachable transport node
python3 -m beacon --config ./config

Docker: see docker-compose.example.yml and config/rns-config.example. Needs a Postgres and an RNS instance that hears the announce stream (a public backbone works).

Configuration is entirely environment driven. Nothing in this repository names a destination, an address or a host: copy .env.example and fill in your own. Every value is optional and degrades rather than failing, because unset is a real configuration. No wiki destination skips federation, no embedder falls back to full text search, no dashboard url omits the cross link.

Ranking

Full text relevance over a generated tsvector (title, headings, description and body weighted separately), a vector similarity pass via pgvector, freshness, anchor text from inbound links, and an announce-trust signal from a node's recency and reachability. Popularity bonuses are relevance-gated, so a heavily-linked page cannot outrank an actual answer.

License

MIT.