JoshBot v1.1.0

JoshBot v1.1.0

JoshBot v1.1.0 is a backwards-compatible minor release. It turns the
post-v1.0.0 operational, reliability, pacing, and authenticated-crawler work
into a supported release boundary for production soak.

Safer and more controllable discovery

• Private operational reporting API with bearer authentication, status,
  sources, crawls, queue, services, audit history, and OpenMetrics /metrics.
• Independent operator control API for processor pause/resume, domain-avoid
  rules, exact-origin block/allow, and append-only audits.
• Source detail reporting with provenance, queue state, latest crawl, and
  stored robots observations.
• Keyset pagination via X-JoshBot-Next-Cursor.

Bounded automatic expansion

• Automatic crawl-source admission with per-run promotion budgets.
• Deferred candidates and pending-probe backpressure so expansion cannot
  silently overwhelm verification.

Crawler reliability

• Shared durable retry policy for transient discovery and verification
  failures.
• Discovery source claims use expiring leases so crashed crawls can be
  reclaimed.
• Worker and discovery loops no longer park unrelated due work behind a
  single origin retry schedule.
• Verification lease renewal and completion use authoritative lease budgets.
• Redirect-loop tracking is local to each fetch attempt so transient
  same-origin redirects can be retried safely.
• Crawl service heartbeats use stable logical service slots.

Crawl-delay and request pacing

• Robots Crawl-delay parsing with JoshBot-over-wildcard precedence,
  fractional seconds, and explicit invalid-value reporting.
• Shared per-origin request scheduling uses the greater of the configured
  request delay and the applicable robots Crawl-delay.

Authenticated crawler identity (Web Bot Auth)

• Ed25519 HTTP Message Signatures with JWK-thumbprint key IDs, short-lived
  signatures, and per-request nonces.
• Dedicated signing-key configuration, optional transition identity, and
  required / unsigned modes.
• Required mode fails closed before crawler work begins.
• Signing is applied at the shared outbound crawler HTTP boundary.
• Private key material stays out of environment dumps and public JWK output.

BotBase conformance tooling

• joshbot conformance web-bot-auth --expect unregistered|verified
• Validates the public signature directory and Cloudflare's fixed Web Bot Auth
  endpoint with the production crawler client.
• Refuses unsigned mode and does not use the database.

Pre-registration conformance may pass with the expected HTTP 401 result.
That is not Cloudflare approval.

After Cloudflare accepts JoshBot, operators should run:

joshbot conformance web-bot-auth --expect verified

Require the verified result before updating public documentation to say
Cloudflare Verified.

Current Cloudflare status

Cloudflare BotBase: submitted, awaiting review

JoshBot is not Cloudflare Verified in this release.

Crawler identity

User-Agent:

Joshternet-Joshbot (+https://joshternet.org/joshbot)

Public crawler information:

https://joshternet.org/joshbot

HTTP Message Signatures directory:

https://joshternet.org/.well-known/http-message-signatures-directory

Operator guidance

• Deploy with image tag joshbot:v1.1.0 and the existing upgrade procedure in
  deploy/README.md.
• Use docs/production-soak.md for production observation, discovery-funnel
  mapping, curated seed research, and evidence-gated follow-up issues.
• Do not invent speculative crawler features during soak. Let production
  behavior decide what comes next.

JoshBot v1.0.0

JoshBot v1.0.0 is the first supported release of the Joshternet discovery, verification, and public registry crawler.

What JoshBot does

• Crawls independently verified Joshternet origins and operator-curated seeds.
• Follows same-origin pages through a bounded breadth-first crawler.
• Turns external public links into independent verification candidates.
• Verifies declarations at /.well-known/josh.
• Stores verification, queue, lease, seed, and discovery state in PostgreSQL.
• Produces deterministic public registry snapshots.
• Publishes exact registry trees through an isolated GitHub API runtime.

Links create candidates. Valid declarations create participants.

Crawler identity

JoshBot uses the HTTP User-Agent Joshternet-Joshbot (+https://joshternet.org/joshbot).

Crawler behavior, retained data, non-archival behavior, resource limits, and robots controls are documented at:

https://joshternet.org/joshbot

Security and operations

JoshBot validates resolved network destinations, guards redirects, enforces robots policy, bounds response processing, and isolates database credentials from publication credentials.

The Docker Compose deployment includes migration, verification, discovery, export, publication, backup, restore, and recovery workflows.

Reporting problems

Unexpected crawler behavior can be reported here:

https://github.com/joshternet/joshbot/issues/new?template=crawler_report.yml

Software defects can be reported here:

https://github.com/joshternet/joshbot/issues/new?template=bug_report.yml

Security vulnerabilities must be reported privately:

https://github.com/joshternet/joshbot/security/advisories/new

Documentation

• Project documentation: https://github.com/joshternet/joshbot
• Joshternet specifications: https://github.com/joshternet/spec
• Changelog: https://github.com/joshternet/joshbot/blob/v1.0.0/CHANGELOG.md
• Deployment guide: https://github.com/joshternet/joshbot/blob/v1.0.0/deploy/README.md

JoshBot is licensed under the BSD 3-Clause License. Copyright belongs to Joshua Morris.