JoshBot v1.1.0

JoshBot v1.1.0

JoshBot v1.1.0 is a backwards-compatible minor release. It turns the
post-v1.0.0 operational, reliability, pacing, and authenticated-crawler work
into a supported release boundary for production soak.

Safer and more controllable discovery

• Private operational reporting API with bearer authentication, status,
  sources, crawls, queue, services, audit history, and OpenMetrics /metrics.
• Independent operator control API for processor pause/resume, domain-avoid
  rules, exact-origin block/allow, and append-only audits.
• Source detail reporting with provenance, queue state, latest crawl, and
  stored robots observations.
• Keyset pagination via X-JoshBot-Next-Cursor.

Bounded automatic expansion

• Automatic crawl-source admission with per-run promotion budgets.
• Deferred candidates and pending-probe backpressure so expansion cannot
  silently overwhelm verification.

Crawler reliability

• Shared durable retry policy for transient discovery and verification
  failures.
• Discovery source claims use expiring leases so crashed crawls can be
  reclaimed.
• Worker and discovery loops no longer park unrelated due work behind a
  single origin retry schedule.
• Verification lease renewal and completion use authoritative lease budgets.
• Redirect-loop tracking is local to each fetch attempt so transient
  same-origin redirects can be retried safely.
• Crawl service heartbeats use stable logical service slots.

Crawl-delay and request pacing

• Robots Crawl-delay parsing with JoshBot-over-wildcard precedence,
  fractional seconds, and explicit invalid-value reporting.
• Shared per-origin request scheduling uses the greater of the configured
  request delay and the applicable robots Crawl-delay.

Authenticated crawler identity (Web Bot Auth)

• Ed25519 HTTP Message Signatures with JWK-thumbprint key IDs, short-lived
  signatures, and per-request nonces.
• Dedicated signing-key configuration, optional transition identity, and
  required / unsigned modes.
• Required mode fails closed before crawler work begins.
• Signing is applied at the shared outbound crawler HTTP boundary.
• Private key material stays out of environment dumps and public JWK output.

BotBase conformance tooling

• joshbot conformance web-bot-auth --expect unregistered|verified
• Validates the public signature directory and Cloudflare's fixed Web Bot Auth
  endpoint with the production crawler client.
• Refuses unsigned mode and does not use the database.

Pre-registration conformance may pass with the expected HTTP 401 result.
That is not Cloudflare approval.

After Cloudflare accepts JoshBot, operators should run:

joshbot conformance web-bot-auth --expect verified

Require the verified result before updating public documentation to say
Cloudflare Verified.

Current Cloudflare status

Cloudflare BotBase: submitted, awaiting review

JoshBot is not Cloudflare Verified in this release.

Crawler identity

User-Agent:

Joshternet-Joshbot (+https://joshternet.org/joshbot)

Public crawler information:

https://joshternet.org/joshbot

HTTP Message Signatures directory:

https://joshternet.org/.well-known/http-message-signatures-directory

Operator guidance

• Deploy with image tag joshbot:v1.1.0 and the existing upgrade procedure in
  deploy/README.md.
• Use docs/production-soak.md for production observation, discovery-funnel
  mapping, curated seed research, and evidence-gated follow-up issues.
• Do not invent speculative crawler features during soak. Let production
  behavior decide what comes next.