Skip to content

docs(ci): self-hosted MCP Toplist rank badge, refreshed monthly from upstream - #242

Merged
cdeust merged 3 commits into
mainfrom
docs/mcp-toplist-badge
Jul 28, 2026
Merged

docs(ci): self-hosted MCP Toplist rank badge, refreshed monthly from upstream#242
cdeust merged 3 commits into
mainfrom
docs/mcp-toplist-badge

Conversation

@cdeust

@cdeust cdeust commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a self-hosted MCP Toplist rank badge to the README, and the monthly cron that keeps its claim from going stale.

The badge is a committed static SVG, not a hotlinked remote image. A remote badge URL is a third-party beacon on every README view and lets its host restate our claim with no commit in our history; a committed file cannot drift. The cost of that choice is that it cannot self-update — the date is part of the claim and goes stale by inaction, and inaction never opens a PR. The second commit closes that loop with a monthly workflow that regenerates the SVG from upstream and proposes it as a PR.

Provenance is carried inside the SVG itself: source path, raw rank and field size, the percentile arithmetic, the date read, and a verify URL — so the claim is re-derivable from the file alone. mcptoplist.com ranked hypermnesia-mcp #964 of 81,919 tracked servers on 2026-07-28 → 964/81919 = 1.18% → "Top 1.2%".

Wording is deliberately attributive. Upstream's published methodology says its score "is a popularity and activity signal, not a quality assessment" and leaves ~25% of the weighting (organic reach, package downloads) undisclosed. The badge, its alt text and its title therefore all say Cortex is ranked in the top 1.2% by MCP Toplist — never that it is a top-1.2% server.

Closes #

Type of change

  • New feature (non-breaking change which adds functionality) — the refresh workflow
  • Documentation only — the badge asset + README line

Test plan

51 new unit tests, no network in any of them (the script's fetch seam is injected):

$ .venv/bin/python -m pytest tests_py/scripts/test_refresh_mcp_toplist_badge.py -q
...................................................                      [100%]
51 passed in 0.79s
  • All existing tests pass (full suite runs in CI on this exact tree).
  • New tests added for new behavior.
  • Mutation survival check: considered below.
  • Manual verification: rendered SVG checked in GitHub's sanitizer-equivalent path; upstream endpoints probed by hand (measurements quoted below).

What the tests would catch. Flipping any validator comparison (rank > 0>=, rank <= total<) is killed by the boundary cases; dropping the XML escaping is killed by the rank-1 case (see below); returning a successful parse from a malformed candidate shape is killed by the strict-shape refusals; swallowing a failed fallback path is killed by the assertions on the emission of the failure report, not only on the final value.

Audit notes

Two extraction paths, one validator.

  1. /data/leaderboard.json — the structured export the site links from its homepage. Returns HTTP 503 (measured 2026-07-28: 3/3 attempts, 8–14s each under a browser UA — a server-side generation timeout, not UA gating or rate limiting). Its schema has therefore never been observed, and writing a parser for a shape I have not seen would be invention. It accepts only documented candidate shapes under strict validation and refuses anything else rather than guessing.
  2. The server page's "ranks #N of M servers tracked" sentence. This is the only construct on that page carrying both numbers — the <title>, the og/twitter meta tags and all four JSON-LD blocks give the rank without the total, so none of them can yield a percentile.

Fail closed. The validator refuses a non-numeric, zero, negative, or beyond-the-field figure — guarding both the division in percentile() and the semantics of the claim. A figure that fails is never written: the script exits non-zero, the badge keeps its last good value, and the run goes red. A fallback is never silent — the failed path is reported even when a later one succeeds.

Found by the tests before shipping: a top-of-field rank renders the tier as Top <0.1%, whose unescaped < made the badge invalid XML. Text and attributes are now escaped, verified at the rank-1, mid-field and last-place boundaries.

Rendering constraints. Palette (#3b3129 / #a53e00) taken from assets/banner.svg. No <style> block and no external font — GitHub's SVG sanitizer strips both; textLength/lengthAdjust pin every string so a substituted font still fits.

⚠️ Operational caveat — BADGE_REFRESH_TOKEN

The workflow opens its refresh PR with secrets.BADGE_REFRESH_TOKEN || secrets.GITHUB_TOKEN. The secret is not currently set. With the default GITHUB_TOKEN, GitHub deliberately does not trigger workflows on the resulting PR, so main's required checks never start and the refresh PR cannot be merged until a maintainer pushes to its branch. This is documented at the step in the workflow file. Setting a PAT with repo scope as BADGE_REFRESH_TOKEN makes those PRs self-checking. This does not affect this PR — only the monthly ones it will open from 1 August onward.

Coding-standards compliance

  • §2.2 Layer dependency — N/A: a standalone build script under scripts/, imported by nothing in the package.
  • §3.2 No any / untyped dicts at boundaries — the fetched payload is validated into typed values at the trust boundary before any use.
  • §4.1 No file > 500 lines — refresh_mcp_toplist_badge.py is 406.
  • §4.2 No function > 50 lines.
  • §4.4 No function with > 4 parameters.
  • §7 Local reasoning preserved.
  • §8 Every constant sourced — the cron cadence, palette, rank/total and percentile all carry their source at the use site.
  • §9 No dead code, no TODOs without issue references.

Completion Ledger (§13.2)

Every code path introduced by git diff origin/main...HEAD, mapped to its asserting evidence.

Path / item Evidence
JSON export path — accepted documented shape test_refresh_mcp_toplist_badge.py shape-acceptance cases
JSON export path — undocumented/malformed shape → refuse strict-shape refusal cases (no guessing on an unobserved schema)
JSON export path — HTTP failure → fall through fallback-transition case; failure emission asserted, not only the downstream value
HTML sentence path — ranks #N of M matched parse cases at rank-1 / mid-field / last-place
HTML sentence path — sentence absent → fail closed non-zero exit asserted; badge file left untouched asserted
Validator — non-numeric / zero / negative / rank > total one refusal case per arm
percentile() — division guarded by validator boundary cases; no unguarded divisor reachable
Tier rendering — Top <0.1% XML escaping rank-1 case asserts well-formed XML (this bug was found by the test)
SVG write — byte-identical output → no-op workflow add-paths + create-pull-request no-op on identical file
Every failure mode emits an actionable signal (F1) each failure case asserts the report line, not a side effect
Nominal path stays quiet (G4) negative assertion on the success case
Regression test for the escaping bug (G2) rank-1 case fails on the pre-fix code
Determinism / isolation (G3) no network, no shared paths, injected fetch seam; 51/51 pass repeatedly
CHANGELOG entry (H4) CHANGELOG.md +1
Docs updated (H4) README.md +1 (badge line)
Boy-scout check (§14, H7) no defect surfaced in touched material during this change; none deferred
Concurrency (B1–B3) N/A — single-shot script, no shared mutable state, no concurrency introduced
Persisted-data compatibility (E3) N/A — no persisted format touched
Secrets (D3) N/A in the diff; BADGE_REFRESH_TOKEN is referenced by name only, never a value. Caveat disclosed above.
Hot-path measurement (C3) N/A — monthly cron, not a hot path
Advertised test count synced 6297 → 6348 (11 sites, 6 files) check_doc_claims.py --test-count 6348doc claims OK; CI Test (Python 3.12) green on cfec4da

CI history — one real failure, fixed

The first push failed Test (Python 3.12), the only job running the doc-claims gate. The 51 new tests moved the canonical suite count 6297 → 6348, which the repo pins in 11 places; the gate compares them against a live pytest --collect-only -q and failed the job. Fixed in cfec4da — all 11 sites updated, including the README badge URL. The CHANGELOG's historical Suite grows 6275 → 6297 is deliberately left alone: it records a past PR, not the current suite.

Not silently changed, flagged for the maintainer: the README test badge is a hotlinked img.shields.io URL — the very pattern this PR argues against. Its number was updated; converting the README's badge row to committed SVGs is a separate decision and is not in this diff.

Reviewer checklist

  • CHANGELOG.md updated.
  • Documentation updated (README).
  • No secrets / credentials / PII in the diff.
  • CI passes on the latest commit — 14/14 green on cfec4da.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BJt9KN23rUNQzwoJv4jRWw

cdeust and others added 3 commits July 28, 2026 19:30
Adds a static SVG badge recording that Cortex ranks in the top 1.2% of
MCP servers tracked by MCP Toplist, and links it to the source listing.

The asset is committed rather than hotlinked. A remote badge URL is a
third-party beacon on every README view and lets its host restate the
claim at any time with no commit in our history; a static file cannot
drift. The cost is that it cannot self-update, so the date is part of
the claim and must be re-checked when the rank moves.

Provenance is recorded in the SVG itself: mcptoplist.com ranked
hypermnesia-mcp #964 of 81,919 tracked servers on 2026-07-28, giving
964/81919 = 1.18% -> "Top 1.2%", with the verify URL alongside it.

Wording is deliberately attributive. Per their published methodology the
score is "a popularity and activity signal, not a quality assessment",
and ~25% of its weighting (organic reach, package downloads) is
undisclosed -- so the badge, alt text and title all say Cortex is ranked
in the top 1.2% by MCP Toplist, never that it is a top-1.2% server.

Palette (#3b3129 / #a53e00) is taken from assets/banner.svg. No <style>
block or external font, since GitHub's SVG sanitizer strips both;
textLength/lengthAdjust pin every string so substituted fonts still fit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJt9KN23rUNQzwoJv4jRWw
The badge is a committed file, so it cannot restate itself without a commit
here -- and equally cannot self-update. Its date is part of the claim and
goes stale by INACTION, which never opens a PR. This closes that loop.

Monthly, not weekly: the badge stamps a month, so that is the least frequent
cadence keeping the date honest and the most frequent one that does not
churn a PR proposing an identical file. No pull_request trigger -- a
required check must not depend on a third party's uptime. The 51 unit tests
cover the logic with no network at all.

Two extraction paths, both feeding one validator:

  1. /data/leaderboard.json -- the structured export the site links from its
     homepage. Returns HTTP 503 (measured 2026-07-28: 3/3 attempts, 8-14s
     each under a browser UA, so a server-side generation timeout rather
     than UA gating or rate limiting). Its schema has therefore never been
     observed, and writing a parser for a shape I have not seen would be
     invention -- so it accepts only documented candidate shapes under
     strict validation and refuses anything else rather than guessing.
  2. The server page's "ranks #N of M servers tracked" sentence. This is the
     ONLY construct on that page carrying both numbers: the <title>, the
     og/twitter meta tags and all four JSON-LD blocks give the rank without
     the total, so none of them can yield a percentile.

Fail closed. The validator refuses a non-numeric, zero, negative or
beyond-the-field figure, guarding both the division in percentile() and the
semantics of the claim. A figure that fails is never written: the script
exits non-zero, the badge keeps its last good value, and the run goes red. A
fallback is never silent -- the failed path is reported even when a later
one succeeds.

Found by the tests before shipping: a top-of-field rank renders the tier as
"Top <0.1%", whose unescaped '<' made the badge invalid XML. Text and
attributes are now escaped, verified at the rank-1, mid-field and last-place
boundaries.

Wording stays attributive. Upstream's methodology page says its score "is a
popularity and activity signal, not a quality assessment" and keeps ~25% of
the weighting undisclosed, so the badge says Cortex is RANKED in this tier,
never that it IS -- and each generated SVG carries the source, raw figures,
arithmetic and verify URL in a comment so the claim is re-derivable from the
file alone.

Caveat recorded in the workflow: PRs opened with the default GITHUB_TOKEN do
not trigger workflows, so main's required checks will not start on them. An
optional BADGE_REFRESH_TOKEN secret is preferred when present.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJt9KN23rUNQzwoJv4jRWw
The suite's advertised count is pinned in six files and gated by
scripts/check_doc_claims.py, which the Python 3.12 job runs against a live
`pytest --collect-only -q`. This branch's 51 new tests moved the canonical
count 6297 -> 6348, so every pinned site disagreed with the repository and
the gate failed the job -- correctly, and on my change.

Updated: the README test badge and its alt text, the README and CONTRIBUTING
run-command comments, CLAUDE.md's test line, the assurance case's CI claim,
and the four .bestpractices.json justifications that quote the figure.

The CHANGELOG's historical "Suite grows 6275 -> 6297" is deliberately left
alone: it records what a past PR did, not what the suite holds now.

Verified: `check_doc_claims.py --test-count 6348` reports "doc claims OK",
and the local collection agrees with CI's at 6348.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BJt9KN23rUNQzwoJv4jRWw
@cdeust
cdeust merged commit 5eedfd9 into main Jul 28, 2026
14 checks passed
@cdeust
cdeust deleted the docs/mcp-toplist-badge branch July 28, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant