malwareremoval-archive-toolkit

Not a programmer? Start with START_HERE.md instead of this file. Everything below assumes you're comfortable with git, Python, and a terminal, or that you're an AI coding assistant reading this on someone else's behalf.

A working methodology and script set for turning a live phpBB forum backup into a scrubbed, static, read-only public archive: extract the data, exclude the areas that were never public, remove personally identifying information, rehost attachments, audit twenty years of outbound links for rot and malware, and generate the actual static HTML site from the result — search included, no server-side dependency required by default.

It was extracted from a real migration — malwareremoval.com, a security/tech-support forum being retired and replaced with a static archive of its own history. The board name is public; nothing here identifies any individual member of it. See docs/known-limitations.md for exactly what's phpBB-specific versus general.

What this actually is

This is a methodology and a working example, not a turnkey tool. No two forums are identical — different schema, different BBCode dialect, different quirks in how content got migrated or mangled over the years. Every script here was built by reading real rows from a real database and iterating until the output looked right, not designed in the abstract. Expect to do the same against yours.

That said, it gets you most of the way there:

• By hand, following MIGRATION_GUIDE.md and adapting the scripts in scripts/ and generator/ to your own schema, this covers roughly 95% of the actual work: the architecture decisions, the PII-scrubbing rule classes (and their real failure modes), the forum-exclusion derivation, the attachment-handling approach, the link-audit methodology, the static-site generator itself, and the verification discipline that catches what a first pass misses.
• With an LLM coding agent (Claude Code or similar) doing the schema-specific adaptation and iteration against your actual data, it goes faster still — read AGENT_PRIMER.md first if you're taking this route. It's not forum-migration-specific; it's the general operating discipline (long-running job handling, verification habits, authorization boundaries) that mattered as much as the code on the reference migration.

Structure

malwareremoval-archive-toolkit/
├── START_HERE.md             — not a programmer? Read this one first, not the rest.
├── README.md                — this file
├── MIGRATION_GUIDE.md        — the methodology: architecture, PII scrubbing (including
│                               regex bug classes worth checking for directly, and a
│                               dedicated section on the sharper risk profile of
│                               corpus-wide name propagation specifically), forum
│                               exclusion, attachments, link audit, static generation,
│                               verification discipline. Every example is synthetic.
├── AGENT_PRIMER.md           — operational lessons for whichever coding agent works
│                               from this repo. Generic — not forum-specific.
├── LICENSE                   — MIT
├── scripts/                  — extraction/scrubbing/export pipeline, genericized
│                               (env-driven DB config, no real board data anywhere)
├── generator/                — the static-site generator: Jinja templates, Python
│                               render scripts, and CSS that turn the scrubbed JSON
│                               export into an actual HTML site with client-side
│                               search (Pagefind) — no Cloudflare or any other specific
│                               host required. See generator/content/ for the
│                               homepage-story placeholder you'll want to replace.
├── config/
│   └── *.example.json        — config files the scripts/generator read, as
│                               .example.json with placeholder values. Copy to the
│                               real filename (e.g. forum_exclude.example.json →
│                               forum_exclude.json, site.example.json → site.json)
│                               and fill in your own board's data before running.
├── worker/                   — OPTIONAL advanced addendum: a Cloudflare Worker +
│                               D1 full-text search, for boards large enough that
│                               Pagefind's client-side index becomes impractical.
│                               Most boards don't need this — see worker/README.md
│                               before reaching for it.
└── docs/
    └── known-limitations.md  — what's phpBB-3.0.x-specific, what a different
                                 platform (IPB, vBulletin, Discourse, ...) needs to
                                 rebuild instead of adapt

Quickstart

1. Get your forum's database into a MySQL/MariaDB container you can query. Load a filtered dump if you have one; work against a scratch copy, never production directly.
2. Set the connection env vars db.py reads:

   export DB_CONTAINER=your-db-container
   export DB_NAME=yourboard
   export DB_USER=root
   export DB_PASSWORD=changeme

3. Copy every config/*.example.json to its real filename (drop .example) and replace the placeholder values with your own: which forums are actually public (or derive this — see step 4), which user IDs need anonymizing, which are confirmed spam, your board's own domain/URL scheme/pagination settings (site.json).
4. Read MIGRATION_GUIDE.md §2 and adapt derive_forum_exclusions.py to your platform's permission model; run it to produce config/forum_exclude.json.
5. Work through the PII-scrubbing scripts (§3): structured field drops (export_users.py), in-content regex scrubbing (pii_scrub.py), spam detection (scan_spam_accounts.py, sfs_lookup.py — read the authorization note in AGENT_PRIMER.md §6 before running this one), and real-name-shaped username review (scan_username_realname.py). If you're building a corpus-wide name-propagation feature (§3f) rather than only a curated anonymize-list, budget real time for iterating on it — it's a genuinely sharper risk profile than the curated-list case, not a drop-in extension of it.
6. Handle attachments (§4): attachments_export.py, attachments_pii_scan.py, apply_attachment_redaction.py.
7. Run the link audit (§5): extract_external_urls.py, then url_reachability_check.py, redirect_parked_check.py, llm_content_classify.py, safe_browsing_check.py, urlhaus_check.py.
8. Render and join (§6): adapt bbcode_render.py to your platform's markup, then export_site_json.py to produce the final per-topic JSON.
9. Run verify_json_export_pii.py last, and keep it in your rebuild pipeline permanently — it's a build-blocking assertion, not a one-time check. See MIGRATION_GUIDE.md §7 for why this catches an entire class of bug that upstream calibration structurally can't.
10. Generate the static site (§8): replace generator/content/homepage_story.example.html with your own board's real story (copy it to homepage_story.html first), replace generator/static/logo.png/favicon.png with your own, then run python3 generator/build.py — writes the full HTML site + a client-side Pagefind search index to output/site/. Upload that directory to any static host.
11. Optional, only if your board is large enough that Pagefind's client-side index becomes impractical (very roughly: high hundreds of thousands of posts and up — measure against your own corpus rather than assuming): the worker/ addendum swaps in a Cloudflare Worker + D1 full-text search. See worker/README.md before reaching for this — it's genuinely more infrastructure, not a strict upgrade.

Every script's own docstring documents its specific inputs/outputs and any quirks found while building it against real data — read the script before running it, not just this list.

What this doesn't cover

Hosting/deployment automation beyond "upload output/site/ to a static host" — which specific host, CDN configuration, custom domain setup, etc. are left to you, since they vary by provider and don't need forum-migration-specific guidance. The optional worker/ addendum is the one exception, included because it's genuinely tied to this project's own PII/verification lessons (see its README).