Claude/startup stack GitHub scraper mbp53h#1
Merged
Conversation
Three additions, none of which change how the stack itself works. **The site.** `web/build.mjs` generates a static site into `_site/` from the repository's own markdown — README, docs, prompts, worksheets and the stack templates' front matter. There is no second copy of the prose, so the site cannot drift from the repo: edit the markdown and the page changes. Zero dependencies, Node 18+. The build fails rather than ship a wrong page: a broken internal link, a missing diagram, a page with no title, a prompt not routed to a group, or a diagram whose colours have drifted from the tokens all stop it. `.github/workflows/pages.yml` publishes it on push to main. Filterable prompt library, per-prompt copy button, command-K search, light and dark, WCAG AA in both. **The scraper.** `tools/scrape-site.mjs` turns a website — the founder's own or a competitor's — into citable markdown in `_inbox/`, ready for the bootstrap prompt. Zero dependencies. Its real job is provenance: every page carries its source URL and fetch date, and everything it writes defaults to [NEEDS VERIFICATION], because a website is a company's published claim about itself. robots.txt is fetched and obeyed, requests are serial with a delay, and no flag turns either off. It never silently truncates — `index.md` lists every URL fetched, skipped or failed, with the reason for each. `prompts/15-scrape-a-site.md` drives it and routes the findings, surfacing conflicts between the site and the stack rather than resolving them. `docs/scraping.md` covers what not to scrape. **AI basics.** `docs/ai-basics.md` is for a founder who has only ever used a chat window: what an AI with access to a folder actually is, the brief-plan-build-review loop, the terminal in plain language, what it genuinely cannot see, why an index makes every question cheaper, and the one rule that never bends. Six hand-built SVG diagrams carry the concepts; they use CSS variables with literal fallbacks, so the same file renders correctly on GitHub and re-skins itself in dark mode on the site. Also: AGENTS.md now says how to cite a scraped page and treat it as a conflict rather than a correction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
An adversarial pass over web/build.mjs found real defects. What changed: - **Double-escaped URLs.** inline() escaped the whole line before the link and image regexes captured the href, so `?a=1&b=2` shipped as `?a=1&amp;b=2` — a browser reads that as a parameter named `amp;b`. Repo-relative targets were corrupted the same way before being checked against the filesystem, so a valid link could fail the build. The href is now un-escaped before it is resolved, and escaped exactly once on the way out. - **404.html used page-relative paths.** GitHub Pages serves it for a URL at any depth, so its stylesheet, script and links broke on anything below the root. It now uses the site's absolute path prefix. - **The header promised a check that did not exist.** It claimed the build fails on a page whose title cannot be found; there was no such check. Added it, for docs pages, prompts and worksheets. - **Links in the prompt index escaped validation.** The tables there are replaced by the live grid, so their links were never checked. The whole file is now rendered into a throwaway buffer to validate them. - **Heading ids could collide,** sending two anchors to the same section. Now counted per page. - **Percent-encoded link targets** are decoded before the filesystem check. - **`data:` URLs** are no longer passed through into an href. - **A fenced block containing a fence** closed early; only a line that is nothing but fence characters closes one now. **Autolinks containing `&`** were not recognised, because the pattern ran after escaping. Also, in the design tokens: the light hero's lightest gradient stop put body text at 3.0:1, below AA, and the diagram arrows sat at 2.1:1 — under the 3:1 that a graphical object carrying meaning needs. Both corrected, with every stated contrast ratio in the file recomputed from the actual values rather than inherited. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
An adversarial pass over the tool found real defects. The two that mattered: - **Relative links resolved one level too high.** They were resolved against the normalised URL, which has had its trailing slash stripped, so on a directory-style page — `/blog/`, the commonest shape on the web — a link to `post.html` resolved to `/post.html` instead of `/blog/post.html`. Whole sections of a site would have been missed. Links now resolve against the address the server actually served. - **Only the first `User-agent: *` group in robots.txt was read.** Records sharing a user-agent are one group; keeping the first dropped the Disallow rules and Crawl-delay in the others. They are merged now. Also: a redirect is re-checked against robots.txt at the address it landed on, rather than only the address requested; `/a/` and `/a/index.html` now normalise to the same URL (the index rewrite has to precede the trailing-slash strip); a malformed percent escape in a path no longer crashes the run; the depth-capped count in `index.md` counts URLs rather than link occurrences; `&constructor;` and friends no longer resolve through the prototype chain; and a single unclosed heading tag no longer turns the rest of the page into headings. The output directory is still never cleared — it may hold pages a founder saved by hand, and a tool that quietly deletes their evidence is worse than one that leaves a stale file. It now says what it found and lets them decide. Three claims in the documentation were wrong and are corrected: a second run merges with the previous crawl rather than replacing it, "the four rules above" counted five, and an arithmetic aside did not follow from the figures beside it. One unsourced claim about what "most founders" find is now described without the statistic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
Two of a set that answers the words a founder meets while running this method, at the density the prose cannot reach. `how-it-fits-together.svg` draws the four layers — you, the AI tool, the command line, your computer — with the request travelling down and the result coming back up, and the approval gate drawn as an explicit boundary between the top two. Nothing crosses it without a yes. `what-is-the-terminal.svg` shows the shape of a terminal window using this repository's own folder as the example, then answers the four things a beginner actually asks: what a command is, what the output is, why it looks so bare, and who types them. It closes on the point the prose makes too — you are not learning to type commands, you are learning to read one before you say yes. Both consume only the --dg-* token contract with literal fallbacks, so they render light on GitHub and re-skin themselves in dark mode on the site. The page that carries them, and the six that go with them, follow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
The workflow only fired on a push to main, so the build was never exercised before a merge — and because the file has never reached main, GitHub has not registered it at all. Now the build job runs on every branch and every pull request. It fails on a broken internal link, a missing diagram, a page with no title, a prompt that is not routed to a group, or a diagram whose colours have drifted from the design tokens, so those are caught before review rather than after. It also checks that the scraper parses and answers, since the tool ships in the repo. Publishing is unchanged and still gated to main. Two things remain that cannot be done from here: this has to reach main before GitHub will register the workflow, and a repo admin has to set Settings → Pages → Source to "GitHub Actions" once. The Actions token cannot enable Pages itself, so the first deploy fails without it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
Three more of the set, covering the words QUICKSTART.md already uses without ever explaining them. `what-is-an-editor.svg` draws the three panes — file tree, file contents, chat panel — with an approve/reject pair in the panel, then the four capabilities that separate an agentic editor from a chat window with an upload button. The file tree deliberately holds notes, a PDF and a spreadsheet rather than source files, because that is what is actually in a founder's folder. `what-is-git.svg` frames a commit as a save point for the whole folder, on a timeline of four, with the changed and unchanged lines marked so it is visible that each point holds everything rather than one file's history. Then the only three words needed — commit, diff, clone — and the habit this repo asks for: commit, run the prompt, read the diff before you keep it. `what-is-github.svg` answers the confusion that Git and GitHub are one thing. Two columns, the two moves that connect them, and what the second copy buys you. It carries a warning card that matters here more than in most repos: a knowledge base holds customer names, prices and contracts, so check whether the repository you made is private before you push. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
…utton The serious one first. The build put the `summary:` line of every stack section onto the public page and into the search index, with no check on `sensitivity:`. AGENTS.md rule 5 says to pull from `public` freely, from `internal` only on an explicit instruction, and never from `restricted` — and a summary line is exactly where a founder is told to put the routing facts: the margin, the runway, the customer name. Nothing leaks from this repo, whose sections are still empty templates, but anyone who forks it and keeps `web/` would have published their own. The README told them to delete `web/`; that is an instruction, not a guarantee. The rule is now enforced in code. A summary reaches the site only if the file is declared `public`, or if it is still the shipped template — status `tbd` with no owner named — where the line documents the template rather than describing a company. Anything held back is named in the build output rather than dropped silently, and the section still appears with its title and its sensitivity; only the summary is withheld. Verified by filling in a section with a margin and a runway and confirming neither string appears anywhere in the built site. That surfaced a front-matter parser bug worth fixing on its own: a quoted value was unquoted before the inline-list check, so `owner: "[TBD]"` was parsed as the one-item list `["TBD"]`. A quoted value is a string, always. Also from the same pass: - The copy button sat inside the `<pre>`, which scrolls sideways, so it slid off with the code and covered the first line on touch devices where it is always visible. Copyable blocks are now wrapped in a container that does not scroll, with room reserved at the top, and the button is no longer part of a hand-selection of the code. - `figure-wide` floored diagrams at 620px. In a 1200-wide viewBox that is 52%, painting the smallest labels at under 8px. The floor is now set from legibility: 880px, where a 15px label still lands near 11px. - The head script restored the theme but not the sidebar state, so a collapsed rail painted open and then snapped shut. - Search hits for a stack section landed at the top of the page. They now point at the section's own row. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
… toolkit `what-is-a-connector.svg` draws one adapter between the AI and the systems a founder already has — mail, a drive, a calendar, a tracker, a repository — set against the tangle of custom wiring it replaces, with the three habits that keep it safe: read-only first, scoped to what the task needs, and with permission. `model-tiers-and-effort.svg` separates two controls that beginners conflate. The tier ladder is labelled by the kind of work rather than by product name — routine reformatting, everyday drafting, ambiguous and high-stakes, the hardest work — because every provider publishes its own names for those rungs and changes them; the ladder does not. Beside it, the effort dial, which changes one answer rather than a subscription. The worked examples are this repo's own tasks: filing the inbox, drafting the recap, deciding what a unit costs. `beyond-the-knowledge-base.svg` answers the four words a founder keeps hearing — folder, repository, host, domain — as a chain where each stage adds exactly one thing, then sets a database deliberately outside that chain, because assuming a host remembers what visitors type is the misconception worth naming. It closes on the scope line: a knowledge base needs none of it. It is a folder of text files. The words are here so they are not a mystery the first time you meet them. No product names anywhere in the three, so none of them goes stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cLKVBJb2hPNi6QALTFPFk
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (14)
📒 Files selected for processing (16)
📝 WalkthroughWalkthroughThis PR adds a Node-based static site generator, themed interactive web assets, a website scraping CLI with evidence controls, documentation and prompts for AI-assisted workflows, and GitHub Pages build/deployment automation. ChangesStatic site and publishing
Website research workflow
AI workflow and onboarding
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Repository
participant GitHubActions
participant GitHubPages
Repository->>GitHubActions: trigger build on push, pull request, or dispatch
GitHubActions->>GitHubActions: run node web/build.mjs and scraper checks
GitHubActions->>GitHubPages: upload and deploy _site from main
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit