Skip to content

Stop shipping build output that nothing can request - #3280

Merged
borland merged 9 commits into
mainfrom
orion/housekeeping
Aug 5, 2026
Merged

Stop shipping build output that nothing can request#3280
borland merged 9 commits into
mainfrom
orion/housekeeping

Conversation

@borland

@borland borland commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What this is

I downloaded a copy of the built package (DocsMicrosite.20260804.503-pullrequest.1) and sent claude on a mission to find and fix any unreachable content.

This PR removes the reachable-by-nothing files and, where the build was generating them, fixes the build so they stop coming back.

How to review this PR

One commit per concern, so they can be reviewed (or dropped) independently.

Results (Claude-generated, with replies from me)

Remove the dead dist-thinner script and a stray debug log

dist-thinner.mjs is not wired into build or any workflow, so it has never run in CI. It is also actively dangerous if anyone ever does run it: it walks dist/docs/img and, for every image whose metadata updated timestamp is more than 14 days old, deletes the original, the /docs/i/x fallback, the metadata file and all three resized variants. Almost every image qualifies, so npm run thin would strip the site of its images rather than thin anything. Also drops a leftover console.log('rehypeWbr = …') that printed on every dev server start and build.

Orion: Verified everything looks good in the staging site and locally, so this must be OK

Delete 367 orphaned images — 27.5 MB

Referenced by nothing: not by any page in the built site, and not by any file under src/. Leftovers from content that has since been rewritten or removed. Source images only — their generated variants and metadata sidecars are pipeline output and are cleaned up in the other PR, so that all changes to generated image output stay on one branch.

Orion: Verified everything looks good in the staging site and locally, so this must be OK

Delete unused Font Awesome webfonts — 5.7 MB

Only two families are referenced: HEADER_SCRIPTS in config.ts loads fontawesome.css, brands.css and solid.css, and the @font-face rules in styles/vars.css point at fa-solid-900 and fa-brands-400. Duotone, light, thin, regular, sharp-solid and v4compatibility shipped in both .ttf and .woff2 for 5.7 MB no stylesheet asks for. The .ttf fallbacks for the two families we do use are kept, since vars.css declares them deliberately.

Orion: Verified everything looks good in the staging site and locally, so this must be OK

Delete a duplicated copy of autocomplete.gif

public/docs/images/autocomplete.gif is byte-identical to public/docs/shared-content/images/autocomplete.gif, and the shared-content path is the one the markdown links to.

Orion: Verified everything looks good in the staging site and locally, so this must be OK

Move the inline-SVG icon components out of public/

Arrow, List, MoonIcon, OctopusLogo, SunIcon and Youtube are Astro components imported by Card, Header and ThemeSwitcher — but they were sitting in public/docs/img/. Everything under public/ is copied verbatim into the deployed package, so each one was compiled into the bundle and published as raw component source at e.g. /docs/img/OctopusLogo.astro. Moved to src/components/icons/.

Orion: Verified everything looks good in the staging site and locally, so this must be OK. E.g. SunIcon appears in the theme switcher and it's very obvious to check that it looks as-expected

Move the Dia diagram sources out of public/

Ten .dia files (editable Dia sources for the Kubernetes diagrams) sat next to the .svg files they export to. Nothing links to them, but public/ meant they were uploaded on every deploy. Moved to a new top-level assets/ with a README explaining why source artifacts stay out of public/. The exported .svg files pages actually reference stay put.

That directory is also excluded from cspell: Dia stores its documents as uncompressed XML, so pnpm spellcheck read them as prose and reported its internal attribute names (lmargin, hguides, autorouting, …) as misspellings — 131 of them across the ten files. Excluding by directory rather than by extension keeps the README spellchecked and covers whatever format the next diagram source arrives in.

Orion: These dia files were never browseable in the docs, moving them is fine

Stop shipping build output that cannot be served

astro.config.mjs already notes the site is proxied onto octopus.com under /docs/ only — that is why build.assets was moved to docs/_astro. But Astro still routes src/pages/index.md, components.mdx, ui-update-sample.mdx and report/** to the site root, so every deploy uploaded /index.html, /components/, /ui-update-sample/, /report/** and /css/report.css. None of those paths are reachable in production.

They are useful under astro dev, which does serve the root, so rather than moving the pages this adds a prune-dist integration that strips everything outside docs/ once the build is done. The rule is the deployment invariant itself, so it keeps holding as pages come and go. It also caught a stray public/.DS_Store that was being published.

Orion: I checked the reports when you run pnpm dev and everything still looks fine
Update: It was a mistake to use pnpm dev to check this, as the pruner only runs as part of the build, not dev. On the staging site, the index.html vanished which was bad. I added another commit to the PR and put it back.

src/pages/varying-azure-subscription-by-environment.md is deleted outright — a root-level redirect stub, already unreachable before this change; the same redirect exists at its real path under /docs/deployment-examples/azure-deployments/.

Orion: I checked the staging site and everything is unchanged. There's a real varying-azure-subscription-by-environment in another folder which is what everyone sees and that didn't change

How the orphan list was verified

Extract every /docs/img/... reference from the built output and from repo source, then diff against what is on disk. Two things a naive scan gets wrong, both of which would have deleted live images:

  • Absolute URLs count. JSON-LD builds its logo URL as `${SITE.subfolder}/img/octopus-logo.png` and emits it as https://octopus.com/docs/img/octopus-logo.png. Scanning only for site-relative paths reports octopus-logo.png as unused, when it is referenced from 1,261 pages. My first pass did exactly this and deleted it; the reference-resolution check below caught it, and I redid the analysis.
  • Filenames with parentheses (e.g. deploying-an-azure-web-app_(,2018.5).png) need an extension-anchored match rather than a delimiter-terminated one.

Then, as an independent check, resolve every asset reference in the built output against the files on disk.

Verification

check result
asset references resolved against files on disk 20,206 checked, 0 missing
npm run crawl (linkinator) 5,390 links, 0 broken
npx playwright test 99 passed, 0 failed
astro build clean, 2,696 pages

Deliberately left alone

  • 33 images (~3 MB) referenced only from pages that exist in src but not in the deployed site (mostly ui-update-sample.mdx). They ship, but deleting them would break pages that still reference them.
  • ~1.7 MB of byte-identical duplicate images in 13 groups (e.g. img/octopus-recommendations/… vs img/shared-content/octopus-recommendations/…). Consolidating needs content edits across many markdown files — not worth the risk for 1.7 MB, but worth a follow-up.
  • .ttf Font Awesome fallbacks (~630 KB) are redundant for any browser since 2016, but they are a deliberate @font-face fallback in vars.css, so that is your call.
  • /docs/icons/ favicon variants (~36 KB). android-chrome-192x192.png, android-chrome-512x512.png, favicon-16x16.png and favicon-32x32.png are unreferenced — there is no web manifest — but they are a conventional favicon bundle and some crawlers probe them, so I left them.

Two pre-existing bugs I found but did not fix

Both need a content or design decision rather than a mechanical fix:

  1. src/pages/docs/search.md sets bannerImage: /devops/img/search.png, which does not exist and is not even under /docs/. The search page's og:image is a 404. Dropping the bannerImage block would fall back to the OPEN_GRAPH default, but that changes the page's social card.
  2. src/themes/octopus/layouts/ and most of src/themes/octopus/components/ is dead source. Every page uses src/layouts/Default.astro; only Search.astro and the utilities/ are imported from the theme tree. It is not build output, but it is a duplicated tree (two Default.astro, two Related.astro, two ArticleList.astro…) that will keep confusing people. src/data/footer.ts — and therefore img/octopus.svg, which I kept — is only reachable through it.

Separately, components.mdx references /docs/img/icon-example.svg, which does not exist on main either. On this branch the problem disappears because that page is no longer published.

🤖 Generated with Claude Code

@team-marketing-branch-protections

Copy link
Copy Markdown

Pull request environment is available at https://stoctodocspr3280.z22.web.core.windows.net.

You can view the ephemeral environment status in Octopus Deploy.

This environment will be automatically deprovisioned when the pull request is closed, or after 7 days of inactivity.

@borland
borland force-pushed the orion/housekeeping branch 2 times, most recently from 560cb17 to 7ea7dc9 Compare August 5, 2026 03:12
@borland
borland requested a review from enf0rc3 August 5, 2026 05:05
@enf0rc3

enf0rc3 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

This change removes the index page that we use on the staging site above:

Pull request environment is available at https://stoctodocspr3280.z22.web.core.windows.net/.

You can view the ephemeral environment status in Octopus Deploy.

This environment will be automatically deprovisioned when the pull request is closed, or after 7 days of inactivity.

Personally i do get value out of the index.html and the reports too thats handy to see, there shouldnt be much harm including those

Agree on the large assets though

If we choose not to include the index, we should address the bots link above

@enf0rc3

enf0rc3 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review by Claude Code, posted on behalf of @WillLaugesen.

Summary: five commits look good, 7ea7dc91 has a regression

I verified the risky parts of this PR independently. The image deletions, icon moves, .dia moves and dist-thinner removal all check out. The prune-dist commit breaks the ephemeral/staging sites and the measured payoff is 628 KB, so my recommendation is to drop it.


Blocker: the /docs/ invariant is production-only

The premise is "the site is proxied onto octopus.com under /docs/, so root output is unreachable." That holds for production. It does not hold for the ephemeral environments.

branch-validation-full.yml deploys every PR to a pr<N> environment (torn down by deprovision-ephemeral-environment.yml on close). Those hosts serve the package from their own root, so /index.html, /report/taxonomy, /report/missing-meta, /report/oldest-content and /components are all live there and are used for debugging.

src/pages/index.md states this in its own frontmatter — "This page is not shown on the production site" — and links to /docs. It is the staging entry point, and prune-dist deletes it.

The verification in the PR description covers astro dev only ("I checked the reports when you run pnpm dev"). astro:build:done never fires under astro dev, so that check could not have caught this.

The payoff is 628 KB

Measured against a local dist/:

entry size
dist/docs 808,858 KB
dist/report 339 KB
dist/components 116 KB
dist/ui-update-sample 88 KB
dist/index.html 80 KB
dist/css + redirect stub 5 KB

628 KB out of 809 MB — 0.08% of the package. The image commit in this same PR saves 27.5 MB, 44× more, with no downside.

If you want this later, do it at packaging

Both deployments run the same astro build, so a build-time hook cannot tell them apart. The split has to happen where the two workflows already diverge — the packaging step:

# microsite-deployment-full.yml only
- uses: OctopusDeploy/create-zip-package-action@v4
  with:
    base_path: "./dist"
    files: |
      docs/**/*.*

Leave microsite-deployment-branch.yml packaging everything. Production ships lean, staging keeps its root, and no build-time file deletion is needed. That is a PR against OctopusDeploy/microsite-deployment.

Worth noting for anyone reaching for a pure-Astro fix: astro:routes:resolved cannot help here. integrations/hooks.js:502 passes routes.map(toIntegrationResolvedRoute(...)), a freshly derived array, so mutating it has no effect on the build.

Worth keeping from that commit

  • Deleting src/pages/varying-azure-subscription-by-environment.md is safe. Nothing links to it, and the real redirect lives at src/pages/docs/deployment-examples/azure-deployments/varying-azure-subscription-by-environment.md.
  • Revert the crawl change back to linkinator ./dist. It exists only to work around the pruned root index.html.

Three smaller findings

  1. The description covers a commit that is not in this PR. "Delete unused Font Awesome webfonts — 5.7 MB" is written up in detail, and the 393-file diff contains zero .woff2, .ttf or css/fa/ changes. Either the commit was dropped or the description came from a different branch.
  2. The .DS_Store claim does not hold. .DS_Store is on .gitignore:23 and untracked, so a CI checkout never contains one. Local builds only.
  3. // Must run last is load-bearing. llmMdEmitter and pruneDist share the astro:build:done hook and are ordered purely by position in the integrations array. Moot if the commit is dropped.

Verified clean

  • 367 image deletions. I converted every deleted path to its URL form and grepped all of src/: 0 hits. I also confirmed there are 0 relative image references anywhere in the markdown, so absolute-path matching gives complete coverage. The octopus-logo.png near-miss described in the write-up is genuinely handled.
  • Icon component moves. All six imports across Card.astro, Header.astro and ThemeSwitcher.astro are updated, with no stale public/docs/img/*.astro references remaining.
  • dist-thinner.mjs. Referenced only by the thin script, which this PR removes. The assessment is correct — it deletes any image whose metadata updated timestamp is over 14 days old, which is nearly all of them.
  • .dia moves and the cspell exclude. Sound. The exported .svg files pages reference stay in public/.

Suggested action

Drop 7ea7dc91, keep the redirect-stub deletion and revert the crawl change, and the remaining five commits are good to merge.

borland and others added 7 commits August 5, 2026 22:03
`dist-thinner.mjs` is not wired into `build` or any workflow, so it has
never run in CI. It is also actively dangerous if anyone ever does run it:
it walks `dist/docs/img`, and for every image whose metadata `updated`
timestamp is more than 14 days old it deletes the original, the `/docs/i/x`
fallback, the metadata file *and* all three resized variants. Almost every
image in the repo qualifies, so `npm run thin` would strip the site of its
images rather than thin anything.

Also drop the `console.log('rehypeWbr = ...')` left behind in
astro.config.mjs, which printed on every dev server start and build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These 27.5 MB of images are referenced by nothing: not by any page in the
built site, and not by any file under src/. They are leftovers from content
that has since been rewritten or removed.

Found by extracting every `/docs/img/...` reference from the built output and
from repo source, then diffing against what is on disk. Absolute
`https://octopus.com/docs/...` URLs count as references too - JSON-LD builds
its logo URL that way, via a template literal in config.ts, so a naive scan
of site-relative paths alone wrongly reports `octopus-logo.png` as unused.
Filenames containing parentheses need the extension-anchored match as well.

Verified after the fact by resolving all asset references in the built output
against the files on disk: zero missing targets. `npm run crawl` is also
clean across 5,390 links.

Images referenced only from pages that exist in src but not in the deployed
site (mostly ui-update-sample.mdx) are deliberately kept.

This removes the source images only. Each one also has generated variants
under `public/docs/i/` and a `.json` metadata sidecar; both are outputs of
the responsive-image pipeline and are cleaned up separately, so that all
changes to generated image output stay on one branch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`public/docs/images/autocomplete.gif` is byte-identical to
`public/docs/shared-content/images/autocomplete.gif`, and the shared-content
path is the one the markdown actually links to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Arrow, List, MoonIcon, OctopusLogo, SunIcon and Youtube are Astro components
imported by Card, Header and ThemeSwitcher, but they were sitting in
`public/docs/img/`. Everything under `public/` is copied verbatim into the
deployed package, so each one was compiled into the bundle *and* published as
raw component source at, for example, /docs/img/OctopusLogo.astro.

Move them to `src/components/icons/` where the rest of the components live,
and fix the `../../public/docs/img/` import paths that went with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten `.dia` files - the editable Dia sources for the Kubernetes diagrams -
were stored next to the `.svg` files they export to, under
`public/docs/deployments/kubernetes/`. Nothing links to them, but because
they live under `public/` they were uploaded to the CDN on every deploy.

Move them to a new top-level `assets/` directory, with a README explaining
why source artifacts must stay out of `public/`. The exported `.svg` files
that pages actually reference stay where they are.

Exclude `assets/diagrams/**` from cspell. Dia stores its documents as
uncompressed XML, so `pnpm spellcheck` reads them as prose and reports its
internal attribute names (`lmargin`, `hguides`, `autorouting`, ...) as
misspellings - 131 of them across the ten files. Excluding by directory
rather than by extension keeps the README spellchecked and covers whatever
format the next diagram source arrives in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
astro.config.mjs already notes that the site is proxied onto octopus.com
under /docs/ only - that is why `build.assets` was moved to `docs/_astro`.
But Astro still routes `src/pages/index.md`, `components.mdx`,
`ui-update-sample.mdx` and `report/**` to the site root, so every deploy
uploaded `/index.html`, `/components/`, `/ui-update-sample/`, `/report/**`
and `/css/report.css`. None of those paths are reachable in production.

They are useful under `astro dev`, which does serve the root, so rather than
moving the pages, add a `prune-dist` integration that strips everything
outside `docs/` once the build is done. The rule is the deployment invariant
itself, so it keeps holding as pages come and go. It also catches a stray
`public/.DS_Store` that was being published.

Delete `src/pages/varying-azure-subscription-by-environment.md` outright: it
is a root-level redirect stub, so it was already unreachable before this
change. The same redirect exists at its real path under
/docs/deployment-examples/azure-deployments/.

Pruning the root index.html breaks `npm run crawl`, because linkinator needs
an entry point at the root of what it serves. Point it at `docs` with
`--server-root ./dist` instead; it scans 5,390 links clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@borland
borland force-pushed the orion/housekeeping branch from 7ea7dc9 to 3ebb086 Compare August 5, 2026 10:16
@borland

borland commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

@enf0rc3 I didn't intend to remove the index.html from the staging site. I used pnpm dev to check it, and everything looked good, but that was a mistake as the pruner only runs for builds, not dev.

I edited the pruner to keep index.html as well as the report folder. I let it prune the components and ui-update-sample though because they aren't linked from anywhere... the only way to get to them is to know that the URL already exists; It felt better to prune them and add a comment saying run pnpm dev locally if you happen to want them.

@enf0rc3 enf0rc3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks for reinstating the default index.html

@borland
borland merged commit 2a0ae97 into main Aug 5, 2026
7 checks passed
@borland
borland deleted the orion/housekeeping branch August 5, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants