Skip to content

site: security headers, entity schema, and the files AI crawlers look for - #5

Merged
ralyodio merged 1 commit into
mainfrom
site/aeo-fixes
Aug 30, 2026
Merged

site: security headers, entity schema, and the files AI crawlers look for#5
ralyodio merged 1 commit into
mainfrom
site/aeo-fixes

Conversation

@ralyodio

@ralyodio ralyodio commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

From the 15-engine AEO audit. Limited to findings I verified against the live site by hand — several engines were wrong, and acting on them would have made things worse.

Findings I did not act on, and why

Claim Reality
"No robots.txt, no sitemap.xml, no JSON-LD" (GPT-5 Mini) All three are served. robots.txt 200, sitemap with 18 URLs, SoftwareApplication + FAQPage in the head.
"Future-dated lastmod" (5 engines) The date was that day. The engines' clocks are behind. The real problem was different — see below.
"Unrendered ${userInput} template literal" (Slop Score) It is inside <code class="language-ts"> on /security — a deliberate example of the injection DiskPush does not do. Fixing it would delete the point of the paragraph.
Claude Sonnet 4.6, DeepSeek V4 Both errored (API limit / invalid key). Zero signal, not zero findings.

What was actually broken

No security headers at all. Only cache-control was set. Added HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy via next.config.mjs.

The CSP keeps 'unsafe-inline' for script and style — Next inlines its hydration payload and the app's styles, so a stricter policy blanks the page. Scripts are otherwise same-origin only, which is the part that limits an injection, and object-src 'none' + frame-ancestors 'none' close the classic bypasses.

Every sitemap URL carried one build timestamp. Not that it was future-dated — that it was identical across all 18 URLs and changed on every deploy. A lastmod like that carries no information, so crawlers discount it, which is worse than sending none. Dates now come from the files each page is built from; doc pages use their own markdown's mtime, which is precisely when that page last changed.

before:  1 distinct lastmod across 18 URLs
after:  18 distinct lastmod across 18 URLs

No Organization. The schema described an application that nothing published — the only trace of the maker was a GitHub URL and the domain on the security address. Organization and WebSite are now in the @graph, and SoftwareApplication names them as publisher/author, plus downloadUrl, license and isAccessibleForFree.

/llms.txt and /.well-known/security.txt 404'd. Both generated, not written: a hand-kept copy of the docs index goes stale, and a stale llms.txt is worse than none because models quote it confidently. The security.txt Expires is computed a year out for the same reason — a hard-coded date silently invalidates the file on a day nobody has diarised.

/security and /docs/security were the same markdown at two URLs, with the same <title>. The promoted page is now canonical for both, so they stop competing without either URL being dropped.

robots.txt names the AI crawlers explicitly. The wildcard already allowed them, so this grants no new access — it states a position rather than leaving it inferred, and makes narrowing it later an edit rather than an unrecorded decision.

Verified by building and serving the site

  • All six headers present on /
  • /llms.txt and /.well-known/security.txt return the expected text
  • 18 distinct sitemap dates
  • /docs/security → rel="canonical" href="https://diskpush.com/security"
  • Organization, WebSite, ContactPoint in the JSON-LD graph

Typecheck clean, full suite 238 passing.

Deliberately left out

Content decisions, not bugs — they need your input on what to say: about/team page, general contact route, on-site changelog, comparison pages, testimonials/social proof. Those were the other high-consensus findings.

One thing outside this repo: DMARC is broken, not just missing. _dmarc.diskpush.com is a CNAME to uixie.porkbun.com, which serves no TXT record — so it resolves to nothing. That needs a DNS change at Porkbun.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y5jnkZKX4AdPgBMzMosxE7

@ralyodio @claude
site: security headers, entity schema, and the files AI crawlers look…
a8da950 
… for

From a 15-engine AEO audit, limited to what was verified against the live site
by hand. Several of the engines' findings were wrong and are not acted on here:
one reported no robots.txt, no sitemap and no JSON-LD when all three are
served; five called the sitemap lastmod "future-dated" when the date was that
day; and the "unrendered ${userInput} template literal" is a TypeScript sample
in a <code> block on the security page, showing the injection DiskPush
deliberately does not do.

What was real:

  - No security headers at all. Only cache-control was set — no HSTS, CSP,
    X-Frame-Options, nosniff, Referrer-Policy or Permissions-Policy. The CSP
    keeps 'unsafe-inline' for script and style because Next inlines its
    hydration payload and the app's styles; scripts are otherwise same-origin,
    and object-src/frame-ancestors 'none' close the usual bypasses.

  - Every sitemap URL carried one build timestamp. A lastmod that is identical
    site-wide and changes on every deploy tells a crawler nothing, so it gets
    discounted — worse than sending none. Dates now come from the files each
    page is built from; doc pages use their own markdown's mtime, which is
    exactly when that page last changed. 18 URLs, 18 distinct dates.

  - No Organization. The schema described an application that nothing
    published: the only trace of who makes DiskPush was a GitHub URL and the
    domain on the security address. Organization and WebSite are now in the
    graph, and SoftwareApplication names them as publisher and author.

  - /llms.txt and /.well-known/security.txt 404'd. Both are generated rather
    than written: a hand-kept copy of the docs index goes stale, and a stale
    llms.txt is worse than none because models quote it confidently. The
    security.txt Expires is computed a year out for the same reason — a
    hard-coded date silently invalidates the file on a day nobody has diarised.

  - /security and /docs/security were the same markdown at two URLs, with the
    same title. The promoted page is now canonical for both.

robots.txt names the AI crawlers explicitly. The wildcard already allowed them,
so this grants nothing new; it states a position rather than leaving it to be
inferred, and makes narrowing it later an edit rather than an unrecorded
decision.

Left alone deliberately, because they are content decisions rather than bugs:
about/team page, general contact route, on-site changelog, comparison pages,
testimonials. Also outside the repo: _dmarc.diskpush.com is a CNAME to
uixie.porkbun.com, which serves no TXT record, so DMARC resolves to nothing.

Verified by building and serving the site: six headers present, llms.txt and
security.txt served, 18 distinct sitemap dates, /docs/security canonical to
/security, and Organization/WebSite in the graph.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y5jnkZKX4AdPgBMzMosxE7
@ralyodio
ralyodio merged commit 4c6511b into main Aug 30, 2026
4 checks passed
@ralyodio ralyodio mentioned this pull request Aug 30, 2026
Merged
ralyodio added a commit that referenced this pull request Aug 30, 2026
@ralyodio @claude
site: take sitemap dates from git, because mtime is a lie in CI (#7)
0975cbd 
The lastmod fix in #5 worked locally and did nothing in production. Deployed,
all eighteen URLs still carried one identical date -- 2026-08-30T04:07:33,
the build's own checkout time.

mtime was the wrong source. A git clone stamps every file it writes with the
moment it wrote it, so on a CI checkout every file in the tree shares one
mtime and the per-page dates collapse straight back into the single build
timestamp the change existed to remove. A working tree has real edit times,
which is exactly why local verification passed and proved nothing.

Reproduced by cloning the repository and comparing the two sources against the
same thirteen files:

  distinct mtimes:    1
  distinct git dates: 6

Six rather than thirteen because several docs were committed together, which
is the honest answer: those pages did change at the same moment.

So the date now comes from `git log -1 --format=%cI -- <path>`, which is a
record of when the content changed rather than when the machine last touched
the file. mtime stays as a fallback for a build with no git history, where it
is no worse than the build clock it replaces, and the build clock remains the
last resort.

Verified against a build rather than a working tree: nine distinct dates
across the eighteen URLs.


Claude-Session: https://claude.ai/code/session_01Y5jnkZKX4AdPgBMzMosxE7

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ralyodio
ralyodio deleted the site/aeo-fixes branch August 30, 2026 05:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ralyodio