fix(seo): disallow status and unsubscribe token routes in robots.txt - #73
Conversation
Both /status/[token] and /unsubscribe/[token] carry a private token in the URL and are linked from outbound email, same as /edit/[token]. Add them to the disallow list and normalize /admin to have a trailing slash like the rest. Fixes AgentPostmortem#57
robots.txt matches on prefix. "/admin/" only covers paths under it, so app/admin/page.tsx, the dashboard itself, would have become crawlable. "/admin" covers both it and everything beneath. The other four are right as they are: /api, /edit, /status and /unsubscribe have no bare page, only tokenised children.
|
Good catch on the token routes, thanks. Merged with one adjustment pushed to this branch. The change from Reverted that one entry to The |
|
Merged, @waterlemonnn. Thank you for taking the time. Genuinely helpful change for agentpostmortem. If this project helps you out, a star helps others find it too. Happy to see more from you whenever you have the time. |
2 participants
Closes #57.
`/status/[token]` and `/unsubscribe/[token]` put a private token in the path and are linked from outbound email, same as `/edit/[token]` — so they should be kept out of `robots.txt` too. Added both to the disallow list, and made `/admin` a trailing-slash path like the others for consistency.
Checked `/status/[token]` and `/unsubscribe/[token]` route metadata — both already set `robots: { index: false }`, so no change needed there.
Verified locally: `npx prettier --check .`, `npm run lint`, `npx tsc --noEmit` all pass, and `curl localhost:3000/robots.txt` shows the new entries:
```
User-Agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /edit/
Disallow: /status/
Disallow: /unsubscribe/
```
`npm run build` fails in my environment due to a missing `RESEND_API_KEY` — confirmed this is pre-existing and unrelated by reproducing it on a clean checkout of `main`.