feat(web): IndexNow ping + Search Console/Bing verification#916
Merged
Conversation
IndexNow (Bing/Yandex/Naver/Seznam/Yep — not Google; Bing feeds Copilot): - public/<key>.txt key file, excluded from the AuthKit middleware matcher so it serves as a static file (not redirected to /auth/login) - lib/indexnow.ts: buildUrlList() reuses sitemap(); submitIndexNow() POSTs to api.indexnow.org and returns the upstream status without throwing - app/api/indexnow/route.ts: CRON_SECRET-guarded GET; logs the failure path - vercel.json: daily Vercel Cron (0 6 * * *) Webmaster verification: extract the GSC/Bing logic into a pure, unit-testable webmasterVerification() helper in lib/seo (behavior identical to the old inline block) and wire it into layout.tsx. Activation (tokens + Vercel env) is a config step documented in docs/frontend/seo-verification.md; .env.local.example updated. Tests (9 new): indexnow buildUrlList, submit happy + bad-key, route 401; and webmasterVerification for both/one/neither tokens. No new deps; no backend route. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Greptile SummaryAdds two SEO indexing features: an IndexNow integration (
Confidence Score: 4/5Safe to merge; both features are activation-gated by env vars and don't affect any existing user-facing path. The core logic is correct and well-tested. The two findings are both in the IndexNow cron path — a missing network-error catch in the route handler and a key-rotation footgun in the static middleware matcher — neither of which affects deployed users until the feature is activated. web/src/app/api/indexnow/route.ts (unhandled network throw) and web/src/middleware.ts (static key exclusion that will need updating on key rotation)
|
| Filename | Overview |
|---|---|
| web/src/lib/indexnow.ts | New IndexNow integration; derives HOST/KEY from DOCS_ORIGIN; submitIndexNow comment claims "Never throws" but fetch network errors are unhandled and can propagate |
| web/src/app/api/indexnow/route.ts | CRON_SECRET-guarded endpoint; no try/catch around submitIndexNow so network errors yield unhandled 500; auth guard correctly absent when CRON_SECRET unset |
| web/src/middleware.ts | Adds the IndexNow key filename to the middleware exclusion list so AuthKit doesn't intercept it; filename is hardcoded, requiring manual sync on key rotation |
| web/src/lib/seo.ts | Extracts webmasterVerification() helper from layout.tsx; clean, unit-tested, correctly returns undefined when no tokens are set |
| web/src/app/layout.tsx | Refactors inline verification spread to use webmasterVerification(); behavior is identical, no regressions |
| web/vercel.json | Adds daily Vercel Cron at 06:00 UTC to /api/indexnow; valid JSON, correct cron expression |
| web/src/lib/indexnow.test.ts | Good unit coverage: buildUrlList, happy-path submit, 403 failure path, and 401 route guard; fetch is cleanly mocked via vi.stubGlobal |
| web/src/lib/seo.test.ts | Adds three assertions covering both-tokens, one-token, and neither-token cases; env vars properly restored in afterEach |
Sequence Diagram
sequenceDiagram
participant VC as Vercel Cron (daily 06:00 UTC)
participant R as /api/indexnow (route.ts)
participant IL as indexnow.ts buildUrlList / submitIndexNow
participant SM as sitemap() (@/app/sitemap)
participant IN as api.indexnow.org
VC->>R: GET /api/indexnow Authorization Bearer CRON_SECRET
R->>R: Validate CRON_SECRET (if set)
R->>IL: buildUrlList()
IL->>SM: sitemap()
SM-->>IL: "[{url...}]"
IL-->>R: string[]
R->>IL: submitIndexNow(urlList)
IL->>IN: "POST /IndexNow {host, key, keyLocation, urlList}"
IN-->>IL: 200 / 202 / 403
IL-->>R: "{status, body}"
alt 200 or 202
R-->>VC: "200 {ok:true, submitted:N}"
else other (e.g. 403)
R-->>VC: "502 {ok:false, indexnowStatus:403}"
end
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
web/src/app/api/indexnow/route.ts:27
Network-level fetch errors are unhandled. `submitIndexNow` documents "never throws on a non-2xx response," but a DNS failure, TCP reset, or timeout will throw from `fetch()` itself — not from a non-2xx response — and that exception will propagate out of this handler as an unhandled rejection, producing an opaque 500 with no structured error body in the cron logs.
```suggestion
let result: { status: number; body: string };
try {
result = await submitIndexNow(urlList);
} catch (err) {
console.error("IndexNow ping threw (network error):", err);
return NextResponse.json(
{ ok: false, error: "network_error" },
{ status: 502 },
);
}
const { status, body } = result;
```
### Issue 2 of 2
web/src/middleware.ts:41
Key filename is hardcoded in the middleware exclusion list. The middleware matcher is a static regex evaluated at build time, so it can't read `INDEXNOW_KEY` dynamically. If the key is ever rotated — new file at `web/public/<NEW_KEY>.txt` — the new filename must also be added here, otherwise AuthKit will intercept requests to the new key file and return a redirect instead of the key content, silently failing IndexNow domain verification.
Reviews (1): Last reviewed commit: "feat(web): add IndexNow ping and wire Se..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| return NextResponse.json({ ok: true, submitted: 0 }); | ||
| } | ||
|
|
||
| const { status, body } = await submitIndexNow(urlList); |
Contributor
There was a problem hiding this comment.
submitIndexNow documents "never throws on a non-2xx response," but a DNS failure, TCP reset, or timeout will throw from fetch() itself — not from a non-2xx response — and that exception will propagate out of this handler as an unhandled rejection, producing an opaque 500 with no structured error body in the cron logs.
Suggested change
| const { status, body } = await submitIndexNow(urlList); | |
| let result: { status: number; body: string }; | |
| try { | |
| result = await submitIndexNow(urlList); | |
| } catch (err) { | |
| console.error("IndexNow ping threw (network error):", err); | |
| return NextResponse.json( | |
| { ok: false, error: "network_error" }, | |
| { status: 502 }, | |
| ); | |
| } | |
| const { status, body } = result; |
Prompt To Fix With AI
This is a comment left during a code review.
Path: web/src/app/api/indexnow/route.ts
Line: 27
Comment:
Network-level fetch errors are unhandled. `submitIndexNow` documents "never throws on a non-2xx response," but a DNS failure, TCP reset, or timeout will throw from `fetch()` itself — not from a non-2xx response — and that exception will propagate out of this handler as an unhandled rejection, producing an opaque 500 with no structured error body in the cron logs.
```suggestion
let result: { status: number; body: string };
try {
result = await submitIndexNow(urlList);
} catch (err) {
console.error("IndexNow ping threw (network error):", err);
return NextResponse.json(
{ ok: false, error: "network_error" },
{ status: 502 },
);
}
const { status, body } = result;
```
How can I resolve this? If you propose a fix, please make it concise.
| matcher: [ | ||
| "/((?!_next/static|_next/image|favicon.ico|og-image.png|twitter-image.png|favicon-16x16.png|favicon-32x32.png|favicon-96x96.png|apple-touch-icon.png|robots.txt|sitemap.xml).*)", | ||
| "/((?!_next/static|_next/image|favicon.ico|og-image.png|twitter-image.png|favicon-16x16.png|favicon-32x32.png|favicon-96x96.png|apple-touch-icon.png|robots.txt|sitemap.xml|265a46be97a2ce1f8891dd452d243327.txt).*)", | ||
| ], |
Contributor
There was a problem hiding this comment.
INDEXNOW_KEY dynamically. If the key is ever rotated — new file at web/public/<NEW_KEY>.txt — the new filename must also be added here, otherwise AuthKit will intercept requests to the new key file and return a redirect instead of the key content, silently failing IndexNow domain verification.
Prompt To Fix With AI
This is a comment left during a code review.
Path: web/src/middleware.ts
Line: 41
Comment:
Key filename is hardcoded in the middleware exclusion list. The middleware matcher is a static regex evaluated at build time, so it can't read `INDEXNOW_KEY` dynamically. If the key is ever rotated — new file at `web/public/<NEW_KEY>.txt` — the new filename must also be added here, otherwise AuthKit will intercept requests to the new key file and return a redirect instead of the key content, silently failing IndexNow domain verification.
How can I resolve this? If you propose a fix, please make it concise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two SEO indexing features from the gap research.
IndexNow — instantly notifies Bing/Yandex/Naver/Seznam/Yep (not Google) when content changes; since Bing feeds Microsoft Copilot, it also helps Copilot freshness.
web/public/<key>.txtkey file, excluded from the AuthKit middleware matcher so it serves statically (not redirected to/auth/login).web/src/lib/indexnow.ts—buildUrlList()reusessitemap()(never drifts);submitIndexNow()POSTs toapi.indexnow.organd returns the upstream status without throwing.web/src/app/api/indexnow/route.ts—CRON_SECRET-guardedGET; logs the failure path.web/vercel.json— daily Vercel Cron (0 6 * * *).Webmaster verification — the GSC/Bing meta wiring already existed inline in
layout.tsx; this extracts it into a pure, unit-testablewebmasterVerification()helper inlib/seo(identical behavior) and adds a runbook.Verification
vitest→ 9/9 new assertions pass: IndexNow (buildUrlList, submit happy + bad-key failure, route 401 whenCRON_SECRETset) andwebmasterVerification(both / one / neither token).tsc --noEmitclean,eslintclean,vercel.jsonvalid JSON.package-lock.jsonchange); no backend route (OpenAPI untouched).Per
docs/frontend/seo-verification.md:INDEXNOW_KEY=265a46be97a2ce1f8891dd452d243327andCRON_SECRET=$(openssl rand -hex 32)in Vercel Production.NEXT_PUBLIC_GSC_VERIFICATION/NEXT_PUBLIC_BING_VERIFICATIONin Vercel Production, redeploy, click Verify, submit the sitemap.Until then the IndexNow ping is harmless (key falls back to the committed literal; cron just pings) and verification renders no meta tag (no crash).
🤖 Generated with Claude Code