feat: Introduces bot-specific routing and static SEO page serving in Caddy

[gocanto]

Summary by CodeRabbit

• New Features
  • Adds rich social and search preview pages for link unfurling and crawlers.
• Performance
  • Improves SPA asset caching and HTML caching for faster page loads.
• Security
  • Applies stricter response headers (Referrer-Policy, X-Content-Type-Options, X-Frame-Options).
• Behavior
  • Separates bot vs. human handling to deliver correct previews while keeping the SPA experience intact.
• Chores
  • Includes SEO/static preview assets in deployments via new read-only mounts.

[coderabbitai]

Walkthrough

Introduces bot-specific routing and static SEO page serving in Caddy, separates bot vs human SPA handling with adjusted exclusions and headers, and mounts a new read-only SEO assets directory into Caddy containers via docker-compose.

Changes

Cohort / File(s)	Summary
Caddy bot/human routing and headers caddy/WebCaddyfile.internal	Adds @sharebots User-Agent matcher; serves /seo/*.seo.html for bots; separates SPA handling for non-bots; updates @html exclusions (adds /relay/*, keeps /seo/* excluded); applies Vary: User-Agent and security headers; preserves 404/SEO-blocking for /seo and /seo/*.
Docker SEO volume mounts docker-compose.yml	Mounts ./public/seo to /usr/share/caddy/seo:ro for caddy-local and caddy-prod to expose static SEO pages to Caddy.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Client
    participant Caddy
    participant SEO_Files as /usr/share/caddy/seo
    participant SPA as SPA/Static

    Client->>Caddy: HTTP Request (path, User-Agent)
    alt UA matches share/social bots (@sharebots)
        note over Caddy: Bot flow (new)
        Caddy->>SEO_Files: Lookup /seo/*.seo.html
        SEO_Files-->>Caddy: File (if exists)
        Caddy-->>Client: 200 SEO HTML + security headers
        opt Missing SEO file
            Caddy-->>Client: 404
        end
    else Non-bot (humans)
        note over Caddy: SPA flow with exclusions<br/>(excludes /seo/*, /relay/*, assets)
        Caddy->>SPA: Serve SPA or static assets
        Caddy-->>Client: 200 with caching + Vary: User-Agent
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• [feature] Integrate docker and caddy #4: Also modifies caddy/WebCaddyfile.internal and docker-compose to serve static files and mount web assets; overlaps on SEO/static asset exposure.
• hotfic: Use Caddy router instead #48: Adjusts Caddy routing around relay paths and route structure; overlaps with this PR’s updated @html exclusions including /relay/*.
• feat: Serve SEO pages #66: Changes Caddy SEO/static routing and headers; closely aligned with this PR’s /seo handling and header configuration.

Poem

In moonlit logs I hop with glee,
New routes for bots, for humans—whee!
SEO burrows, read-only, neat,
SPA trails cached under my feet.
With headers snug and paths just right,
I thump deploy—goodnight, good night. 🐇✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.
Title Check	✅ Passed	The title clearly and concisely summarizes the main change by describing the introduction of bot-specific routing and static SEO page serving in Caddy. It focuses on the primary feature without extraneous details or noise. This makes it immediately informative for team members reviewing the pull request.

✨ Finishing touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch hotfix/fix-seo-server

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

caddy/WebCaddyfile.internal (1)

54-54: Consider adding more social media bots to the regex pattern.

The current regex covers major search engines and social platforms, but you may want to include additional bots commonly used for link previews and social sharing.
Based on the search results, you may want to consider adding additional bot user agents commonly used for link previews and social sharing. Here are some notable ones that could be added to your regex pattern:

• iframely - commonly used for link previews
• redditbot - for Reddit link previews
• mastodon - for Mastodon social platform
• applebot - for Apple ecosystem link previews
• amazonbot - for Amazon's services
• bingpreview - for Microsoft services beyond Bing search

Consider updating your regex to include these additional patterns:

-@sharebots header_regexp ua User-Agent "(?i)(googlebot|bingbot|duckduckbot|yandex(bot|images)|baiduspider|facebookexternalhit|twitterbot|linkedinbot|slackbot|telegrambot|discordbot|pinterest|whatsapp|skypeuripreview|bitlybot|vkshare|quora link preview|embedly|outbrain|nuzzel|ios link preview|snapchat|applebot)"
+@sharebots header_regexp ua User-Agent "(?i)(googlebot|bingbot|duckduckbot|yandex(bot|images)|baiduspider|facebookexternalhit|twitterbot|linkedinbot|slackbot|telegrambot|discordbot|pinterest|whatsapp|skypeuripreview|bitlybot|vkshare|quora link preview|embedly|outbrain|nuzzel|ios link preview|snapchat|applebot|iframely|redditbot|mastodon|amazonbot|bingpreview)"

Based on search results showing common social media bot user agents in 2024-2025.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d048731 and 668f264.

📒 Files selected for processing (2)

• caddy/WebCaddyfile.internal (2 hunks)
• docker-compose.yml (2 hunks)

🔇 Additional comments (4)

docker-compose.yml (1)

36-36: LGTM! SEO volume mounts correctly configured.

The read-only SEO asset mounts are properly configured for both local and production environments, supporting the new bot-specific routing functionality introduced in the Caddy configuration.

Also applies to: 64-64

caddy/WebCaddyfile.internal (3)

55-100: LGTM! Well-structured bot handling with proper fallback.

The bot handling logic is well-implemented with:

• Proper try_files sequence with SEO file fallbacks
• Appropriate caching headers for different asset types
• Security headers applied to bot responses
• Consistent compression settings

---

103-135: LGTM! Human SPA handling is correctly implemented.

The human SPA handling provides:

• Clean fallback to SPA for non-bot traffic
• Consistent caching strategies matching the bot path
• Proper security headers
• Appropriate User-Agent variance handling

---

84-84: Good addition of /relay/ exclusion.*

Adding /relay/* to the @html exclusion list is appropriate since these paths are handled by the reverse proxy configuration and shouldn't receive HTML caching headers.

Also applies to: 121-121