Add no_index option to block search engine crawlers

[mxpv]

Closes: #772

[Copilot]

Pull request overview

Adds a no_index server configuration option to help prevent search engines from indexing the hosted content by serving a restrictive robots.txt and applying an X-Robots-Tag response header when enabled.

Changes:

• Extend services/web server config with NoIndex (toml:"no_index") and wire it into server initialization.
• Add middleware to attach X-Robots-Tag: noindex, nofollow and a /robots.txt handler when configured.
• Add config parsing tests and update docs to describe the new option and endpoint.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
services/web/server.go	Adds NoIndex config flag, /robots.txt handler, and middleware to set X-Robots-Tag.
services/web/server_test.go	Adds tests covering default disabled behavior and enabled behavior for robots/header.
cmd/podsync/config_test.go	Adds TOML config parsing tests for no_index.
CLAUDE.md	Documents the new no_index setting and /robots.txt endpoint.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.