Security: Add `X-Robots-Tag` header and `robots.txt` file to control crawlers

[lastzero]

As a user with a private photo collection, I do not want search engines such as Google to crawl, index and/or cache the content on my server so that it remains private.

The X-Robots-Tag HTTP header controls how search engines index and crawl web pages and other content. It works similarly to a robots.txt file or the robots meta tag in HTML, but offers more flexibility and can be applied to any file type, not just web pages. This makes it particularly useful for controlling the indexing of multimedia content:

• https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

Related Issues:

• Sharing: Public Image Wall #669
• What's the best way to prevent search engines from indexing my Photoprism instance? #4571

[lastzero]

You will be able to test these changes with our preview build shortly:

• https://docs.photoprism.app/getting-started/updates/#development-preview

If you use a reverse proxy, you can alternatively set the X-Robots-Tag header there (or serve a custom robots.txt file):

• https://docs.photoprism.app/getting-started/proxies/traefik/