fix: introduce measures to avoid bots crawling and indexing activities

[doncicuto]

Add robots.txt, robots meta tag and X-Robots-Tag to avoid bots crawling and indexing activities for a Zitadel instance

Acceptance Criteria

• We don't want bots to crawl Instance URLs, using robots.txt
• We don't want bots that visit links to the Console or Login pages to index content from it, using meta tag robots set to none -> none equals noindex, nofollow
• Also we don't want REST endpoints to be indexed, using "X-Robots-Tag: none"

Here are some screenshots:

HTML source code

API Response Headers

Robots.txt found

Tasks

Beta Give feedback

We're having trouble rendering your tasklist, it's most likely has one of the following formatting errors:

• indented tasks (i.e. nested tasklist; support for this is coming!)
• an empty task (i.e. - [ ] on a line by itself)
• any empty new lines (before/after the tasklist or in between tasks)
• duplicate issue/pr links
• a draft task exceeds 512 characters

Please check the tasklists documentation for more information. 🔗 Feedback Discussion

Thank you for participating in the Private Beta ❤️

1. I am happy with the code
2. Short description of the feature/issue is added in the pr description
3. PR is linked to the corresponding user story
4. Acceptance criteria are met
5. All open todos and follow ups are defined in a new ticket and justified
6. Deviations from the acceptance criteria and design are agreed with the PO and documented.
7. No debug or dead code
8. Critical parts are tested automatically
9. Where possible E2E tests are implemented
10. Documentation/examples are up-to-date
11. All non-functional requirements are met
12. Functionality of the acceptance criteria is checked manually on the dev system.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 5, 2023 7:23am

[codecov]

Codecov Report

Merging #5728 (59f2f76) into main (11f0f54) will increase coverage by 0.00%.
The diff coverage is 65.51%.

@@           Coverage Diff           @@
##             main    #5728   +/-   ##
=======================================
  Coverage   44.40%   44.41%           
=======================================
  Files        1173     1175    +2     
  Lines      103086   103114   +28     
=======================================
+ Hits        45778    45799   +21     
- Misses      55159    55166    +7     
  Partials     2149     2149           

Impacted Files	Coverage Δ
internal/api/http/header.go	12.00% <ø> (ø)
internal/api/grpc/server/gateway.go	52.32% <12.50%> (-4.09%)	⬇️
cmd/start/start.go	58.02% <50.00%> (-0.03%)	⬇️
internal/api/api.go	65.71% <100.00%> (+0.24%)	⬆️
...nal/api/grpc/server/middleware/auth_interceptor.go	100.00% <100.00%> (ø)
...rnal/api/http/middleware/robots_tag_interceptor.go	100.00% <100.00%> (ø)
internal/api/robots_txt/robots_txt.go	100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

[hifabienne]

Thanks @doncicuto
We are currently finishing up the current sprint and will then have a look at it.

[peintnermax]

IMHO we can disallow robots on our login page, but disabling it in our console would also make our other meta tags unnecessary which are used to render a preview in chats / social media or doesn't it? 🤔

[hifabienne]

IMHO we can disallow robots on our login page, but disabling it in our console would also make our other meta tags unnecessary which are used to render a preview in chats / social media or doesn't it? 🤔

Who do you ask this? 😃

[peintnermax]

IMHO we can disallow robots on our login page, but disabling it in our console would also make our other meta tags unnecessary which are used to render a preview in chats / social media or doesn't it? 🤔

Who do you ask this? 😃

All who can answer it 😀 IMO the robot metatag prevents the other metatags from being read and a preview would no longer be generated

[peintnermax]

The PR and console changes look good to me. Maybe @adlerhurst or @livio-a can review the golang code

[livio-a]

The PR and console changes look good to me. Maybe @adlerhurst or @livio-a can review the golang code

i'll try check in the afternoon or at latest tomorrow

[adlerhurst]

please add the handler as well to the grpc-web handler here: https://github.com/doncicuto/zitadel/blob/6068796a436fba5699e39038d3c8922d27fb5437/internal/api/api.go#L176

[doncicuto]

Hi @adlerhurst, I've added the x-robots-tag to grpcwebserver as requested, thanks for the review

[adlerhurst]

hi @doncicuto

Thanks for your contribution. 🙏🏻
Looks good to me.

[github-actions]

🎉 This PR is included in version 2.24.0-ignore-me2.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀