Robotstxt is a lightweight http-server, just serving a disallow-robots.txt file using the Zig programming language(https://ziglang.org/).
Robots.txt basically works like a “No Trespassing” sign. It actually, tells robots whether we want them to crawl the website or not. With this role, we are disallowing all robots to crawl and avoid indexing in search engines.
| Details | |||
|---|---|---|---|
| :material-home: Project home{: .header-icons } | :octicons-link-16: Docs{: .header-icons } | :octicons-mark-github-16: Github{: .header-icons } | :material-docker: Docker{: .header-icons } |
sb install sandbox-traefik-robotstxtHTTP/1.1 200 OK
Content-Length: 26
User-agent: *
Disallow: /
When you want to reach *.yourdomain.tld/robots.txt
- :octicons-link-16: Documentation: Robotstxt Docs{: .header-icons }