-
-
Notifications
You must be signed in to change notification settings - Fork 85
/
robots.txt.scroll
30 lines (23 loc) · 911 Bytes
/
robots.txt.scroll
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import ../code/conceptPage.scroll
id robots.txt
name Robots.txt
appeared 1994
creators Martijn Koster
tags configFormat
conceptDescription A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.
country United Kingdom
originCommunity https://web.archive.org/web/20131029200350/http://inkdroid.org/tmp/www-talk/4113.html
reference https://developers.google.com/search/docs/advanced/robots/intro
reference https://en.wikipedia.org/wiki/Robots_exclusion_standard
example
User-agent: googlebot # all Google services
Disallow: /private/ # disallow this directory
User-agent: googlebot-news # only the news service
Disallow: / # disallow everything
User-agent: * # any robot
Disallow: /something/ # disallow this directory
lineCommentToken #
hasLineComments true
# A comment
hasComments true
# A comment