-
Notifications
You must be signed in to change notification settings - Fork 4
/
974.txt
142 lines (108 loc) · 5.88 KB
/
974.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
[2] 仕様書:
- [CITE[robotstxt.org]] <http://www.robotstxt.org/>
-- [ROBOTS94] [CITE[A Standard for Robot Exclusion]]
<http://www.robotstxt.org/wc/norobots.html>
-- [ROBOTS97] [CITE[A Standard for Robot Exclusion]]
<http://www.robotstxt.org/wc/norobots-rfc.html>
- [[HTML 4]]
-- [CITE[The robots.txt file]]
<IW:HTML4:"appendix/notes.html#h-B.4.1.1">
[ROBOTS94] が1994年の合意で、その後1997年に [[Internet Draft]] [ROBOTS97]
が書かれましたが、未完成のままです。 HTML 4 は附属書 B (参考)
の中で解説していますが、規定はしていません。
[1] ''robotはぢきについて'' <http://c-moon.jp/robots.shtml>
[3]
HTML 4.0 の解説には間違いが沢山ありました。
HTML 4.01 では修正されています。
HTML 4.01
[CSECTION[A.1.2 Errors that were corrected]]
<IW:HTML4:"appendix/changes.html#h-A.1.2">
([[名無しさん]] [sage])
[4]
[CITE[sitemaps.org - Protocol]] ([TIME[2007-04-11 20:52:23 +09:00]] 版) <http://www.sitemaps.org/protocol.html#submit_robots>
([[名無しさん]] [WEAK[2007-04-12 11:00:38 +00:00]])
[5] [CITE[Creating Video Sitemaps - Webmasters/Site owners Help]] ([TIME[2009-02-08 17:41:37 +09:00]] 版) <http://www.google.com/support/webmasters/bin/answer.py?answer=80472>
[6] [CITE[自分のサイトを更新チェックされたくない - はてなアンテナのヘルプ]]
([TIME[2011-09-09 12:18:55 +09:00]] 版)
<http://hatenaantenna.g.hatena.ne.jp/keyword/%E8%87%AA%E5%88%86%E3%81%AE%E3%82%B5%E3%82%A4%E3%83%88%E3%82%92%E6%9B%B4%E6%96%B0%E3%83%81%E3%82%A7%E3%83%83%E3%82%AF%E3%81%95%E3%82%8C%E3%81%9F%E3%81%8F%E3%81%AA%E3%81%84?kid=19#robots>
[7] [CITE[ACAP - Automated Content Access Protocol v1]]
( ([[IPTC PR Team]] 著, [TIME[2012-09-19 08:03:13 +09:00]] 版))
<http://www.the-acap.org/documents.php>
[8] [CITE[WWW::RobotsRules - search.cpan.org]]
( ([TIME[2013-03-10 05:21:28 +09:00]] 版))
<http://search.cpan.org/dist/lcwa/lib/lwp/lib/WWW/RobotRules.pm>
[9] [CITE[WWW::RobotRules::Parser - search.cpan.org]]
( ([TIME[2013-03-10 05:22:34 +09:00]] 版))
<http://search.cpan.org/dist/WWW-RobotRules-Parser/lib/WWW/RobotRules/Parser.pm>
[10] [CITE[WWW::RobotRules - search.cpan.org]]
( ([TIME[2013-03-10 05:23:35 +09:00]] 版))
<http://search.cpan.org/dist/WWW-RobotRules/lib/WWW/RobotRules.pm>
[11] [CITE[WWW::RobotRules::Extended - search.cpan.org]]
( ([TIME[2013-03-10 05:28:09 +09:00]] 版))
<http://search.cpan.org/dist/WWW-RobotRules-Extended/lib/WWW/RobotRules/Extended.pm>
[12] [CITE[robots.txtにおけるAllowとDisallowとSitemapの優先順位 - 45式::雑記]]
( ([[渡辺四ん五(4n5)]] 著, [TIME[2012-02-29 17:58:01 +09:00]] 版))
<http://www.45shiki.net/blog/2009/12/b000924.htm>
[13] [CITE@en[Robots exclusion standard - Wikipedia, the free encyclopedia]]
( ([TIME[2013-03-10 00:00:25 +09:00]] 版))
<http://en.wikipedia.org/wiki/Robots_exclusion_standard>
[14] [CITE[Official Google Webmaster Central Blog: Improving on Robots Exclusion Protocol]]
( ([TIME[2014-03-19 08:31:54 +09:00]] 版))
<http://googlewebmastercentral.blogspot.jp/2008/06/improving-on-robots-exclusion-protocol.html>
[15] [CITE[The Web Robots Pages]]
( ([TIME[2013-12-03 09:21:42 +09:00]] 版))
<http://www.robotstxt.org/faq/future.html>
[16] [CITE@en[Robots.txt Specifications - Webmasters — Google Developers]]
( ([TIME[2012-08-02 09:24:38 +09:00]] 版))
<https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt>
[17] [CITE@en[How to Create a Robots.txt File - Bing Webmaster Tools]]
( ([TIME[2014-03-20 07:09:51 +09:00]] 版))
<http://www.bing.com/webmaster/help/how-to-create-a-robots-txt-file-cb7c31ec>
[18] [CITE@ja[著作権法施行規則]]
( ([TIME[2014-10-09 01:08:42 +09:00]] 版))
<http://law.e-gov.go.jp/htmldata/S45/S45F03501000026.html#1000000000007000000000000000000000000000000000000000000000000000000000000000000>
[19] [CITE@en[Semantic Web Crawling: A Sitemap Extention]]
([TIME[2015-04-30 13:00:34 +09:00]] 版)
<http://web.archive.org/web/20150104205905/http://sw.deri.org/2007/07/sitemapextension/>
[FIG(quote)[
[FIGCAPTION[
[20] [CITE@ja[Google 検索用 App Indexing | Google Developers]]
([TIME[2014-08-14 05:48:11 +09:00]] 版)
<https://developers.google.com/app-indexing/webmasters/server>
]FIGCAPTION]
> <link> 要素をウェブサイトの XML サイトマップに指定して、ページとそのページをアプリで開く方法の関係を示すこともできます。以下に例を示します:
> <?xml version="1.0" encoding="UTF-8" ?>
> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
> xmlns:xhtml="http://www.w3.org/1999/xhtml">
> <url>
> <loc>http://example.com/gizmos</loc>
> <xhtml:link rel="alternate" href="android-app://com.example.android/example/gizmos" />
> </url>
> ...
> </urlset>
]FIG]
[21] [CITE@en[Official Google Webmaster Central Blog: Best practices for XML sitemaps & RSS/Atom feeds]]
([TIME[2015-06-11 14:18:33 +09:00]] 版)
<http://googlewebmastercentral.blogspot.jp/2014/10/best-practices-for-xml-sitemaps-rssatom.html>
[FIG(quote)[
[FIGCAPTION[
[22] [CITE@en[Official Google Webmaster Central Blog: Best practices for XML sitemaps & RSS/Atom feeds]]
([TIME[2015-06-11 14:18:33 +09:00]] 版)
<http://googlewebmastercentral.blogspot.jp/2014/10/best-practices-for-xml-sitemaps-rssatom.html>
]FIGCAPTION]
>
> <entry>
> <link href="http://example.com/mypage" />
]FIG]
[23] [[ResourceSync]] は[[XML名前空間]]を使って独自拡張しています。
[FIG(quote)[
[FIGCAPTION[
[24] [CITE@en[Providing Structured Data | Custom Search | Google Developers]]
([TIME[2015-12-02 02:24:14 +09:00]] 版)
<https://developers.google.com/custom-search/docs/structured_data?csw=1#using-pagemaps>
]FIGCAPTION]
>
> <url>
> <loc>http://www.example.com/bar</loc>
> <PageMap xmlns="http://www.google.com/schemas/sitemap-pagemap/1.0">
]FIG]