Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
tree: 01330ec25d
Fetching contributors…

Cannot retrieve contributors at this time

59 lines (50 sloc) 2.273 kB
__NAME__ purpose
specify user-agents that will be classified as crawler bots (search engines)
__END__
__NAME__ see also
RobotUA, RobotIP
__END__
__NAME__ synopsis
<arg choice='plain' rep='repeat'><replaceable>useragent_string</replaceable></arg>
__END__
__NAME__ description
The &conf-RobotUA; directive defines a list of useragent strings which will be
classified as crawler robots (search engines), and cause &IC; to alter its
behavior to improve the chance of &IC;-served content being crawled
and indexed.
</para><para>
Note that this directive (and all other work done to identify robots)
only serves to improve the way in which &IC; pages are indexed, and to
reduce server overhead for clients that don't require our full attention
in the way humans do (for example, session information is not kept around
for spider bots).
Using this to "tune" the actual page content depending on a crawler
visiting does not earn you extra points, and may in fact be
detected by the robot and punished.
</para><para>
It's important to note that the directive accepts a wildcard list similar
to globbing &mdash;
<literal>*</literal> represents any number of characters, while
<literal>?</literal> represents a single character.
__END__
__NAME__ notes
For more details regarding web spiders/bots and &IC;, see
&glos-robot; glossary entry.
__END__
__NAME__ example: Defining RobotUA
<programlisting><![CDATA[
RobotUA <<EOR
ATN_Worldwide, AltaVista, Arachnoidea, Aranha, Architext, Ask, Atomz,
BackRub, Builder, CMC, Contact, Digital*Integrity, Directory, EZResult,
Excite, Ferret, Fireball, Google, Gromit, Gulliver, Harvest, Hubater,
H?m?h?kki, INGRID, IncyWincy, Jack, KIT*Fireball, Kototoi, LWP, Lycos,
MegaSheep, Mercator, Nazilla, NetMechanic, NetResearchServer, NetScoop,
ParaSite, Refiner, RoboDude, Rover, Rutgers, Scooter, Slurp, Spyder,
T-H-U-N-D-E-R-S-T-O-N-E, Toutatis, Tv*Merc, Valkyrie, Voyager, WIRE,
Walker, Wget, WhizBang, Wire, Wombat, Yahoo, Yandex, ZyBorg, appie,
asterias, bot, contact, crawl, collector, fido, find, gazz, grabber,
griffon, archiver, legs, marvin, mirago, moget, newscan, seek, speedy,
spider, suke, tarantula, agent, topiclink, whowhere, winona, worm, xtreme,
EOR
]]></programlisting>
__END__
Jump to Line
Something went wrong with that request. Please try again.