Skip to content
This repository was archived by the owner on Nov 19, 2024. It is now read-only.

Fix misleading description of robots functionality #843

Merged
leslietilling merged 3 commits into from
Nov 10, 2020
Merged

Fix misleading description of robots functionality #843

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
93184c0
Fix misleading description of robots functionality
mpchadwick Nov 6, 2020
189dfdb
Update src/marketing/search-engine-robots.md
mpchadwick Nov 10, 2020
6697b74
Merge branch 'master' into patch-1
leslietilling Nov 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/marketing/search-engine-robots.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Search Engine Robots
---

The Magento configuration includes settings to generate and manage instructions for web crawlers and bots that index your site. The instructions are saved in a file called `robots.txt` that resides in the root of your Magento installation. The instructions are directives that are recognized and followed by most search engines.
The Magento configuration includes settings to generate and manage instructions for web crawlers and bots that index your site. If the request for `robots.txt` reaches Magento (rather than a physical file) it is dynamically routed to the robots controller. The instructions are directives that are recognized and followed by most search engines.

By default, the robots.txt file that is generated by Magento contains instructions for web crawler to avoid indexing certain parts of the site that contain files that are used internally by the system. You can use the default settings, or define your own custom instructions for all, or for specific search engines. There are many articles online that explore the subject in detail.

Expand Down