Web Files

##Robots Web site owners use the /robots.txt file to give instructions about their site to web robots.

http://www.robotstxt.org/robotstxt.html

##Sitemap A sitemap is a file where you can list the web pages of your site to tell Google and other search engines about the organization of your site content.

Search engine web crawlers like Googlebot read this file to more intelligently crawl your site.

https://support.google.com/webmasters/answer/156184

##Favicon A favicon (short for favorite icon), also known as a shortcut icon, website icon, tab icon, URL icon or bookmark icon, is a file containing one or more small icons, most commonly 16×16 pixels, associated with a particular website or web page.

https://en.wikipedia.org/wiki/Favicon