We don't want search engines to crawl outdated docs. After we turn off docs on YDN we want the YUI3 docs to rise to the top of the results. This robots.txt file helps prevent old YUI 2.x docs from showing up in search results.
adding robots.txt to prevent search engines from crawling YUI 2.x docs
Adding newline to robots.txt
/cc @ericf Need this to prevent yui.github.io/yui2 from being crawled by search engines
I don't think this is a good idea. If people are still using YUI 2, then we want them to be able to find the documentation. I we prefer that we solve this problem by placing a banner across the top of all the archived docs HTML pages.
Ok, btw the banner across the top of the archived docs will be ready to go soon. yui/yui2#14