The problem: if you search on Google for many things in the D documentation, you get results from a scatter of places. For example, for this search I'm getting results from /cutting-edge/ and /phobos-prerelease/, with the main docs being omitted due to similarity, which is usually not that useful. Many other searches also return results from various combinations of domains: dlang.org, www.dlang.org, d-programming-language.org, www.d-programming-language.org.
This pull request does two things to rectify the problem:
<link rel="canonical" ...>
While I'd also like to add a canonical link to Phobos docs, I don't see a good way to do it with the limitations of DDoc. Since it completely lacks string processing macros, it's not possible to create a correct link based on the existing predefined $(TITLE) and $(DOCFILENAME) macros. It would be possible by adding another predefined macro to DMD, or specifying the corresponding documentation filename in every Phobos/DRuntime module. Alternatively, just add the bloody redirect :)
Add canonical link for main site
Add robots.txt (exclude phobos-prerelease and cutting-edge)
@andralex: Could we please finally make all the other domains 301-redirect to dlang.org? I can help with server configuration or even host the redirections, if you need.
Took the rest to email.