Search engine optimizations

[CyberShadow]

The problem: if you search on Google for many things in the D documentation, you get results from a scatter of places. For example, for this search I'm getting results from /cutting-edge/ and /phobos-prerelease/, with the main docs being omitted due to similarity, which is usually not that useful. Many other searches also return results from various combinations of domains: dlang.org, www.dlang.org, d-programming-language.org, www.d-programming-language.org.

This pull request does two things to rectify the problem:

1. Establish a canonical link to dlang.org. I've bugged Andrei to set up redirects from the non-canonical domains to the canonical domain several times before, so since it looks like that's not going to happen, this is the next best thing. I only tested this with the Windows makefile - someone please test with posix.mak and check that the <link rel="canonical" ...> line points to valid URLs. Currently this doesn't include Phobos docs - see below.
2. Add a robots.txt which prevents search engines from returning the generally less-useful results from the /cutting-edge/ and /phobos-prerelease/ subdirectories. (Alternatively, this could be achieved with an XML sitemap specifying the importance of those pages.)

While I'd also like to add a canonical link to Phobos docs, I don't see a good way to do it with the limitations of DDoc. Since it completely lacks string processing macros, it's not possible to create a correct link based on the existing predefined $(TITLE) and $(DOCFILENAME) macros. It would be possible by adding another predefined macro to DMD, or specifying the corresponding documentation filename in every Phobos/DRuntime module. Alternatively, just add the bloody redirect :)

[dnadlinger]

@andralex: Could we please finally make all the other domains 301-redirect to dlang.org? I can help with server configuration or even host the redirections, if you need.

[andralex]

Took the rest to email.