symlinks.rst

How we use symlinks

Read the Docs stays highly available by serving all documentation pages out of nginx. This means that they never hit our Python layer, meaning that they never hit our database. This reduces the total number of servers to serve a request to 1, each of which is redundant.

Nginx

We handle a couple of different types of requests in nginx:

• Requests to a readthedocs.io subdomain
• Requests to a custom domain

Subdomains

For subdomains, this is a simple lookup of the project slug, using the subdomain portion of the request's hostname. This doesn't require symlinks, but it shows the basic logic that we need to replicate.

When a user navigates to http://pip.readthedocs.org/en/latest/, we know that they want the pip documentation. So we simply serve them the documentation:

location ~ ^/en/(.+)/(.*) {
    alias /home/docs/checkouts/readthedocs.org/user_builds/$domain/rtd-builds/$1/$2;
    error_page 404 = @fallback;
    error_page 500 = @fallback;
} 

location @fallback {
    proxy_pass http://127.0.0.1:8888;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    add_header X-Deity Asgard;
}

Note

The @fallback directive is hit when we don't find the proper file. This will cause things to hit the Python backend, so that proper action can be taken.

Custom domains

Custom domains add a bit of difficulty, because at the nginx layer we don't know what documentation to serve. When someone requests http://docs.fabfile.org/en/latest/, we can't look at the URL to know to serve the fabric docs.

This is where symlinks come in. When someone requests http://docs.fabfile.org/en/latest/ the first time, it hits the Python layer. In that Python layer we record that docs.fabfile.org points at fabric. When we build the fabric docs, we create a symlink for all domains that have pointed at fabric before.

So, when we get a request for docs.fabfile.org in the future, we will be able to serve it directly from nginx. In this example, $host would be docs.fabfile.org:

location ~ ^/en/(?P<doc_version>.+)/(?P<path>.*) {
    alias /home/docs/checkouts/readthedocs.org/cnames/$host/$doc_version/$path;
    error_page 404 = @fallback;
    error_page 500 = @fallback;
}

Notice that nowhere in the above path is the project's slug mentioned. It is simply there in the symlink in the cnames directory, and the docs are served from there.