The Mantid documentation is written in reStructuredText and processed using Sphinx. It uses a custom bootstrap theme for Sphinx and both are required to build the documentation.
To install Sphinx and the bootstrap theme use easy_install:
easy_install -U Sphinx
easy_install -U sphinx-bootstrap-theme
or pip:
pip install Sphinx
pip install sphinx-bootstrap-theme
This may require admin privileges on some environments.
CMake produces a docs-html target that is used to build the documentation (only if you set the cmake variable DOCS_HTML=ON). The output files will appear in a html sub directory of the main build/docs directory.