-
Notifications
You must be signed in to change notification settings - Fork 361
Directions for editing CLM documentation on github and sphinx
The CTSM documentation is written using reStructuredText markup language. ReStructuredText is a markup language, like HTML, markdown or latex. (See below for more resources on learning reStructuredText.)
Sphinx is a python tool for publishing reStructuredText documents in other formats such as HTML and PDF.
The CTSM documentation source is stored in the doc/source directory of
the main CTSM repository. The built documentation is stored in the
gh-pages branch of the ESCOMP/ctsm-docs
repository. The Sphinx-generated
HTML pages are accessible from URL
https://escomp.github.io/ctsm-docs/
As a contributor to CTSM, you don't necessarily need to know all of the
information laid out below. If you need to update the tech note for some
changes you have made, you should add or edit the necessary rst files in
doc/source/tech_note on your CTSM branch. Then simply commit your
changes to these rst files just as you would do for changes to Fortran
source files, and include these rst changes in your Pull
Request. Ideally these documentation changes will be on the same branch
and the same Pull Request as the related source file changes, but they
can also be made later, on a separate branch, and submitted to us via a
separate Pull Request. In either case, the CTSM maintainers will review
the documentation changes in the Pull Request and merge these changes to
CTSM's master branch when ready.
You do not need to open a Pull Request for the built documentation: one of the CTSM maintainers will rebuild the documentation later. However, if you have made more than minor edits to the documentation, we prefer if you have tested the documentation build to make sure there are no errors in the edited rst files and that the html documentation appears how you intended. But you do not need to send us or point us to your rebuilt documentation: One of the CTSM maintainers will build the documentation themselves to review the build.
If you need to build the documentation, and/or if you need to change or
add any images, you will need to keep reading for instructions on
installing and using git lfs.
In order to give concrete examples, we assume a particular directory
structure for these directions: We assume you have a directory in your
home directory named ctsm-repos, and that you clone the main CTSM
repository and the ctsm-docs repository inside that directory. So you
will have something like:
~/ctsm-repos/ctsm
~/ctsm-repos/ctsm-docs
It's fine for your ctsm directory to be named something more specific,
for example, including the branch name as is recommended elsewhere in
this
wiki. You
will just need to use the appropriate directory name in commands given
below.
You can use a different directory organization if you wish, but if you are using the Docker container-based method, then it is best to have both your main CTSM clone and your ctsm-docs clone reside somewhere under a common parent directory (where that parent directory is somewhere below your home directory).
The following packages need to be installed on the machine where you want to build the documentation:
- Git-LFS (git large file support; needed to obtain or modify image files)
- Sphinx
- The correct version of the Sphinx theme we use
- latexmk (needed for building the PDF and for creating the equations in the html documentation)
We document a few methods for installing the necessary packages for building the documentation. The preferred method is to use the ESCOMP Docker container. We also document methods for using your Mac (which can be adapted to work on Linux) and for building the documentation on NCAR's cheyenne machine.
This is our preferred method. It avoids the complexity of installing software on your local machine, and ensures that all people building the documentation are using the same versions of sphinx and other utilities.
The escomp/base Docker container has all the necessary software
(Git-LFS, Sphinx, the Sphinx theme, latexmk, etc), and will work on
Windows, Linux and Mac systems equally. Installation, then, simply
requires three steps; these should all be done on your personal desktop
or laptop.
-
Download and install Docker on your desktop or laptop
-
Download the
escomp/basecontainer image (see below for other options) -
Run the container, mapping a local directory into it
Download from Docker and follow the instructions.
You will then need to launch the Docker application.
Once Docker is successfully installed and launched, you can obtain the
escomp/base container by running the following command:
docker pull escomp/base
(This should work in the Windows command-prompt, Mac shell and Linux terminals.)
The image, in several layers, is about 1.5GB in size, so it'll take a little bit to download.
Alternative: If you have already obtained a version of the CESM
container (e.g., escomp/cesm-2.2) and are using it for other purposes,
then you can use that container instead of escomp/base. The CESM
container contains everything in the base container and more, so
everything will work the same as in these instructions, except you will
need to replace references to escomp/base with escomp/cesm-2.2.
A rare issue to be aware when running either container is a 'system error' when writing to a mounted directory. In the event this happens, simply re-running the previous command has worked every time. This is a Docker issue.
The final step is to run the Docker container. Since this needs to be done every time you build the documentation (rather than being one-time setup), it is documented below. See the below section, If using Docker: Run the container.
This section documents an alternative, in case you prefer to install all of the required software directly on your local system rather than using Docker. It is better to use the Docker method as described above, partly because it is easier and partly because it ensures that everyone is using the same versions of sphinx, etc.
These instructions have been tested on a Mac. Previously, support for the documentation build was also provided on CGD Linux machines (e.g., sunshine.cgd.ucar.edu and thorodin.cgd.ucar.edu), but this may not work currently due to requirements for git-lfs and the necessary Sphinx theme.
The following documentation assumes that you have the following available on your machine:
-
A recent version of git, ideally configured according to the recommendations in https://github.com/ESCOMP/CTSM/wiki/Recommended-git-setup#one-time-git-configuration.
-
A recent version of python (python3.x)
-
If using a Mac, the Homebrew package manager
Follow the instructions on the Git LFS page for installing git-lfs on your platform.
On a mac using homebrew, this can be done with:
brew install git-lfs
Although the actual installation only needs to be done once per machine, each user who wants to use it must run the following command to set up git-lfs for your user account on this machine:
git lfs install
Sphinx and the necessary Sphinx theme are python packages that can be
installed with pip install:
pip install sphinx
pip install git+https://github.com/esmci/sphinx_rtd_theme.git@version-dropdown-with-fixes
For more details on installing Sphinx, see the Sphinx documentation.
You may then need to update your PATH environment variable to include
the path to the sphinx-build script.
On a mac using homebrew, this can be done with:
brew cask install mactex
Note that you will need to open a new terminal window for the latexmk
tool to be added to your path (it is in /Library/TeX/texbin/latexmk,
which should have been added to your path by the above brew command).
If you want to build the PDF (not just the html web pages), you may also need rst2pdf. This can be installed with:
pip install rst2pdf
Finally, although we haven't figured out a way to view the built documentation on cheyenne, you can test the build there, and then transfer the files to your local machine for viewing.
To do so, first ensure that you are using the git and python modules
on cheyenne rather than the default system versions, by doing module load git and module load python. (Note that the default git module
has git lfs bundled with it.)
Then, the first time you build the documentation, do this one-time setup:
pip install --user sphinx
pip install --user sphinxcontrib-programoutput
pip install --user git+https://github.com/esmci/sphinx_rtd_theme.git@version-dropdown-with-fixes
and add the following to your .bashrc or similar file if using bash:
export PATH=/glade/u/home/$USER/.local/bin:$PATH
or, if you're using tcsh, add to your .tcshrc:
setenv PATH /glade/u/home/$USER/.local/bin:$PATH
If you are using the recommended Docker-based method described above, then there is an additional step you will need to do to run the container. To run the container interactively, assuming the directory structure recommended above, use a command like the following (for Mac/Linux):
docker run -it --rm -v ${HOME}/ctsm:/home/user escomp/base
This tells docker to run the container (docker run), interactively
with a terminal (-it), to clean up the container after running
(--rm), and most importantly, to map the ${HOME}/ctsm directory into
the /home/user directory inside the container (-v ${HOME}:/home/user).
(It's up to the user to choose what directory to mount; mapping your
whole home directory will make it all accessible to the container, but
this can have issues with the container reading startup files like
.bashrc in your home system, which can cause problems.)
After running this command, you will see a command prompt, but now you will be in the Docker environment. This means:
- You will be able to see directories and files contained in
${HOME}/ctsm, but nothing else.${HOME}in the Docker environment refers to${HOME}/ctsmin your native system environment. Changed or added files here will be visible in your native system, in the${HOME}/ctsmdirectory. This will become important later in these directions, since you will view the generated documentation from your native environment. - Any commands you run from this environment will use the commands bundled with the Docker image, not the commands in your native system environment.
You will want to have at least two terminals open: one containing this
Docker prompt and one in your local environment. Then, when running any
of the below commands that begin with make BUILDDIR=..., you
should use the Docker command prompt. On the other hand, git commit
commands, as well as commands to view the built documentation, should
be run from your local environment. (git commit should be run from
your local environment so that it uses your personal git
configuration. Viewing the built documentation requires programs like a
web browser that are only available in your local environment.) Commands
like git clone and mkdir can be run from either environment, but
keep in mind the directory remapping described above (so a directory
with path ~/my-ctsm-docs in the Docker environment has path
~/ctsm/my-ctsm-docs in your local environment, assuming the directory
setup recommended above).
When you are done using this Docker container, simply type exit from
the command prompt.
For far more information on running Docker containers, you can look at Docker's documentation.
Image files are tracked using Git LFS (Large File Support). The images are stored somewhere on GitHub, but aren't actually part of the main repository. Instead, the repository contains small text files that direct git-lfs to the appropriate storage location on GitHub.
For the most part, you can remain blissfully unaware of the details of how this works. For example, if you want to add a new image file to the repository, the process is just like adding any other file to the repository, as long as you have installed git-lfs. In addition, building the documentation does not require any additional steps - again, as long as you have installed git-lfs. But here are some notes that may help you:
We have configured git-lfs for CTSM so that it does not pull down any images by default, since these images are only rarely needed. If you have cloned CTSM or updated an existing clone and want to get the latest version of the image files, you can run:
git lfs pull --exclude=""
Note that this is done automatically when building the documentation via the Makefile, as described below.
To see what file types (extensions) are tracked by git-lfs, run the following from within your CTSM clone:
git lfs track
To see every file currently being managed by git-lfs, run the following from within your CTSM clone:
git lfs ls-files --exclude=""
If you want to add a new image file type, with an extension not currently being managed by git-lfs, use this process before adding the file to the git repository. Note that this assumes that you are in the top-level directory of your CTSM clone:
git lfs track "*.extension"
git add .gitattributes
git commit -m "Use git-lfs for *.extension files"
Then you can check that it worked by running git lfs track. Finally,
you can add your new file with standard git add and git commit
commands.
The following procedure assumes you have a clone of CTSM that is checked
out at the branch from which you want to build documentation, and that
you have run ./manage_externals/checkout_externals to update your
externals in your checkout (this is needed to get the README files in
PTCLM, which are referenced by the documentation build).
If you are simply building a test version of the documentation for your
own review, you can do this in a temporary folder anywhere you
want. (But you may want to follow a procedure similar to the one
documented below for updating the official html documentation: That
allows you to do an incremental build rather than building the whole
documentation from scratch, which can take about 1/2 hour. You can just
avoid committing and pushing when following that procedure, if all you
want is to preview the documentation for your own review.) For example,
to do this in a directory named my-ctsm-docs in your home directory,
do the following from the doc directory of your CTSM clone:
mkdir ~/my-ctsm-docs
make BUILDDIR=~/my-ctsm-docs/ html
You can then view your changes in a local browser window using a command like one of these:
open ~/my-ctsm-docs/html/index.html
firefox ~/my-ctsm-docs/html/index.html
konqueror ~/my-ctsm-docs/html/index.html
(Note that the version dropdown menu will not work in these local previews.)
If you want to update the official html documentation, follow the below procedure. (You can also use this procedure if you only intend to build a test version for your own review. This can be helpful because this procedure allows for an incremental build rather than building the entire documentation from scratch.) First clone the ctsm-docs repository to somewhere outside of your main CTSM repository. Here we'll assume that you are cloning this to your home directory:
cd ~
git clone https://github.com/ESCOMP/ctsm-docs.git
Or, if you already have a copy of ctsm-docs, make sure it is up-to-date as follows:
cd ~/ctsm-docs
git checkout -- .
git pull
Next, cd to the doc directory of your main CTSM clone; for example,
if CTSM is also checked out in your home directory:
cd ~/ctsm/doc
Then, to perform an incremental build (only updating files that need to
be updated), run the following. Note that the following assumes you
are building the documentation for the master branch; if you are
building the documentation for one of the release branches, replace
master with the appropriate version.
make BUILDDIR=~/ctsm-docs/versions/master/ html
However, if there have been significant changes to the documentation since the last documentation build, and you are intending to push the new build back to GitHub, it's probably best to first clean out the old build in order to remove any no-longer-relevant files. The downside of doing this is that the full documentation rebuild will take about 1/2 hour:
make BUILDDIR=~/ctsm-docs/versions/master/ clean
make BUILDDIR=~/ctsm-docs/versions/master/ html
You can preview the documentation as noted above.
Then commit and push the rebuilt documentation as follows:
cd ~/ctsm-docs
git add .
git commit -m "YOUR MESSAGE"
git push origin gh-pages
Within a few minutes, the official documentation at https://escomp.github.io/ctsm-docs/ will be updated automatically. Note that you can also fork the ctsm-docs repository and push the gh-pages branch to your fork to allow others to preview the documentation before you overwrite the official documentation.
If you want to add a new version of the built documentation, so that it appears in the dropdown menu, follow this process:
-
In the ctsm-docs repository, create a new directory under
versionswith the name of the new version. Our convention is that this name should be the same as the name of the branch in the main CTSM repository that contains this version of the documentation source. -
In the ctsm-docs repository, add a line in the file
versions/versions.json. Note that each line is a colon-delimited mapping; the name before the colon is the directory name (i.e., the name of the directory you just created, which is the same as the branch name in the CTSM repository), and the name after the colon is the version name that you want to appear in the dropdown menu. Make sure there is a comma at the end of every line except for the last in this file. -
In the main CTSM repository, check out the branch from which you want to build the documentation.
-
In the main CTSM repository, edit the file
doc/source/conf.py: edit theversionandreleasevariables (these should probably be the same as the version name that will appear in the dropdown menu) and anything else that needs to be changed for this release (copyright, etc.) -
Build the documentation as described above, being sure to specify your new directory for
BUILDDIR.
To build a pdf of the tech note (note that we currently do not support
building a pdf of the user's guide), do the following from the doc
directory of your CTSM clone:
mkdir ~/my-ctsm-docs-pdf
make BUILDDIR=~/my-ctsm-docs-pdf/ latexpdf
The pdf will appear at ~/my-ctsm-docs-pdf/latex/clmdoc.pdf.
-
General
-
Documents
-
Bugs/Issues
-
Tutorials
-
Development guides
CTSM Users:
CTSM Developer Team
-
Meetings
-
Notes