Skip to content

Directions for editing CLM documentation on github and sphinx

Jump to bottom
Bill Sacks edited this page Oct 30, 2020 · 56 revisions

Table of contents

Created by gh-md-toc

Intro to the CTSM documentation

The CTSM documentation is written using reStructuredText markup language. ReStructuredText is a markup language, like HTML, markdown or latex. (See below for more resources on learning reStructuredText.)

Sphinx is a python tool for publishing reStructuredText documents in other formats such as HTML and PDF.

The CTSM documentation source is stored in the doc/source directory of the main CTSM repository. The built documentation is stored in the gh-pages branch of the ESCOMP/ctsm-docs repository. The Sphinx-generated HTML pages are accessible from URL https://escomp.github.io/ctsm-docs/

What do I, as a contributor, need to do to update the tech note for my new feature?

As a contributor to CTSM, you don't necessarily need to know all of the information laid out below. If you need to update the tech note for some changes you have made, you should add or edit the necessary rst files in doc/source/tech_note on your CTSM branch. Then simply commit your changes to these rst files just as you would do for changes to Fortran source files, and include these rst changes in your Pull Request. Ideally these documentation changes will be on the same branch and the same Pull Request as the related source file changes, but they can also be made later, on a separate branch, and submitted to us via a separate Pull Request. In either case, the CTSM maintainers will review the documentation changes in the Pull Request and merge these changes to CTSM's master branch when ready.

You do not need to open a Pull Request for the built documentation: one of the CTSM maintainers will rebuild the documentation later. However, if you have made more than minor edits to the documentation, we prefer if you have tested the documentation build to make sure there are no errors in the edited rst files and that the html documentation appears how you intended. But you do not need to send us or point us to your rebuilt documentation: One of the CTSM maintainers will build the documentation themselves to review the build.

If you need to build the documentation, and/or if you need to change or add any images, you will need to keep reading for instructions on installing and using git lfs.

Recommended directory structure for these directions

In order to give concrete examples, we assume a particular directory structure for these directions: We assume you have a directory in your home directory named ctsm-repos, and that you clone the main CTSM repository and the ctsm-docs repository inside that directory. So you will have something like:

~/ctsm-repos/ctsm
~/ctsm-repos/ctsm-docs

It's fine for your ctsm directory to be named something more specific, for example, including the branch name as is recommended elsewhere in this wiki.

You can use a different directory organization if you wish, but if you are using the preferred, Docker container-based method, then it is important for the ctsm-docs directory to be outside of the main ctsm directory, and it is best to have both your main ctsm directory and your ctsm-docs directory reside under a common parent directory that is somewhere below your home directory). (These are important so that the Docker image gets mounted somewhere outside of the ctsm clone, which is critical to avoid issues with the documentation build, but below your home directory, which can be important to avoid conflicts with your personal dot files like .bashrc.)

One-time setup needed for a given machine

The directions here assume use of a Docker container-based method for building the documentation. Other methods are documented in an Appendix below. We prefer the Docker-based method because it avoids the complexity of installing multiple packages on your local machine, and ensures tat all people building the documentation are using the same versions of Sphinx and other utilities.

Install Docker and download the escomp/base container image

Download Docker to your personal desktop or laptop from Docker and follow the instructions. Then launch the Docker application.

You can then obtain the necessary Docker container by running:

docker pull escomp/base

Install git-lfs

Git-LFS (Large File Support) is needed if you will be adding or editing any image files. If there is even a chance that you will be doing so, it is best to go ahead and install this now, while you are thinking about it. This will prevent you from accidentally committing an image file directly to the repository, which is something we try very hard to avoid. This is also needed for building the documentation if not using the Docker-based method.

Follow the instructions on the Git LFS page for installing git-lfs on your platform.

On a mac using homebrew, this can be done with:

brew install git-lfs

Although the actual installation only needs to be done once per machine, each user who wants to use it must then run the following command to set up git-lfs for your user account on this machine:

git lfs install

Management of image files

If you are adding or changing any image files, be sure you have installed git-lfs according to the [above instructions](#Install git-lfs), including running git lfs install. If you are not sure whether you have already run git lfs install, it is safe to rerun that command to be sure.

Image files are tracked using Git LFS (Large File Support). The images are stored somewhere on GitHub, but aren't actually part of the main repository. Instead, the repository contains small text files that direct git-lfs to the appropriate storage location on GitHub.

For the most part, you can remain blissfully unaware of the details of how this works. For example, if you want to add a new image file to the repository, the process is just like adding any other file to the repository, as long as you have installed git-lfs. In addition, building the documentation does not require any additional steps - again, as long as you have installed git-lfs. But here are some notes that may help you:

Obtaining the image files

We have configured git-lfs for CTSM so that it does not pull down any images by default, since these images are only rarely needed. If you have cloned CTSM or updated an existing clone and want to get the latest version of the image files, you can run:

git lfs pull --exclude=""

Note that this is done automatically when building the documentation via the Makefile, as described below.

Seeing what files are tracked by git-lfs

To see what file types (extensions) are tracked by git-lfs, run the following from within your CTSM clone:

git lfs track

To see every file currently being managed by git-lfs, run the following from within your CTSM clone:

git lfs ls-files --exclude=""

Adding a new image file type

If you want to add a new image file type, with an extension not currently being managed by git-lfs, use this process before adding the file to the git repository. Note that this assumes that you are in the top-level directory of your CTSM clone:

git lfs track "*.extension"
git add .gitattributes
git commit -m "Use git-lfs for *.extension files"

Then you can check that it worked by running git lfs track. Finally, you can add your new file with standard git add and git commit commands.

Procedure for building the html documentation

The following procedure assumes you have a clone of CTSM that is checked out at the branch from which you want to build documentation.

Initial steps for building the documentation

Before building the documentation, you need to do the following:

First, run ./manage_externals/checkout_externals -o to update your externals; this is needed both to get PTCLM (which has some files referenced by the documentation build) and to get the doc-builder external that is used to do the build. Note the use of the -o (--optional) flag here; this is needed because doc-builder is an optional external.

Next, if it's not already running, launch the Docker desktop application.

If you haven't updated the escomp/base image in a while, you may want to update to the latest version with:

docker pull escomp/base

Building a test version of the documentation for your own review

If you are just building a test version of the documentation for your own review, then you don't need to do anything with the ctsm-docs GitHub repository. Instead, you can create a directory according to the above recommendations for directory structure.

(However, you may want to follow a procedure similar to the one documented below for updating the official html documentation: That allows you to do an incremental build rather than building the whole documentation from scratch, which can take about 1/2 hour. You can just avoid committing and pushing when following that procedure, if all you want is to preview the documentation for your own review.)

If you are using the above recommendations, do the following from the doc directory of your ctsm clone:

mkdir ~/ctsm-repos/ctsm-docs
./build_docs -b ~/ctsm-repos/ctsm-docs -d

(where the -d flag instructs build_docs to use the escomp/base Docker container to do the build).

Previewing the built documentation

You can then view your changes in a local browser window using a command like one of these:

open ~/ctsm-repos/ctsm-docs/html/index.html
firefox ~/ctsm-repos/ctsm-docs/html/index.html
konqueror ~/ctsm-repos/ctsm-docs/html/index.html

Note that the version dropdown menu will not work in these local previews.

Updating the official html documentation

If you want to update the official html documentation, follow the below procedure. (You can also use this procedure if you only intend to build a test version for your own review. This can be helpful because this procedure allows for an incremental build rather than building the entire documentation from scratch.) First clone the ctsm-docs repository to somewhere outside of your main CTSM repository. Here we'll assume that you are using the above recommendations for directory structure.

cd ~/ctsm-repos
git clone https://github.com/ESCOMP/ctsm-docs.git

Or, if you already have a copy of ctsm-docs, make sure it is up-to-date as follows:

cd ~/ctsm-repos/ctsm-docs
git checkout -- .
git pull

Next, cd to the doc directory of your main CTSM clone:

cd ~/ctsm-repos/ctsm/doc

Then, to perform an incremental build (only updating files that need to be updated), run the following. Note that the following assumes you are building the documentation for the master branch; if you are building the documentation for one of the release branches, replace master with the appropriate version.

./build_docs -b ~/ctsm-repos/ctsm-docs/versions/master -d

(where the -d flag instructs build_docs to use the escomp/base Docker container to do the build).

However, if there have been significant changes to the documentation since the last documentation build, and you are intending to push the new build back to GitHub, it's probably best to first clean out the old build in order to remove any no-longer-relevant files. The downside of doing this is that the full documentation rebuild will take about 1/2 hour:

./build_docs -b ~/ctsm-repos/ctsm-docs/versions/master -d -c

(where the -c flag instructs build_docs to first run make clean in the specified directory).

You can preview the documentation as noted [above](#Previewing the built documentation).

Then commit and push the rebuilt documentation as follows:

cd ~/ctsm-repos/ctsm-docs
git add .
git commit -m "YOUR MESSAGE"
git push origin gh-pages

Within a few minutes, the official documentation at https://escomp.github.io/ctsm-docs/ will be updated automatically. Note that you can also fork the ctsm-docs repository and push the gh-pages branch to your fork to allow others to preview the documentation before you overwrite the official documentation.

Adding a new version

If you want to add a new version of the built documentation, so that it appears in the dropdown menu, follow this process:

  • In the ctsm-docs repository, create a new directory under versions with the name of the new version. Our convention is that this name should be the same as the name of the branch in the main CTSM repository that contains this version of the documentation source.

  • In the ctsm-docs repository, add a line in the file versions/versions.json. Note that each line is a colon-delimited mapping; the name before the colon is the directory name (i.e., the name of the directory you just created, which is the same as the branch name in the CTSM repository), and the name after the colon is the version name that you want to appear in the dropdown menu. Make sure there is a comma at the end of every line except for the last in this file.

  • In the main CTSM repository, check out the branch from which you want to build the documentation.

  • In the main CTSM repository, edit the file doc/source/conf.py: edit the version and release variables (these should probably be the same as the version name that will appear in the dropdown menu) and anything else that needs to be changed for this release (copyright, etc.)

  • Build the documentation as described above, being sure to specify your new directory for BUILDDIR.

Building a pdf of the tech note

To build a pdf of the tech note (note that we currently do not support building a pdf of the user's guide), do the following from the doc directory of your CTSM clone:

mkdir ~/ctsm-repos/ctsm-docs-pdf
./build_docs -b ~/ctsm-repos/ctsm-docs-pdf -d -t latexpdf

The pdf will appear at ~/ctsm-repos/ctsm-docs-pdf/latex/clmdoc.pdf.

Resources for learning markup with reStructuredText and using Sphinx.

Appendix: Non-Docker installation methods

If you choose not to use the Docker-based method described above, you can instead install Sphinx and some related packages locally. This section documents some alternatives. We do not recommend these methods, because they can be more challenging to set up than the method described above. In addition, the instructions here are not regularly tested, so may no longer work. Finally, the use of one of these alternatives can mean that your Sphinx version differs from the one in use to generate the official documentation, which can potentially cause problems. However, if you cannot use the Docker method for some reason, then you can try one of these alternatives.

Installing required software on your desktop/laptop (instructions for Mac)

Prerequisites / assumptions

The following documentation assumes that you have the following available on your machine:

Installing Sphinx and the necessary Sphinx theme

Sphinx and the necessary Sphinx theme are python packages that can be installed with pip install:

pip install sphinx
pip install git+https://github.com/esmci/sphinx_rtd_theme.git@version-dropdown-with-fixes

For more details on installing Sphinx, see the Sphinx documentation.

You may then need to update your PATH environment variable to include the path to the sphinx-build script.

Installing latexmk

latexmk is needed both for building the PDF and for creating the equations in the html documentation. On a mac using homebrew, this can be installed with:

brew cask install mactex

Note that you will need to open a new terminal window for the latexmk tool to be added to your path (it is in /Library/TeX/texbin/latexmk, which should have been added to your path by the above brew command).

Optionally installing components needed to build the PDF

If you want to build the PDF (not just the html web pages), you may also need rst2pdf. This can be installed with:

pip install rst2pdf

Building the documentation on cheyenne

Finally, although we haven't figured out a way to view the built documentation on cheyenne, you can test the build there, and then transfer the files to your local machine for viewing.

To do so, first ensure that you are using the git and python modules on cheyenne rather than the default system versions, by doing module load git and module load python. (Note that the default git module has git lfs bundled with it.)

Then, the first time you build the documentation, do this one-time setup:

pip install --user sphinx
pip install --user sphinxcontrib-programoutput
pip install --user git+https://github.com/esmci/sphinx_rtd_theme.git@version-dropdown-with-fixes

and add the following to your .bashrc or similar file if using bash:

export PATH=/glade/u/home/$USER/.local/bin:$PATH

or, if you're using tcsh, add to your .tcshrc:

setenv PATH /glade/u/home/$USER/.local/bin:$PATH

Clone this wiki locally