Skip to content

Directions for editing CLM documentation on github and sphinx

Jump to bottom
Bill Sacks edited this page May 6, 2020 · 56 revisions

Intro to the CTSM documentation

The CTSM documentation is written using reStructuredText markup language. ReStructuredText is a markup language, like HTML, markdown or latex. (See below for more resources on learning reStructuredText.)

Sphinx is a python tool for publishing reStructuredText documents in other formats such as HTML and PDF.

The CTSM documentation source is stored in the doc/source directory of the main CTSM repository. The built documentation is stored in the gh-pages branch of the ESCOMP/ctsm-docs repository. The Sphinx-generated HTML pages are accessible from URL https://escomp.github.io/ctsm-docs/

What do I, as a contributor, need to do to update the tech note for my new feature?

As a contributor to CTSM, you don't necessarily need to know all of the information laid out below. If you need to update the tech note for some changes you have made, you should add or edit the necessary rst files in doc/source/tech_note on your CTSM branch. Then simply commit your changes to these rst files just as you would do for changes to Fortran source files, and include these rst changes in your Pull Request. Ideally these documentation changes will be on the same branch and the same Pull Request as the related source file changes, but they can also be made later, on a separate branch, and submitted to us via a separate Pull Request. In either case, the CTSM maintainers will review the documentation changes in the Pull Request and merge these changes to CTSM's master branch when ready.

You do not need to open a Pull Request for the built documentation: one of the CTSM maintainers will rebuild the documentation later. However, if you have made more than minor edits to the documentation, we prefer if you have tested the documentation build to make sure there are no errors in the edited rst files and that the html documentation appears how you intended. But you do not need to send us or point us to your rebuilt documentation: One of the CTSM maintainers will build the documentation themselves to review the build.

If you need to change or add any images, you will need to keep reading for instructions on installing and using git lfs.

One-time setup needed for a given machine

The following packages need to be installed on the machine where you want to build the documentation:

  • Git-LFS (git large file support; needed to obtain or modify image files)
  • Sphinx
  • The correct version of the Sphinx theme we use

These instructions have been tested on a Mac. Previously, support for the documentation build was also provided on CGD Linux machines (e.g., sunshine.cgd.ucar.edu and thorodin.cgd.ucar.edu), but this may not work currently due to requirements for git-lfs and the necessary Sphinx theme. We will update these instructions as we add support on CGD machines. For now, at least, we recommend working on your own machine - e.g., your Mac desktop / laptop.

Prerequisites / assumptions

The following documentation assumes that you have the following available on your machine:

Installing git-lfs

Follow the instructions on the Git LFS page for installing git-lfs on your platform.

On a mac using homebrew, this can be done with:

brew install git-lfs

Although the actual installation only needs to be done once per machine, each user who wants to use it must run the following command to set up git-lfs for your user account on this machine:

git lfs install

Installing Sphinx and the necessary Sphinx theme

Sphinx and the necessary Sphinx theme are python packages that can be installed with pip install:

pip install sphinx
pip install git+https://github.com/esmci/sphinx_rtd_theme.git@version-dropdown-with-fixes

For more details on installing Sphinx, see the Sphinx documentation.

You may then need to update your PATH environment variable to include the path to the sphinx-build script.

Optionally installing components needed to build the PDF

If you want to build the PDF (not just the html web pages), you will need the latexmk tool. On a Mac, this can be obtained with:

brew cask install mactex

You may also need rst2pdf. This can be installed with:

pip install rst2pdf

Management of image files

Image files are tracked using Git LFS (Large File Support). The images are stored somewhere on GitHub, but aren't actually part of the main repository. Instead, the repository contains small text files that direct git-lfs to the appropriate storage location on GitHub.

For the most part, you can remain blissfully unaware of the details of how this works. For example, if you want to add a new image file to the repository, the process is just like adding any other file to the repository, as long as you have installed git-lfs. In addition, building the documentation does not require any additional steps - again, as long as you have installed git-lfs. But here are some notes that may help you:

Obtaining the image files

We have configured git-lfs for CTSM so that it does not pull down any images by default, since these images are only rarely needed. If you have cloned CTSM or updated an existing clone and want to get the latest version of the image files, you can run:

git lfs pull --exclude=""

Note that this is done automatically when building the documentation via the Makefile, as described below.

Seeing what files are tracked by git-lfs

To see what file types (extensions) are tracked by git-lfs, run the following from within your CTSM clone:

git lfs track

To see every file currently being managed by git-lfs, run the following from within your CTSM clone:

git lfs ls-files --exclude=""

Adding a new image file type

If you want to add a new image file type, with an extension not currently being managed by git-lfs, use this process before adding the file to the git repository. Note that this assumes that you are in the top-level directory of your CTSM clone:

git lfs track "*.extension"
git add .gitattributes
git commit -m "Use git-lfs for *.extension files"

Then you can check that it worked by running git lfs track. Finally, you can add your new file with standard git add and git commit commands.

Procedure for building the html documentation

The following procedure assumes you have a clone of CTSM that is checked out at the branch from which you want to build documentation, and that you have run ./manage_externals/checkout_externals to update your externals in your checkout (this is needed to get the README files in PTCLM, which are referenced by the documentation build).

Building a test version of the documentation for your own review

If you are simply building a test version of the documentation for your own review, you can do this in a temporary folder anywhere you want. (But you may want to follow a procedure similar to the one documented below for updating the official html documentation: That allows you to do an incremental build rather than building the whole documentation from scratch, which can take about 1/2 hour. You can just avoid committing and pushing when following that procedure, if all you want is to preview the documentation for your own review.) For example, to do this in a directory named my-ctsm-docs in your home directory, do the following from the doc directory of your CTSM clone:

mkdir ~/my-ctsm-docs
make BUILDDIR=~/my-ctsm-docs/ html

You can then view your changes in a local browser window using a command like one of these:

open ~/my-ctsm-docs/html/index.html
firefox ~/my-ctsm-docs/html/index.html
konqueror ~/my-ctsm-docs/html/index.html

(Note that the version dropdown menu will not work in these local previews.)

Updating the official html documentation

If you want to update the official html documentation, follow the below procedure. (You can also use this procedure if you only intend to build a test version for your own review. This can be helpful because this procedure allows for an incremental build rather than building the entire documentation from scratch.) First clone the ctsm-docs repository to somewhere outside of your main CTSM repository. Here we'll assume that you are cloning this to your home directory:

cd ~
git clone https://github.com/ESCOMP/ctsm-docs.git

Or, if you already have a copy of ctsm-docs, make sure it is up-to-date as follows:

cd ~/ctsm-docs
git checkout -- .
git pull

Next, cd to the doc directory of your main CTSM clone; for example, if CTSM is also checked out in your home directory:

cd ~/ctsm/doc

Then, to perform an incremental build (only updating files that need to be updated), run the following. Note that the following assumes you are building the documentation for the master branch; if you are building the documentation for one of the release branches, replace master with the appropriate version.

make BUILDDIR=~/ctsm-docs/versions/master/ html

However, if there have been significant changes to the documentation since the last documentation build, and you are intending to push the new build back to GitHub, it's probably best to first clean out the old build in order to remove any no-longer-relevant files. The downside of doing this is that the full documentation rebuild will take about 1/2 hour:

make BUILDDIR=~/ctsm-docs/versions/master/ clean
make BUILDDIR=~/ctsm-docs/versions/master/ html

You can preview the documentation as noted above.

Then commit and push the rebuilt documentation as follows:

cd ~/ctsm-docs
git add .
git commit -m "YOUR MESSAGE"
git push origin gh-pages

Within a few minutes, the official documentation at https://escomp.github.io/ctsm-docs/ will be updated automatically. Note that you can also fork the ctsm-docs repository and push the gh-pages branch to your fork to allow others to preview the documentation before you overwrite the official documentation.

Adding a new version

If you want to add a new version of the built documentation, so that it appears in the dropdown menu, follow this process:

  • In the ctsm-docs repository, create a new directory under versions with the name of the new version. Our convention is that this name should be the same as the name of the branch in the main CTSM repository that contains this version of the documentation source.

  • In the ctsm-docs repository, add a line in the file versions/versions.json. Note that each line is a colon-delimited mapping; the name before the colon is the directory name (i.e., the name of the directory you just created, which is the same as the branch name in the CTSM repository), and the name after the colon is the version name that you want to appear in the dropdown menu. Make sure there is a comma at the end of every line except for the last in this file.

  • In the main CTSM repository, check out the branch from which you want to build the documentation.

  • In the main CTSM repository, edit the file doc/source/conf.py: edit the version and release variables (these should probably be the same as the version name that will appear in the dropdown menu) and anything else that needs to be changed for this release (copyright, etc.)

  • Build the documentation as described above, being sure to specify your new directory for BUILDDIR.

Building a pdf of the tech note

To build a pdf of the tech note (note that we currently do not support building a pdf of the user's guide), do the following from the doc directory of your CTSM clone:

mkdir ~/my-ctsm-docs-pdf
make BUILDDIR=~/my-ctsm-docs-pdf/ latexpdf

The pdf will appear at ~/my-ctsm-docs-pdf/latex/clmdoc.pdf.

Resources for learning markup with reStructuredText and using Sphinx.

Clone this wiki locally