Refresh of docs (#201)

Updates of some links and content and then a shiny new theme!!

.gitignore

@@ -7,3 +7,7 @@ staginghub/charts/
 monitoring/charts/
 docs/build/
 hub-ops.gitcrypt.key
+.DS_Store
+requirements.lock
+hub-charts/ea-hub/requirements.lock
+monitoring/requirements.lock

.travis.yml

@@ -17,9 +17,10 @@ cache: pip
 install:
 - |
   # Stage 1: Install pytest + test dependencies
-  pip install -U pyyaml pytest requests sphinx sphinx-autobuild
+  # pip install -U pyyaml pytest requests sphinx sphinx-autobuild
+  pip install -r dev-requirements.txt
 script:
-- | 
+- |
   # Build staginghub
   python ./deploy.py --no-setup --build staginghub
 - |

CONTRIBUTING.rst

@@ -0,0 +1,157 @@
+=======================
+Contributing Guidelines
+=======================
+
+This repository contains the code needed to maintain and update the Earth Lab
+Jupyter Hub.
+
+
+Get Started!
+============
+
+To work on the hub-ops repo, you will need to setup the following.
+
+1. Fork the repository on GitHub
+--------------------------------
+
+To create your own copy of the repository on GitHub, navigate to the
+`earthlab/hub-ops <https://github.com/earthlab/hub-ops>`_ repository
+and click the **Fork** button in the top-right corner of the page.
+
+2. Clone your fork locally
+--------------------------
+
+Use ``git clone`` to get a local copy of your Hub-Ops repository on your
+local filesystem::
+
+    $ git clone git@github.com:your_name_here/hub-ops.git
+    $ cd hub-ops/
+
+3. Set up your fork for local development
+-----------------------------------------
+
+Install Dependencies
+^^^^^^^^^^^^^^^^^^^^
+
+Next you can install the development requirements::
+
+    $ pip install -r dev-requirements.txt
+    $ pre-commit install
+
+4. Create a branch for local development
+----------------------------------------
+
+Use the ``git checkout`` command to create your own branch, and pick a name
+that describes the changes that you are making::
+
+    $ git checkout -b name-of-your-bugfix-or-feature
+
+Now you can make your changes locally.
+
+5. Build the Docs Locally
+-------------------------
+
+Ensure that the tests pass, and the documentation builds successfully::
+
+    $ make html
+
+**Note to Windows users**
+
+To use ``make`` you will need to install and configure GNU Make for Windows,
+e.g., using chocolatey: https://chocolatey.org/packages/make
+
+
+6. Commit and push your changes
+-------------------------------
+
+Once you are sure that all tests are passing, you can commit your changes
+and push to GitHub::
+
+    $ git add .
+    $ git commit -m "Your detailed description of your changes."
+    $ git push origin name-of-your-bugfix-or-feature
+
+7. Submit a pull request on GitHub
+----------------------------------
+
+When submitting a pull request:
+
+- All existing tests should pass. Please make sure that the test
+  suite passes, both locally and on
+  `Travis CI <https://travis-ci.org/earthlab/hub-ops>`_
+  Status on
+  Travis will be visible on a pull request. If you want to enable
+  Travis CI on your own fork, please read the
+  `getting started docs <https://docs.travis-ci.com/user/getting-started/>`_.
+
+- New functionality should include tests. Please write reasonable
+  tests for your code and make sure that they pass on your pull request.
+
+- Classes, methods, functions, etc. should have docstrings. The first
+  line of a docstring should be a standalone summary. Parameters and
+  return values should be documented explicitly.
+
+- The API documentation is automatically generated from docstrings, which
+  should conform to NumpPy styling. For examples, see the `Napoleon docs
+  <https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html>`_.
+
+- Please note that tests are also run via Travis-CI on our documentation.
+  So be sure that any ``.rst`` file submissions are properly formatted and
+  tests are passing.
+
+
+Documentation Updates
+=====================
+
+Improving the documentation and testing for code already in Hub-Ops
+is a great way to get started if you'd like to make a contribution. Please note
+that our documentation files are in
+`ReStructuredText (.rst)
+<http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_
+format and format your pull request accordingly.
+
+To create a vignette for an Hub-Ops workflow, create a ``.py`` file that shows the
+necessary steps to complete the workflow. Make sure the file name begins with
+``plot`` in order to ensure that the vignette is correctly built. Store the
+vignette in the ``examples`` folder within the ``hub-ops`` directory. Hub-Ops
+uses Sphinx Gallery to build vignettes. Help for formatting and building
+vignettes can be found on `their website <https://sphinx-gallery.github.io>`_.
+
+
+To build the documentation, use the command::
+
+    $ make docs
+
+By default ``make docs`` will only rebuild the documentation if source
+files (e.g., .py or .rst files) have changed. To force a rebuild, use
+``make -B docs``.
+You can preview the generated documentation by opening
+``docs/_build/html/index.html`` in a web browser.
+
+
+Code Style
+==========
+
+- Hub-Ops currently only supports Python 3 (3.5+). Please test code locally
+  in Python 3 when possible (all supported versions will be automatically
+  tested on Travis CI).
+
+- Hub-Ops uses a pre-commit hook that runs the black code autoformatter.
+  Be sure to execute `pre-commit install` as described above, which will cause
+  black to autoformat code prior to commits. If this step is skipped, black
+  may cause build failures on Travis CI due to formatting issues.
+
+- Follow `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_ when possible.
+  Some standards that we follow include:
+
+    - The first word of a comment should be capitalized with a space following
+      the ``#`` sign like this: ``# This is a comment here``
+    - Variable and function names should be all lowercase with words separated
+      by ``_``.
+    - Class definitions should use camel case - example: ``ClassNameHere`` .
+
+- Imports should be grouped with standard library imports first,
+  3rd-party libraries next, and Hub-Ops imports third following PEP 8
+  standards. Within each grouping, imports should be alphabetized. Always use
+  absolute imports when possible, and explicit relative imports for local
+  imports when necessary in tests.

dev-requirements.txt

@@ -0,0 +1,7 @@
+pyyaml
+pytest
+requests
+sphinx
+sphinx-autobuild
+sphinx_copybutton
+pydata_sphinx_theme

docs/source/additional-hub-configuration.rst

@@ -1,7 +1,7 @@
-Additional ideas and snippets to customise your hub
-===================================================
+Customize Your Hub
+==================
 
-This is a collection of snippets and pointers to further customise your hub.
+This is a collection of snippets and pointers to further customize your hub.
 
 
 Custom user image

docs/source/authentication.rst

@@ -1,6 +1,6 @@
 .. _authentication:
 
-Authentication for your hub
+Hub Authentication
 ===========================
 
 This section documents the different options of configuring authentication

docs/source/conf.py

@@ -15,13 +15,13 @@
 # import os
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))
-
+from datetime import date
 
 # -- Project information -----------------------------------------------------
 
-project = 'Earthlab hub-ops'
-copyright = '2018, Earthlabbers'
-author = 'Earthlabbers'
+project = 'Earth Lab JupyterHub Operations'
+copyright = '{year}, Earth Lab Contributors'.format(year=date.today().year)
+author = 'Earth Lab'
 
 # The short X.Y version
 version = ''
@@ -38,8 +38,7 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = [
-]
+extensions = ['sphinx_copybutton']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -74,7 +73,8 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+html_theme = 'pydata_sphinx_theme'#'alabaster'
+html_logo = '_static/images/earth-lab-logo.png'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -152,4 +152,4 @@
     (master_doc, 'Earthlabhub-ops', 'Earthlab hub-ops Documentation',
      author, 'Earthlabhub-ops', 'One line description of project.',
      'Miscellaneous'),
-]
+]

docs/source/daily-operations/hub-monitoring.rst

@@ -0,0 +1,82 @@
+Monitoring the Hubs
+===================
+
+To get an overview of the health of the hubs and infrastructure visit the
+`Grafana page <https://grafana.hub.earthdatascience.org/>`_.
+
+The ``Hub monitor`` page lets you see how many pods are running, launch success
+rate, which users are using a lot of CPU, etc. For each hub separately.
+
+The ``Node monitor`` page contains information about each of the compute nodes
+that are part of the cluster.
+
+The best way to use the monitoring is to watch it for a while when not a lot
+is happening to get a feeling for what "baseline" looks like. Then login to
+a hub and observe what it looks like on the monitoring. Next time a workshop
+is happening keep an eye on the monitoring to see what happens when lots of
+people login at the same time.
+
+
+Operations
+----------
+
+This section contains commands and snippets that let you inspect the state of
+the cluster and perform tasks that are useful when things are broken.
+
+To perform these commands you need to have ``kubectl`` installed and setup
+on your local laptop (see :ref:`google-cloud` for details).
+
+
+Inspecting what is going on in the cluster
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To see what is running on the cluster (and get a feeling for what is the normal
+state of affairs) run ``kubectl get pods --all-namespaces``. This will list all
+pods that are running, including service pods that we don't ever interact with.
+
+You should see at least two pods in each namespace that is associated to a hub.
+The namespace and hubname are the same, so ``staginghub`` lives in the
+``staginghub`` namespace.
+
+Pods in the ``kube-system``, ``monitoring`` and ``router`` namespaces are best
+left alone.
+
+Each hub specific namespace should contain at least two pods: ``hub-77fbd96bb-dh2b5``
+and ``proxy-6549f4fbc8-8zn67``. Everything after hub- and proxy- will change
+when you restart the hub or make configuration changes. The status of these
+two pods should be ``Running``.
+
+To see what a pod is printing to its terminal run ``kubectl logs <podname> --namespace <hubname>``.
+This will let you see errors or exceptions that happened.
+
+You can find out more about a pod by running ``kubectl describe pod <podname> --namespace <hubname>``.
+This will give you information on why a pod is not running or what it is up to
+while you are waiting for it to start running.
+
+When someone logs in to the hub and starts their server a new pod will appear in
+the namespace of the hub. The pod will be named ``jupyter-<username>``. You can
+inspect it with the usual ``kubectl`` commands.
+
+
+Known problems and solutions
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There is a known problem in one of JupyterHub's components that means sometimes
+the hub misses that a user's pod has started and will keep that user waiting
+after they logged in. The symptoms of this are that there is a running pod for
+a user in the right namespace but the login process does not complete. In this
+case restart the offending hub by running ``kubectl delete pod hub-... --namespace <hubname>``,
+replacing the ... in the hub name with the proper name of the hub pod. This should
+not interrupt currently active users and fixes a lot of things that can go wrong.
+
+
+Inspecting virtual machines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To tell how many virtual machines (or nodes) are part of the cluster run
+``kubectl get nodes``. There should be at least one node with ``core-pool`` in
+its name running at all times. Once users login and start their servers new
+nodes will appear that have ``user-pool`` in their name. These nodes are
+automatically created and destroyed based on demand.
+
+You can learn about a node by running ``kubectl describe node <nodename>``.

docs/source/daily-operations/index.rst

@@ -0,0 +1,17 @@
+================
+Daily Operations
+================
+
+This section outlines how to perform typical tasks for the Earth Lab, University
+of Colorado Jupyter Hubs.
+
+Get started
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Daily Operations
+
+   scaling
+   hub-monitoring
+   new-hub
+   modify-remove-hub