Merge pull request #13 from lsst-sqre/tickets/DM-20954

DM-20954: Support asset files associated with a report template

.travis.yml

@@ -6,7 +6,7 @@ matrix:
     - python: "3.6"
       env: PYPI_DEPLOY=true LTD_SKIP_UPLOAD=false
 install:
-  - "pip install -e .[dev]"
+  - "pip install .[dev]"
   - "pip install ltd-conveyor"
 script:
   - "python setup.py test"

CHANGELOG.rst

@@ -2,6 +2,15 @@
 Change log
 ##########
 
+0.7.3 (2019-08-28)
+==================
+
+- New ``assets`` field in the ``nbreport.yaml`` configuration file lets a user specify files in the template repository that are meant to be copied into a report instance's directory.
+  Assets can be data files or Python modules that are needed for the report notebook to run.
+- Added reference documentation for the ``nbreport.yaml`` file and also added how-to pages related to assets.
+
+`DM-20954 <https://jira.lsstcorp.org/browse/DM-20954>`__.
+
 0.7.2 (2019-08-08)
 ==================
 

docs/index.rst

@@ -35,8 +35,9 @@ Documentation
 =============
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
+   repository-guide/index
    cli-reference
    api-reference
    changelog

docs/repository-guide/how-to-include-data-files-as-assets.rst

@@ -0,0 +1,75 @@
+.. _how-to-use-assets:
+
+############################################################
+How to include data files with a notebook template as assets
+############################################################
+
+Your report notebook might require additional files to run.
+These might be data files, like CSV, YAML, HDF5, or FITS.
+
+One approach is to maintain these data files on a web server and use a Python library like Requests_ to download that data on demand.
+
+In cases where the data files are highly coupled and specific to the report, though, it's natural to bundle these files with the templated report itself.
+This approach ensures that data files are maintained and versioned in step with the report template since both the data and code are co-located in the same Git repository.
+nbreport enables you to do this with a feature called :ref:`assets <yaml-assets>`.
+
+This pages guides you through using the assets feature.
+
+.. seealso::
+
+   Another application of assets is for including Python modules alongside a notebook.
+   See :doc:`how-to-use-python-modules` for details.
+
+Step 1. Commit the data files into the Git repository of the notebook template
+==============================================================================
+
+Commit the data files into the Git repository of the notebook template.
+These files must be in the same directory as the notebook (``ipynb`` file) or a subdirectory relative to the notebook file.
+
+For example, the template repository contents might look like this:
+
+.. code-block:: text
+
+   .
+   ├── conf.yaml
+   ├── images
+   │   ├── image1.files
+   │   └── image2.files
+   ├── nbreport.yaml
+   └── TEST-000.ipynb
+
+The assets are :file:`conf.yaml` file and the contents of the :file:`images` directory.
+
+Step 2. Register the files as "assets" in the nbreport.yaml file
+================================================================
+
+Open the :file:`nbreport.yaml` file in the notebook template repository.
+Add a key called ``assets`` to that YAML file, and then add list items to that key that match the paths of data files.
+File paths, directory names, and globs are supported — see the :ref:`assets reference documentation <yaml-assets>` for details.
+
+To match the assets in the previous example, the ``assets`` configuration might be:
+
+.. code-block:: yaml
+
+   assets:
+     - 'conf.yaml'
+     - 'images'
+
+Alternatively, to ensure that only FITS files in the :file:`images` directory (and its subdirectories) are matched, you might instead use a glob:
+
+.. code-block:: yaml
+
+   assets:
+     - 'conf.yaml'
+     - 'images/**/*.fits'
+
+When a new report instance is created with the `nbreport init`_ or `nbreport issue`_ commands, the files marked as assets are copied from the notebook template repository into the instance's working directory.
+Directory structure is preserved.
+These files **are not** processed by `Jinja templates`_.
+
+For more information about the ``assets`` field in the :file:`nbreport.yaml` file, see the :ref:`nbreport.yaml reference <yaml-assets>`.
+
+.. _nbreport init: ../cli-reference.html#nbreport-init
+.. _nbreport issue: ../cli-reference.html#nbreport-issue
+.. _Requests: https://2.python-requests.org/en/master/
+.. _Jinja templates: https://palletsprojects.com/p/jinja/

docs/repository-guide/how-to-use-python-modules.rst

@@ -0,0 +1,69 @@
+.. _how-to-use-modules:
+
+#######################################################
+How to use Python modules in a report notebook template
+#######################################################
+
+Your report notebook might import Python modules in order to work.
+Importing modules is useful because it means that code can be reused across multiple applications.
+It also means that the implementation is hidden from the rendered report instances so that your report presentation can focus on results, rather than code.
+
+There are two ways to use external Python modules from a report notebook:
+
+1. Import from an installed package (for example, ``import astropy``, or ``from lsst.daf.butler import Butler``).
+2. Import from Python modules (``*.py`` files) in the report's directory.
+
+In the first case, nbreport doesn't currently help you set up a Python environment and install packages with ``pip``, ``conda``, or ``eups distrib install``.
+It's assumed that you have set up your Python environment before running nbreport and computing a report instance.
+
+The rest of this page deals with the second case of working with local Python modules in notebook template repositories.
+
+Step 1. Commit the Python module into the notebook repository
+=============================================================
+
+Suppose that your notebook imports a function from a module named :file:`analysis.py`:
+
+.. code-block:: py
+
+   from analysis import run
+
+In this case, commit the file :file:`analysis.py` into the report repository, alongside the Jupyter notebook file.
+
+The report template repository contents will look like this:
+
+.. code-block:: text
+
+   .
+   ├── analysis.py
+   ├── cookiecutter.json
+   ├── nbreport.yaml
+   └── TEST-000.ipynb
+
+Step 2. Designate the Python module as an asset in nbreport.yaml
+================================================================
+
+Edit the :file:`nbreport.yaml` file to designate the Python module as an *asset*.
+Specifically add these lines:
+
+.. code-block:: yaml
+
+   assets:
+     - 'analysis.py'
+
+Alternatively, to ensure that *any* Python module is treated as an asset you can use a glob:
+
+.. code-block:: yaml
+
+   assets:
+     - '**/*.py'
+
+Marking Python files as assets is necessary so that these files are copied from the report repository when a report instance is rendered and computed by the `nbreport init`_ or `nbreport issue`_ commands.
+
+Keep in mind that assets are not rendered as templates.
+*Assets aren't processed by Jinja.*
+The natural way to make code in a module respond to template variables is to pass those variables as arguments from the notebook code to the module code.
+
+For more information about the ``assets`` field in the :file:`nbreport.yaml` file, see the :ref:`nbreport.yaml reference <yaml-assets>`.
+
+.. _nbreport init: ../cli-reference.html#nbreport-init
+.. _nbreport issue: ../cli-reference.html#nbreport-issue

docs/repository-guide/index.rst

@@ -0,0 +1,10 @@
+#######################################
+Guide to developing report repositories
+#######################################
+
+.. toctree::
+   :maxdepth: 1
+
+   how-to-include-data-files-as-assets
+   how-to-use-python-modules
+   nbreport-yaml-reference

docs/repository-guide/nbreport-yaml-reference.rst

@@ -0,0 +1,200 @@
+.. _yaml-reference:
+
+###################################
+nbreport.yaml file format reference
+###################################
+
+Every report template repository has an :file:`nbreport.yaml` file that defines metadata about the report repository.
+This page describes every field you might find in an :file:`nbreport.yaml` file, keeping in mind that some fields are optional.
+
+An example :file:`nbreport.yaml` file:
+
+.. code-block:: yaml
+
+   handle: TESTR-001
+   title: Characterization Metric Report Demo
+   git_repo: https://github.com/lsst-sqre/nbreport
+   ipynb: TESTR-001.ipynb
+   assets:
+     - '**/*.fits'
+     - 'mydata.csv'
+   git_repo_subdir: tests/TESTR-001
+   ltd_product: testr-001
+   published_url: https://testr-001.lsst.io
+   ltd_url: https://keeper.lsst.codes/products/testr-001
+
+handle
+======
+
+The handle is a short identifier for the report series.
+nbreport does not prescribe a format for handles, but typically there is a series prefix and a serial number suffix:
+
+.. code-block:: yaml
+
+   handle: TESTR-001
+
+.. _yaml-title:
+
+title
+=====
+
+The title of the report series:
+
+.. code-block:: yaml
+
+   title: Characterization Metric Report Demo
+
+.. _yaml-ipynb:
+
+ipynb
+=====
+
+Set this field to the name of the report's Jupyter Notebook file, including the ``.ipynb`` extension:
+
+.. code-block:: yaml
+
+   ipynb: TESTR-001.ipynb
+
+.. _yaml-assets:
+
+assets (optional)
+=================
+
+If the report's Jupyter Notebook relies on other files to execute, and those files are part of the notebook template repository, you can mark them as *assets* with the ``assets`` field.
+Assets are files, other than the ``ipynb`` notebook file, that are copied from the report repository into the report instance when it is rendered and computed (see `nbreport init`_).
+Unlike the notebook (``ipynb`` file), **assets are not templated** — they are copied as-is.
+Directory structure is preserved when asset files are copied from the report repository to the instance.
+
+The assets field takes a **list** of strings.
+Each string is a rule for matching asset file paths:
+
+.. code-block:: yaml
+
+   assets:
+     - '**/*.fits'
+     - 'mydata.csv'
+
+There are three types of rules that designate assets:
+
+- **The file name.**
+
+  Simply name the file (relative to the root of the report repository) and it will be designated as an asset:
+
+  .. code-block:: yaml
+
+     assets:
+       - 'mydata.csv'
+       - 'images/image.fits`
+
+  This example names two files, :file:`mydata.csv` and :file:`images/images.fits`, as assets.
+
+- **A directory name.**
+
+  The directory, and any files and subdirectories contained inside it, are copied from the report repository to the instance:
+
+  .. code-block:: yaml
+
+     assets:
+       - 'images'
+
+  In this example, the entire :file:`images` directory is designated as an asset.
+
+- **A glob pattern.**
+
+  In addition to simple globs (such as ``*.fits``), recursive globs are also supported (``**/*.fits``):
+
+  .. code-block:: yaml
+
+     assets:
+       - 'images/*.fits'
+       - '**/*.csv'
+
+  In this example, any FITS file in the :file:`images` directory is designated as an asset.
+  The second rule designates all CSV files as assets, regardless of what subdirectory contains them.
+
+.. seealso::
+
+   :doc:`how-to-use-python-modules`
+
+.. _yaml-git-repo:
+
+git\_repo (optional)
+====================
+
+The URL of the Git repository that this report template is published to.
+This field is not necessary for local demos, though it is expected by the `nbreport register`_ command.
+Setting this field also helps nbreport include information about the source Git repository in published report instances:
+
+.. code-block:: yaml
+
+   git_repo: https://github.com/lsst-sqre/nbreport
+
+.. _yaml-git-repo-subdir:
+
+git\_repo\_subdir (optional)
+============================
+
+If the report template is part of a Git repository (:ref:`git\_repo <yaml-git-repo>` is set), but the repository *is not* located at the root of that Git repository, you can specify the subdirectory where the report template is located by setting the ``git_repo_subdir`` field:
+
+.. code-block:: yaml
+
+   git_repo_subdir: tests/TESTR-001
+
+In this example, the report repository is located in the ``tests/TESTR-001`` directory of the ``https://github.com/lsst-sqre/nbreport`` Git repository:
+
+If the report template occupies the root of the Git repository, this field should be omitted:
+
+.. _yaml-ltd-product:
+
+ltd\_product (optional)
+=======================
+
+This is the name of the report's registered `product name`__ in the *LSST the Docs* RESTful HTTP API (see also :ref:`ltd\_url <yaml-ltd-url>`):
+
+.. code-block:: yaml
+
+   published_url: https://testr-001.lsst.io
+
+.. __: https://ltd-keeper.lsst.io/products.html
+
+Normally this field is automatically added when you run the `nbreport register`_ command to register the report with the nbreport server.
+If the report is not formally published, this field should not be set.
+
+.. _yaml-published-url:
+
+published\_url (optional)
+=========================
+
+The ``published_url`` field is the URL for the homepage of a published report.
+The homepage indexes all available instances of a report series:
+
+.. code-block:: yaml
+
+   published_url: https://testr-001.lsst.io
+
+Normally this field is automatically added when you run the `nbreport register`_ command to register the report with the nbreport server.
+If the report is not formally published, this field should not be set.
+
+.. _yaml-ltd-url:
+
+ltd\_url (optional)
+===================
+
+The ``ltd_url`` field is the URL for the report in the *LSST the Docs* RESTful HTTP API.
+*LSST the Docs* is the service that hosts LSST documentation, including notebook-based reports:
+
+.. code-block:: yaml
+
+   ltd_url: https://keeper.lsst.codes/products/testr-001
+
+Normally this field is automatically added when you run the `nbreport register`_ command to register the report with the nbreport server.
+If the report is not formally published, this field should not be set.
+
+.. seealso::
+
+   The `LTD Keeper documentation`__ describes this API.
+
+.. __: https://ltd-keeper.lsst.io/products.html#get--products-(slug)
+
+.. _nbreport register: ../cli-reference.html#nbreport-register
+.. _nbreport init: ../cli-reference.html#nbreport-init