Skip to content

Commit

Permalink
Add read the docs documentation (#11)
Browse files Browse the repository at this point in the history
Signed-off-by: David Brown <dmlb2000@gmail.com>
  • Loading branch information
dmlb2000 committed Dec 17, 2018
1 parent d7901ed commit 3c38e03
Show file tree
Hide file tree
Showing 20 changed files with 492 additions and 84 deletions.
74 changes: 2 additions & 72 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,76 +9,6 @@ Pacifica Python Library for Downloading Data
This repository provides a library so python applications can
download data from Pacifica.

## Downloader API

The entrypoint for this library is the `Downloader` class in
the `downloader` module. Instances of this class are created
with a download directory and a
[Pacifica Cartd](https://github.com/pacifica/pacifica-cartd)
endpoint url.

### Downloader Class

The [constructor](pacifica/downloader/downloader.py#16) takes
two arguments `location` and `cart_api_url`. The `location` is
a download directory to be created by a download method. The
`cart_api_url` is the endpoint for creating carts.

The other methods in the `Downloader` class are the supported
download methods. Each method takes appropriate input for that
method and the method will download the data to the location
defined in the constructor.

[CloudEvents](https://github.com/cloudevents/spec) is a
standard for how to send messages between services in cloud
environments. The `cloudevent()` method
([here](pacifica/downloader/downloader.py#45))
consumes the event emitted by the
[Pacifica Notifications](https://github.com/pacifica/pacifica-notifications)
service and downloads the data.

## Internal Classes and Methods

The internal classes help organize the work around the cart API.

### Cart API

The [CartAPI](pacifica/downloader/cartapi.py#11) class has two
methods used for setting up a cart and waiting for completion.

The `setup_cart()` method takes a callable argument that returns
an iterator. The iterator returns a list that is directly sent to
the [Cartd API](https://github.com/pacifica/pacifica-cartd). The
`setup_cart()` method returns the full url to the cart created.

The `wait_for_cart()` method takes a cart url returned from the
`setup_cart()` method and polls the endpoint until the cart is
ready to download.

### CloudEvent

The `CloudEvent` class ([here](pacifica/downloader/cloudevent.py#7))
contains the `cloudevent()` method. It
requires the cloud event as an argument. The `cloudevent()`
generates a method that yields the cart file objects from the
cloud event.

## Examples

### CloudEvents Example

This is a basic stub code to download data from a cloud event.
This assumes the cloudevent is saved to a file for processing.
There are a number of ways to get the cloud event this is just
a minimum representation of whats required.

```
from json import loads
from tempfile import mkdtemp
from pacifica.downloader import Downloader
cloud_event = loads(open('cloudevent.json').read())
down_path = mkdtemp()
down = Downloader(down_path, 'http://metadata.example.com')
down.cloudevent(cloud_event)
```
See the [Read the Docs](https://pacifica-downloader.readthedocs.io)
documentation for more information
19 changes: 19 additions & 0 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SOURCEDIR = .
BUILDDIR = _build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
Empty file added docs/_static/.gitkeep
Empty file.
190 changes: 190 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
#!/usr/bin/python
# -*- coding: utf-8 -*-
"""This is an autogenerated sphinx config module."""
# pylint: disable=invalid-name
# pylint: disable=redefined-builtin
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('..'))
from recommonmark.parser import CommonMarkParser

# -- Project information -----------------------------------------------------

project = 'Pacifica Python Downloader'
copyright = '2018, David Brown'
author = 'David Brown'

# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = ''


# -- General configuration ---------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.viewcode',
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
source_parsers = {
'.md': CommonMarkParser,
}
source_suffix = ['.rst', '.md']

# The master toctree document.
master_doc = 'index'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'

# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}


# -- Options for HTMLHelp output ---------------------------------------------

# Output file base name for HTML help builder.
htmlhelp_basename = 'PacificaDownloaderdoc'


# -- Options for LaTeX output ------------------------------------------------

latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',

# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',

# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',

# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}

# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'PacificaDownloader.tex', 'Pacifica Python Downloader Documentation',
'David Brown', 'manual'),
]


# -- Options for manual page output ------------------------------------------

# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'pacificadownloader', 'Pacifica Python Downloader Documentation',
[author], 1)
]


# -- Options for Texinfo output ----------------------------------------------

# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'PacificaDownloader', 'Pacifica Python Downloader Documentation',
author, 'PacificaDownloader', 'One line description of project.',
'Miscellaneous'),
]


# -- Options for Epub output -------------------------------------------------

# Bibliographic Dublin Core info.
epub_title = project

# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''

# A unique identification for the text.
#
# epub_uid = ''

# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']


# -- Extension configuration -------------------------------------------------
autodoc_default_options = {
'member-order': 'bysource',
'special-members': '__init__',
'undoc-members': None,
'exclude-members': '__weakref__'
}
7 changes: 7 additions & 0 deletions docs/downloader.cartapi.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
CartAPI Python Module
=============================================

.. automodule:: pacifica.downloader.cartapi
:members:
:private-members:
:special-members:
7 changes: 7 additions & 0 deletions docs/downloader.cloudevent.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
CloudEvent Python Module
=============================================

.. automodule:: pacifica.downloader.cloudevent
:members:
:private-members:
:special-members:
7 changes: 7 additions & 0 deletions docs/downloader.downloader.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Downloader Python Module
=============================================

.. automodule:: pacifica.downloader.downloader
:members:
:private-members:
:special-members:
7 changes: 7 additions & 0 deletions docs/downloader.policy.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Policy Python Module
=============================================

.. automodule:: pacifica.downloader.policy
:members:
:private-members:
:special-members:
16 changes: 16 additions & 0 deletions docs/downloader.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Downloader Python Module
=============================================

.. toctree::
:maxdepth: 2
:caption: Contents:

downloader.downloader
downloader.cartapi
downloader.cloudevent
downloader.policy

.. automodule:: pacifica.downloader
:members:
:private-members:
:special-members:
52 changes: 52 additions & 0 deletions docs/exampleusage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Code Examples

The Python Downloader has two architectural pieces. The first are a
set of methods to manipulate metadata about files and generate a
Cartd friendly metadata document. The second is a set of methods to
interact with the Cartd service to wait and download the files.

## Download Transaction Info

The download setup described below creates a temporary directory to
download the data to.

```python
from tempfile import mkdtemp

down_path = mkdtemp()
down = Downloader(down_path, 'http://127.0.0.1:8081')
resp = requests.get('http://127.0.0.1:8181/status/transactions/by_id/67')
assert resp.status_code == 200
down.transactioninfo(resp.json())
```

## Download Cloud Event

Often CloudEvents are handled in web server frameworks. Here's an
example of using the downloader in [CherryPy](https://cherrypy.org/).
This example can be launched as a consumer of CloudEvents sent by
the [Pacifica Notifications](https://pacifica-notifications.readthedocs.io)
service.

```python
from tempfile import mkdtemp
import cherrypy

class Root(object):
exposed = True
@cherrypy.tools.json_in()
@cherrypy.tools.json_out()
def POST(self):
"""Accept the cloud event data and return the local download path."""
down_path = mkdtemp()
down = Downloader(down_path, 'http://127.0.0.1:8081')
down.cloudevent(cherrpy.request.json)
return { 'download_path': down_path }


cherrypy.quickstart(Root(), '/', {
'/': {
'request.dispatch': cherrypy.dispatch.MethodDispatcher()
}
})
```

0 comments on commit 3c38e03

Please sign in to comment.