docs: minor updates from doing a full readdown of all docs.

dbprocessing/inspector.py

@@ -18,10 +18,12 @@
    * The class must implement the :meth:`~inspector.inspect` method (see that
      documentation for details).
 
-See also :doc:`../inspector_examples`.
-
 A :sql:table:`inspector` record must also then be created referencing the
 inspector file and with any necessary keywords.
+
+See Also
+--------
+:doc:`../inspector_examples`
 """
 from __future__ import absolute_import
 from __future__ import print_function

docs/concepts.rst

@@ -259,6 +259,24 @@ link deleted. The file pointed to by the link should already be in its
 final location according to its product: the file is not moved if it is
 in the "wrong" location, and this can cause problems finding it later!
 
+Implementation
+--------------
+:meth:`~.ProcessQueue.checkIncoming` checks for all files in the incoming
+directory and adds their names to a queue of files to ingest, removing
+any duplicate files.
+
+:meth:`~.ProcessQueue.importFromIncoming` iterates over these filenames.
+For each, checks if it is already in the database (:meth:`.getFileID`).
+If not, calls :meth:`~.ProcessQueue.figureProduct`, which runs each
+:ref:`inspector <concepts_inspectors>` to determine the product. If
+there is a match, :meth:`~.ProcessQueue.figureProduct`:
+
+   1. uses :meth:`~.ProcessQueue.diskfileToDB` to take the :class:`.Diskfile`
+      populated by the inspector and create the :sql:table:`file` record,
+   2. moves the file to the appropriate final place based on the product,
+   3. and adds the file to the :ref:`process queue <concepts_process_queue>`
+      (:meth:`~.ProcessqueuePush`) for further processing.
+
 .. _concepts_process_queue:
 
 Process Queue
@@ -312,6 +330,22 @@ the console output from that code is placed in the
 file if it has been created (this may, of course, be only a partial file,
 given the error).
 
+Implementation
+--------------
+For each file on the :ref:`process queue <concepts_process_queue>`, calls
+:meth:`~.ProcessQueue.buildChildren`, which calculates all possible output
+products and makes a :class:`~.runMe.runMe` object for every possible command
+to run.
+
+Once these are created (and the process queue empty), all :class:`~.runMe.runMe` objects are passed to :func:`~.runMe.runner` at once. :func:`~.runMe.runner`
+calculates the command line for every object, then begins starting processes
+to actually run the data processing commands.
+
+Processes are started up to the maximum count, and polled for completion.
+Outputs of successful runs are moved to incoming and then ingested; failures
+are handled as described above. New processes are then started back to the
+maximum.
+
 .. _concepts_missions:
 
 Missions

docs/conf.py

@@ -336,6 +336,7 @@
     'python': ('https://docs.python.org/', None),
     'sphinx': ('https://www.sphinx-doc.org/en/master/', None),
     'sqlalchemy': ('https://docs.sqlalchemy.org/en/stable/', None),
+    'numpy': ('https://numpy.org/doc/stable/', None),
 }
 
 import sphinx.ext.autosummary.generate

docs/developer/code_standards.rst

@@ -119,7 +119,9 @@ first-line summary. See also :doc:`pull_requests`.
 Lines must not exceed 76 characters (``git log`` adds a
 four-space indent to the commit message on display).
 
-An example::
+An example:
+
+.. code-block:: none
 
     Add code of conduct
 
@@ -131,18 +133,18 @@ An example::
 
 Testing
 =======
-All code should be tested in the ``unittest``-based test suite of
+All code should be tested in the :mod:`unittest`-based test suite
 in the ``unit_tests`` directory.
 
 Testing should cover a reasonable fraction of the lines of code: at
 least 80%, with 90+% preferred. They should also cover a reasonable
 range of possible inputs and input types.
 
-Unit tests using the ``assertEquals`` or similar methods should
-specify the expected value as the first argument and the actual output
+Unit tests using the :meth:`~unittest.TestCase.assertEqual` or similar methods
+should specify the expected value as the first argument and the actual output
 from the tested method as the second; this makes the displayed diff
-rational.  ``assert_array_equal`` and other ``numpy`` based methods
-should be the other way around (actual, then expected) to make the
+rational.  :func:`~numpy.testing.assert_array_equal` and other :mod:`numpy`
+based methods should be the other way around (actual, then expected) to make the
 diff easy to read.
 
 Checklist

docs/developer/documentation.rst

@@ -2,15 +2,24 @@
 Documentation
 *************
 
-`Sphinx <http://www.sphinx-doc.org/>`_ documentation is stored in the ``sphinx`` directory.
+`Sphinx <http://www.sphinx-doc.org/>`_ documentation sources are stored in the
+``docs`` directory; the build system and output are in the ``sphinx`` directory.
 
 ``Makefile`` is used to build the documentation.
 
 Run ``make html`` in the ``sphinx`` directory to build the html documentation, output is in ``build/html``.
 
-``build`` contains the built documentation (and intermediate files); it can safely be deleted.
+``sphinx/build`` contains the built documentation (and intermediate files);
+it can safely be deleted.
 
-``source`` contains the ReStructuredText source files; note that a large quantity of the documentation is not built from here but from the Python source files. ``source/autosummary`` contains the docs extracted from those source files; it can safely be deleted. If anything's weird about the docs generated from the Python source, try deleting ``autosummary`` first and then rebuilding. The ``autosummary`` docs are extracted from the version of the module in top-level ``build``, i.e., run ``python setup.py build`` before generating the documentation.
+``docs`` contains the ReStructuredText source files; note that a large
+quantity of the documentation is not built from here but from the Python
+source files. ``docs/developer/autosummary`` contains the docs extracted
+from those source files; it can safely be deleted. If anything's weird
+about the docs generated from the Python source, try deleting ``autosummary``
+first and then rebuilding. The ``autosummary`` docs are extracted from the
+version of the module in top-level ``build``, i.e., run
+``python setup.py build`` before generating the documentation.
 
 .. contents::
    :local:
@@ -34,8 +43,7 @@ outputs. Cross-referencing of documentation should be generous.
 
 If a directive spans multiple lines, proper indentation is essential
 for readability and Sphinx parsing. Subsequent lines must align with
-the start of the description of the parameter, i.e. the second column
-after the closing ``:`` of the directive.
+the start of the description of the parameter.
 
 Documentation must build in Sphinx without warnings and the output
 be checked for proper formatting.
@@ -51,11 +59,6 @@ they can be found later. Consider opening an issue instead.  Raising a
 commenting-out code rather than deleting it, except for short-term
 testing; it can always be retrieved from version control later.
 
-More information will be provided here regarding the use of
-autodoc/autosummary, particularly its interaction with numpydoc and
-the fully-automatic build of the API documentation, but that is
-pending a full setup of that documentation structure.
-
 Sphinx rst standards
 ====================
 For static Sphinx documentation (i.e. ``.rst`` files), keep in mind the
@@ -83,9 +86,6 @@ relevant.) Use :mod:`~sphinx.ext.intersphinx` to link to other projects that
 use Sphinx for documentation. ``make linkcheck`` in the ``sphinx`` directory
 will verify links (both intersphinx and HTTP).
 
-More information will be provided here on the proper separation
-of API documentation from user/design documentation.
-
 The documentation is not, at this time, completely up to these standards
 (or completely consistent.)
 

docs/developer/index.rst

@@ -6,16 +6,13 @@ Developer Information
    :maxdepth: 1
 
    api
-   build
    tests
    code_standards
    documentation
    github
    pull_requests
    repository
    release
-   notes
    tables
-   todo
    inspector_examples
    ../CONTRIBUTING

docs/developer/inspector_examples.rst

@@ -4,7 +4,7 @@ Inspector examples
 
 These are some examples of :ref:`concepts_inspectors`.
 
-Example 2
+Example 1
 =========
 
 .. code-block:: python
@@ -43,7 +43,7 @@ Example 2
         self.diskfile.params['version'] = inspector.extract_Version(self.basename)
         return "That is not my dog." # anything that is not None is good
 
-Example 3
+Example 2
 =========
 .. code-block:: python
 

docs/developer/pull_requests.rst

@@ -71,7 +71,9 @@ in which case ``closes`` is preferred.
 The template includes a checklist;
 consider every item on the list and check it if completed. If an item
 is not relevant, check it, add "(N/A)" to the start of the line, and
-include an explanation below the checklist. E.g.::
+include an explanation below the checklist. E.g.:
+
+.. code-block:: none
 
    - [X] ...
    - [X] (N/A) Major new functionality has appropriate Sphinx documentation

docs/developer/repository.rst

@@ -4,7 +4,8 @@ Repository organization
 
 top-level
 =========
-There should be(but there totally are) no committed files at the top level besides ``setup.py``, and ``.gitignore`` (not normally visible).
+There should be (but currently are) no committed files at the top level
+besides ``setup.py``, and ``.gitignore`` (not normally visible).
 
 build
 =====
@@ -14,43 +15,37 @@ dbprocessing
 ============
 Source tree for the main ``dbprocessing`` module. If it's in this directory, it's meant to be installed.
 
-Documents
+developer
 =========
-?
+Miscellaneous bits for sharing between developers. Included in the source
+tarball but not in binary distributions.
+
+docs
+====
+Source for :doc:`documentation`.
+
+examples
+========
+Examples of how to set up dbprocessing for a project, including
+configuration and scripts that are specific to missions.
+
+functional_test
+===============
+Full dbprocessing setup for end-to-end functional test (:ref:`functional`).
 
 gui
 ===
-?
-
-OneOffs
-=======
-?
+Initial work on a Qt-based GUI; not complete.
 
 scripts
 =======
 Scripts meant to be called from the command line that should be installed with the module. They should be added to the ``scripts`` list in ``setup.py`` and documented in :doc:`../scripts` (source file ``scripts.rst``).
 
 sphinx
 ======
-See :doc:`documentation`.
+Built :doc:`documentation`.
 
-test_codes
-==========
-?
 
-testDB
-======
-See :ref:`functional`.
-
-Testing_Utils
-=============
-?
-
-tests
-=====
+unit_tests
+==========
 See :ref:`unit`.
-
-tests_scripts
-=============
-?
-