add more documentation for v0.2.7 (#66)

* bump version to v0.2.7 Brown Bicycle Bears * remove history from readme and just link to github releases page * fix pep8 errors from setup.py * update documentation * add announcements * add new style pvpower demo model to examples * update pvpower tests with new and old models * add announcement regarding changes to interval and sim_length in simulations class * * update quickstart docs section in getting started * models test wasn't testing model #1 which specifies parameter path as class attributes * fix typos in docstring for quickstart * add parameter specification methods section to docs * in getting started explain why there are 2 different methods for specifying parameters * get rid of the batch example in tests that was never used * add docstrings to sim test to explain that it tests specifying the sim param file as class attrs * rewrite tutorial 1 using new style parameter specifications * fix italic format typos, not markdown * add ordered list to parameters section on model styles * fix cross reference name error in tutorial 5 * reorg outputs tutorial to put class attrs spec style first * add Outputs to tutorial title and Outputs Class section after intro * add Outputs Registry section with link to last tutorial and promise for further discussion * change material property example from soiling to degradation, and give power as non-material property. * add model test for no model parameters and check state is "uninitialized" * add data, formulas, scripts, models, layers, readers and scripts pages to docs * use argparse in carousel-quickstart script instead of checking sys.argv * fix indent for example code in data_readers.py JSONReader class * add links to model.py module docstring for model->models plural and Model class, also make sentence less wordy * add models and layers modules to core.rst, remove extra spaces and promote Core header up a level * don't autodoc UREG, instead use explicit markup and make a link to Pint documentation * add Carousel JSON Encoder to core.rst too * fix there was no data-sources or data-readers labelsin the docs, but there were references to them, huh? * there were references to non-existing formulas lable too, true! * don't autodoc carousel-quickstart since it can't be imported easily since it uses a hyphen, instead use explicit markup and move the docstring to scripts.rst since the argument parser provides help instead * finish rewrite of tutorial 1: Outputs - finish alternatel parameter file style and examples - add table of calculated output attributes - include timeseries in first attr table - add explaination re: timeseries to constant flag section * add more detail re: quickstart in getting started about the PROJ_PATH constant in the project package __init__.py file and links to API * start rewriting of calculation tutorial * also update getting started references to PVLIB * also explain that example is in repo and sdist * finish improving tutorial 2 * also remove the TODO in FormulaBase about converting methods in the formulas class to either static methods or whatever. This question was closed in issue # 65 * fix wordy sentence and typos in getting started * show full path to Output class in tutorial 1, remove extra spaces from fancy format table * change "resize outptus for output" to "outputs for writing files" * more typos everywhere * add "Calculations" to tutorial 2 title * add calculation attributes section and change `always_calc` to ignores thresholds * move parameter file style to end instead of v.v. Signed-off-by: Mark Mikofski <mark.mikofski@sunpowercorp.com> * start on tutorial 3 * make tests for numexpr importer * also fix constants not used in DataBase meta class for data_reader and data_cache flags * fix bugs in NumericalExpressionImporter - formulas were not returned! return the formulas! - transpose retval so uncertainty wrapper knows how many observations there are! * pop formula_importer from attr in FormulaBase so it can be overwritten in subclass * start section on formula attributes in tutorial 3, and add :: markup for code examples * new test also checks that uncertainty and jacobian or correct, even for multiple observations * add detail section on units and uncertainty * add attributes section to tutorial-3 * fix don't need trailing None for input data args, oops! * reshape output for formulas in test_sim * recursively load packages before loading model if not on path * always check if formulas are already imported or on python path * add details to formulas tutorial about specifying module, package and path * add tutorials for data, models and simulations * finish documentation quickly (so it's rough) * complete data source attributes section, including data reader and enable cache * add data reader section * add simulation class example, attributes section * add model class example and demo of running model simulation * demo how to retrieve registry data * add contrib to api so links to new readers work
BreakingBytes · Oct 11, 2016 · 674557e · 674557e
1 parent 2216619
commit 674557e
Show file tree

Hide file tree

Showing 27 changed files with 2,014 additions and 370 deletions.
diff --git a/README.rst b/README.rst
@@ -53,7 +53,7 @@ online. It's also included in the distribution and can be built by running the
 ``Makefile`` found in the ``docs`` folder of the Carousel package.
 Documentation uses Sphinx, and built documentation will be found in the
 ``_build`` folder under the tree corresponding to the type of documentation
-built. _EG_: HTML documentation is in ``docs/_build/html``.
+built. *EG*: HTML documentation is in ``docs/_build/html``.
 
 Contributions
 -------------
@@ -65,32 +65,6 @@ create pull requests. Discuss the roadmap or download presentations on the
 
 History
 -------
-This is the change log.
-
-v0.2.2 2016-07-21
-~~~~~~~~~~~~~~~~~
-`Balancing Act <https://github.com/SunPower/Carousel/releases/tag/v0.2.2>`_:
-
-* fixes #38 let data sources accept any kind of argument, not just filename
-* fixes #39 allow data caching to be disabled
-* fixes #40 use Meta class for options
-* fixes #41 pass data to ``load_data()`` not reader
-
-v0.2.1 2016-07-19
-~~~~~~~~~~~~~~~~~
-`Bumper Cars <https://github.com/SunPower/Carousel/releases/tag/v0.2.1>`_:
-
-* fixes #37 pop ``data_reader`` from data source
-
-v0.2 2016-05-03
-~~~~~~~~~~~~~~~
-`Big Top <https://github.com/SunPower/Carousel/releases/tag/v0.2>`_:
-
-* documentation and tutorial with PV Power demo
-* rename package Carousel
-
-v0.1 2016-02-11
-~~~~~~~~~~~~~~~
-`Acrobats <https://github.com/SunPower/Carousel/releases/tag/v0.1>`_:
-
-* quickstart script
+The
+`change log for all releases <https://github.com/SunPower/Carousel/releases>`_
+is on GitHub.
diff --git a/carousel-quickstart.py b/carousel-quickstart.py
@@ -2,35 +2,15 @@
 # -*- coding: utf-8 -*-
 
 """
-Creates a basic file structure to start a Carousel project.
-
-    Project
-    |
-    +-+- project
-    | |
-    | +- __init__.py
-    |
-    +- models
-    | |
-    | +- default.json
-    |
-    +- simulation
-    |
-    +- outputs
-    |
-    +- calculations
-    |
-    +- formulas
-    |
-    +- data
+carousel-quickstart.py command-line script - more info in docs/api/scripts
 """
 
-# import argparse
+import argparse
 import json
 import logging
 import os
 import re
-# import shutil
+from carousel import __version__
 import sys
 
 # set up logging
@@ -45,13 +25,31 @@
 DFLT = 'my_model.json'
 MODELS = 'models'
 PATHS = [MODELS] + LAYERS
+INIT_CONTENT = """
+import os
+
+__version__ = '0.1'
+__author__ = 'your name'
+__email__ = 'your.name@company.com'
+
+PKG_PATH = os.path.abspath(os.path.dirname(__file__))
+PROJ_PATH = os.path.dirname(PKG_PATH)
+"""
+DESCRIPTION = """
+Creates a Carousel project.  See documentation for more information.
+"""
 
 # run from command line
 if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=DESCRIPTION)
+    parser.add_argument('project', help='name of Carousel project to create')
+    parser.add_argument('--version', action='version',
+                        version=('%(prog)s' + ' %s' % __version__))
+    args = parser.parse_args()
     # exit with error if no project name specified
     if len(sys.argv) < 2:
         sys.exit('No project directory was specified.')
-    project_name = sys.argv[1]  # get project name
+    project_name = args.project  # get project name
     # check if name is alpha-numeric, underscore okay
     match = re.findall(OKAY, project_name)  # find all alpha-numeric matches
     clean = ''.join(match)  # clean alpha-numeric project name
@@ -73,6 +71,7 @@
     pkg_init = os.path.join(project_name, project_pkg, '__init__.py')
     with open(pkg_init, 'w') as init:
         init.write('"""\nThis is the %s package.\n"""\n' % project_pkg)
+        init.write(INIT_CONTENT)
     # make default model
     with open(os.path.join(project_name, MODELS, DFLT), 'w') as dflt:
         json.dump(DFLT_MODEL, dflt, indent=2)

diff --git a/carousel/__init__.py b/carousel/__init__.py
@@ -7,5 +7,5 @@
 __author__ = u'Mark Mikofski'
 __email__ = u'mark.mikofski@sunpowercorp.com'
 __url__ = u'https://github.com/SunPower/Carousel'
-__version__ = u'0.2.6'
-__release__ = u'Bicycle Bears'
+__version__ = u'0.2.7'
+__release__ = u'Brown Bicycle Bears'
diff --git a/carousel/contrib/readers.py b/carousel/contrib/readers.py
@@ -39,12 +39,12 @@ class ArgumentReader(DataReader):
     which consist of the names and attributes of the positional and keyword
     arguments respectively. For example::
 
-    {
-        'GHI': {'units': 'W/m**2', 'isconstant': False, 'argpos': 0},
-        'azimuth': {'units': 'degrees', 'isconstant': False, 'argpos': 1},
-        'DNI': {'units': 'W/m**2', 'isconstant': False},
-        'zenith': {'units': 'degrees', 'isconstant': False}
-    }
+        {
+            'GHI': {'units': 'W/m**2', 'isconstant': False, 'argpos': 0},
+            'azimuth': {'units': 'degrees', 'isconstant': False, 'argpos': 1},
+            'DNI': {'units': 'W/m**2', 'isconstant': False},
+            'zenith': {'units': 'degrees', 'isconstant': False}
+        }
 
     """
     #: True if reader accepts ``filename`` argument

diff --git a/carousel/core/data_readers.py b/carousel/core/data_readers.py
@@ -80,44 +80,44 @@ class JSONReader(DataReader):
 
     For example::
 
-    {
-        "data": {
-            "DNI": [834, 523, 334, 34, 0, 0],
-            "zenith": [21, 28, 45, 79, 90, 90]
-        },
-        "param_file": "path/to/corresponding/param_file.json",
-        "data_source": "MyDataSource"
-    }
+        {
+            "data": {
+                "DNI": [834, 523, 334, 34, 0, 0],
+                "zenith": [21, 28, 45, 79, 90, 90]
+            },
+            "param_file": "path/to/corresponding/param_file.json",
+            "data_source": "MyDataSource"
+        }
 
     Parameters can be specified in a JSON file. ::
 
-    {
-        "DNI": {
-            "description": "direct normal insolation",
-            "units": "W/m*^2",
-            "isconstant": false
-        },
-        "zenith": {
-            "description": "solar zenith",
-            "units": "degrees",
-            "isconstant": false
+        {
+            "DNI": {
+                "description": "direct normal insolation",
+                "units": "W/m*^2",
+                "isconstant": false
+            },
+            "zenith": {
+                "description": "solar zenith",
+                "units": "degrees",
+                "isconstant": false
+            }
         }
-    }
 
     Parameters can also be specified in the data source as class attributes. ::
 
-    class MyDataSrc(DataSource):
-        data_reader = JSONReader
-        DNI = {
-            "description": "direct normal insolation",
-            "units": "W/m*^2",
-            "isconstant": false
-        }
-        zenith = {
-            "description": "solar zenith",
-            "units": "degrees",
-            "isconstant": false
-        }
+        class MyDataSrc(DataSource):
+            data_reader = JSONReader
+            DNI = {
+                "description": "direct normal insolation",
+                "units": "W/m*^2",
+                "isconstant": false
+            }
+            zenith = {
+                "description": "solar zenith",
+                "units": "degrees",
+                "isconstant": false
+            }
 
     """
     def __init__(self, parameters, data_reader=None):

diff --git a/carousel/core/data_sources.py b/carousel/core/data_sources.py
@@ -150,9 +150,9 @@ def __new__(mcs, name, bases, attr):
         attr = mcs.set_param_file_or_parameters(attr)
         # set data-reader attribute if in subclass, otherwise read it from base
         if reader is not None:
-            attr['data_reader'] = reader
+            attr[mcs._reader_attr] = reader
         if cache_enabled is not None:
-            attr['data_cache_enabled'] = cache_enabled
+            attr[mcs._enable_cache_attr] = cache_enabled
         if meta is not None:
             attr['_meta'] = meta
         return super(DataSourceBase, mcs).__new__(mcs, name, bases, attr)

diff --git a/carousel/core/formulas.py b/carousel/core/formulas.py
@@ -83,35 +83,55 @@ def import_formulas(self):
         package = self.parameters.get('package')  # package read from params
         name = package + module if package else module  # concat pkg + name
         path = self.parameters.get('path')  # path read from parameters
-        # import module using module & package keys in parameter file
+        # import module using module and package
+        mod = None
         # SEE ALSO: http://docs.python.org/2/library/imp.html#examples
-        if not path:
+        try:
+            # fast path: see if module was already imported
+            mod = sys.modules[name]
+        except KeyError:
             try:
-                # fast path: see if module was already imported
-                mod = sys.modules[name]
-            except KeyError:
                 # import module specified in parameters
                 mod = importlib.import_module(module, package)
-        else:
-            # expand ~, environmental variables and make it absolute path
-            if not os.path.isabs(path):
-                path = os.path.expanduser(os.path.expandvars(path))
-                path = os.path.abspath(path)
-            # paths must be a list
-            paths = [path]
-            # import module and path from parameters file.
-            # FYI: don't combine statements in try blocks, otherwise you won't
-            # know what raised the exception!
-            # FYI: imp.load_source() is more suited to loading a module as
-            # something other than its filename into sys.modules dictionary.
-            # Find the module by name and path, return open file, pathname, &c.
-            fp, filename, description = imp.find_module(name, paths)
-            # try to load the module (reloads if already loaded)
-            try:
-                mod = imp.load_module(name, fp, filename, description)
-            finally:
-                if fp:
-                    fp.close()
+            except ImportError as err:
+                if not path:
+                    msg = ('%s could not be imported either because it was not '
+                           'on the PYTHONPATH or path was not given.')
+                    LOGGER.exception(msg, name)
+                    raise err
+                else:
+                    # import module using path
+                    # expand ~, environmental variables and make path absolute
+                    if not os.path.isabs(path):
+                        path = os.path.expanduser(os.path.expandvars(path))
+                        path = os.path.abspath(path)
+                    # paths must be a list
+                    paths = [path]
+                    # imp does not find hierarchical module names, find and load
+                    # packages recursively, then load module, see last paragraph
+                    # https://docs.python.org/2/library/imp.html#imp.find_module
+                    pname = ''  # full dotted name of package to load
+                    # traverse namespace
+                    while name:
+                        # if dot in name get first package
+                        if '.' in name:
+                            pkg, name = name.split('.', 1)
+                        else:
+                            pkg, name = name, None  # pkg is the module
+                        # Find package or module by name and path
+                        fp, filename, desc = imp.find_module(pkg, paths)
+                        # full dotted name of package to load
+                        pname = pkg if not pname else '%s.%s' % (pname, pkg)
+                        LOGGER.debug('package name: %s', pname)
+                        # try to load the package or module
+                        try:
+                            mod = imp.load_module(pname, fp, filename, desc)
+                        finally:
+                            if fp:
+                                fp.close()
+                        # append package paths for imp.find_module
+                        if name:
+                            paths = mod.__path__
         formulas = {}  # an empty list of formulas
         formula_param = self.parameters.get('formulas')  # formulas key
         # FYI: iterating over dictionary is equivalent to iterkeys()
@@ -145,7 +165,9 @@ def import_formulas(self):
         for f, p in formula_param.iteritems():
             formulas[f] = lambda *args: ne.evaluate(
                 p['expression'], {k: a for k, a in zip(p['args'], args)}, {}
-            )
+            ).reshape(1, -1)
+            LOGGER.debug('formulas %s = %r', f, formulas[f])
+        return formulas
 
 
 class FormulaBase(CommonBase):
@@ -154,18 +176,20 @@ class FormulaBase(CommonBase):
     """
     _path_attr = 'formulas_path'
     _file_attr = 'formulas_file'
+    _importer_attr = 'formula_importer'
 
     def __new__(mcs, name, bases, attr):
         # use only with Formula subclasses
         if not CommonBase.get_parents(bases, FormulaBase):
             return super(FormulaBase, mcs).__new__(mcs, name, bases, attr)
-        # TODO: convert any methods starting with f_ to static methods
-        # for a, v in attr.iteritems():
-        #     if a.startswith('f_'):
-        #         attr[a] = staticmethod(v)
+        # pop the data reader so it can be overwritten
+        importer = attr.pop(mcs._importer_attr, None)
         # set param file full path if formulas path and file specified or
         # try to set parameters from class attributes except private/magic
         attr = mcs.set_param_file_or_parameters(attr)
+        # set data-reader attribute if in subclass, otherwise read it from base
+        if importer is not None:
+            attr[mcs._importer_attr] = importer
         return super(FormulaBase, mcs).__new__(mcs, name, bases, attr)
 
 

diff --git a/carousel/core/models.py b/carousel/core/models.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 """
-This is the Carousel model module. The model module contains the definition
-for the Model class. In general a Model contains Layers.
+This is the Carousel :mod:`~carousel.core.models` module that contains
+definitions for the :class:`~carousel.core.models.Model` class.
 
 The Carousel model contains five layers:
 :class:`~carousel.core.layers.Data`,

diff --git a/carousel/docs/api/contrib.rst b/carousel/docs/api/contrib.rst
@@ -0,0 +1,26 @@
+.. _contrib:
+
+Contributions
+=============
+This sub-package of carousel contains contributions that are considered
+important parts of development, but have not been merged into the core
+sub-package yet.
+
+Readers
+-------
+.. automodule:: carousel.contrib.readers
+
+Argument Reader
+---------------
+.. autoclass:: ArgumentReader
+   :members:
+
+Django Model Reader
+-------------------
+.. autoclass:: DjangoModelReader
+   :members:
+
+HD5F Reader
+-----------
+.. autoclass:: HDF5Reader
+   :members: