Merge 67bfe50 into 2b1caad

docs/sphinx/api.rst

@@ -7,13 +7,16 @@ Database connections
 
 .. automodule:: sdssdb.connection
    :members: DatabaseConnection, PeeweeDatabaseConnection, SQLADatabaseConnection
-   :undoc-members:
    :show-inheritance:
 
 
 Peewee
 ------
 
+.. autoclass:: sdssdb.peewee.ReflectMeta
+   :members:
+   :show-inheritance:
+
 .. autoclass:: sdssdb.peewee.BaseModel
    :members:
    :show-inheritance:
@@ -32,7 +35,11 @@ SQLAlchemy
 Utils
 -----
 
-.. automodule:: sdssdb.utils.database
+.. automodule:: sdssdb.utils.ingest
+    :members:
+    :show-inheritance:
+
+.. automodule:: sdssdb.utils.internals
     :members:
     :show-inheritance:
 

docs/sphinx/conf.py

@@ -49,7 +49,7 @@
 # ones.
 extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon', 'sphinx.ext.autosummary',
               'sphinx.ext.todo', 'sphinx.ext.viewcode', 'sphinx.ext.mathjax',
-              'sphinx.ext.intersphinx', 'releases']
+              'sphinx.ext.intersphinx', 'sdsstools.releases', 'sphinx.ext.napoleon']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -133,7 +133,8 @@
 napoleon_use_rtype = False
 napoleon_use_ivar = True
 
-rst_epilog = """
+rst_epilog = f"""
+.. |sdssdb_version| replace:: {__version__}
 .. |numpy_array| replace:: Numpy array
 """
 
@@ -292,11 +293,9 @@ def generate_schema_graphs():
                       UserWarning)
         return
 
-    schemas = ['sdss5db.targetdb', 'sdss5db.catalogdb', 'operationsdb.platedb',
-               'operationsdb.mangadb', 'operationsdb.apogeeqldb']
+    schemas = ['operationsdb.platedb', 'operationsdb.mangadb', 'operationsdb.apogeeqldb']
 
-    output_dir = os.path.join(os.path.dirname(__file__),
-                              '_static/schema_graphs/auto')
+    output_dir = os.path.join(os.path.dirname(__file__), '_static/schema_graphs/auto')
 
     if os.path.exists(output_dir):
         shutil.rmtree(output_dir)

docs/sphinx/contributing.rst

@@ -7,8 +7,8 @@ Contributing to sdssdb
 Contributions to ``sdssdb`` are most welcome. Product development happens on its `GitHub repository <https://www.github.com/sdss/sdssdb>`__. For details on how to develop for an SDSS product refer to the `coding style guide <https://sdss-python-template.readthedocs.io/en/latest/standards.html>`__. All contributions to ``sdssdb`` need to be done as pull requests against the master branch.
 
 
-Contributing a new database
----------------------------
+Contributing a new database or schema
+-------------------------------------
 
 In addition to improvements to the code, you can contribute database connections and model classes for your databases. To do so, first remember the directory structure of ``sdssdb``
 
@@ -59,7 +59,7 @@ For an example of how to implement a database with Peewee you can look at the `s
 
 The first two lines simply import the base classes for the database connection and base model class. We then subclass `~sdssdb.connection.PeeweeDatabaseConnection` to create the connection for ``awesomedb``, overriding the `~sdssdb.connection.PeeweeDatabaseConnection.dbname` attribute. We then instantiate the database connection as ``database``. Note the ``autoconnect=True`` parameter which tells the database connection to try to use the best available profile to connect when the class gets instantiated. Finally, we subclass `~sdssdb.peewee.BaseModel` and we bind the database connection to it.
 
-Next we need to creates the model classes themselves. At its simplest, a model class represents a table in a given schema and contains a list of the columns in the table, each one as a class attribute. Model classes must subclass from a base class (``AwesomedbModel`` in our example) that has been linked to the database connection. Differently from SQLAlchemy, Peewee requires that all the columns be explicitly described, as opposed to autoloaded. To help with this task you can use the `pwiz <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pwiz-a-model-generator>`__ model generator. For example, to create a file with the list of model classes for ``stupendous`` you would run, from a terminal ::
+Next we need to creates the model classes themselves. At its simplest, a model class represents a table in a given schema and contains a list of the columns in the table, each one as a class attribute. Model classes must subclass from a base class (``AwesomedbModel`` in our example) that has been linked to the database connection. The default mode in Peewee is to explicitely define all columns, as opposed to autoloaded. To help with this task you can use the `pwiz <http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pwiz-a-model-generator>`__ model generator. For example, to create a file with the list of model classes for ``stupendous`` you would run, from a terminal ::
 
     python -m pwiz -e postgresql -s stupendous awesomedb > stupendous.py
 
@@ -72,6 +72,30 @@ Once the file has been generated you will need to do some changes. On the top of
 
 The first line conveniently allows for access to the database connection from the schema submodule. The second one renames ``AwesomedbModel`` to ``BaseModel`` so that all the model classes in the file inherit from it. You'll probably need to make some other changes to the file, especially to the foreign keys to make sure they match your naming requirements.
 
+Using reflection with Peewee
+''''''''''''''''''''''''''''
+
+Peewee provides a :ref:`reflection <peewee:reflection>` utility (internally used by ``pwiz``). Base on this tool we developed a `reflection metaclass <.ReflectMeta>` that can be used to expedite the creating of models by only requiring to define foreign keys. Note that this is not an official component of Peewee and it comes with certain caveats. Before using the reflection metaclass, make sure to read the `API documentation <.ReflectMeta>`.
+
+To define a base class with reflection with do ::
+
+    import peewee
+    from sdssdb.peewee import ReflectMeta
+
+    class ReflectBaseModel(peewee.Model, metaclass=ReflectMeta):
+        class Meta:
+            primary_key = False     # To make sure Peewee doesn't add its own PK.
+            use_reflection = False  # We'll enable reflection manually for certain models.
+            database = database
+
+    class AwesomedbModel(ReflectBaseModel):
+        class Meta:
+            use_reflection = True
+            schema = 'stupendous'
+            table_name = 'stupendous_table'
+
+When the connection is created this model will be reflected and autocompleted with all the columns that exist in the table. The reflection does not include `foreign keys <peewee:ForeignKeyField>`, which must be created manually (along with their referenced columns). You can check the `catalogdb <https://github.com/sdss/sdssdb/blob/master/python/sdssdb/peewee/sdss5db/catalogdb.py>`__ models for an implementation of this type.
+
 
 SQLAlchemy
 ^^^^^^^^^^

docs/sphinx/installation.rst

@@ -10,11 +10,13 @@ Installation
 
     pip install sdssdb
 
-then you can check that it works by running a python terminal and doing ::
+then you can check that it works by running a python terminal and doing
+
+.. parsed-literal::
 
     >>> import sdssdb
     >>> sdssdb.__version__
-    0.3.3dev
+    |sdssdb_version|
 
 If you are working from Utah, ``sdssdb`` is installed as a module and you should be able to do ::
 

docs/sphinx/intro.rst

@@ -21,6 +21,8 @@ This will returns the first 10 results from Gaia DR2 with g magnitude in the ran
     >>> session = database.Session()
     >>> targets = session.query(GaiaDR2Source).filter((GaiaDR2Source.phot_g_mean_mag > 15) & (GaiaDR2Source.phot_g_mean_mag < 16)).limit(10).all()
 
+.. warning:: Note that the implementation of ``catalogdb`` in SQLALchemy is very limited and should not be used in general.
+
 
 Available databases
 -------------------
@@ -111,15 +113,15 @@ Note that the level of readiness is not necessarily identical in both Peewee and
             <td class="active">sdss5db</td>
             <td class="active">catalogdb</td>
             <td class="success"></td>
-            <td class="success"></td>
-            <td align="center"><a class="glyphicon glyphicon-download-alt" href="_static/schema_graphs/auto/sdss5db.catalogdb.pdf"></a></td>
+            <td class="warning"></td>
+            <td align="center"><a class="glyphicon glyphicon-download-alt" href="_static/schema_graphs/sdss5db.catalogdb.pdf" alt="catalogdb full version"></a> <a class="glyphicon glyphicon-download-alt" style="color:green" href="_static/schema_graphs/sdss5db.catalogdb_lite.pdf" alt="catalogdb reduced version"></td>
         </tr>
         <tr>
             <td></td>
             <td class="active">targetdb</td>
             <td class="success"></td>
             <td class="success"></td>
-            <td align="center"><a class="glyphicon glyphicon-download-alt" href="_static/schema_graphs/auto/sdss5db.targetdb.pdf"></a></td>
+            <td align="center"><a class="glyphicon glyphicon-download-alt" href="_static/schema_graphs/sdss5db.targetdb.pdf"></a></td>
         </tr>
         <tr>
             <td class="active">archive</td>
@@ -185,8 +187,7 @@ Now imagine the case in which you are running ``sdssdb`` from your local compute
     >>> database
     <PeeweeDatabaseConnection (dbname='apodb', profile='local', connected=True)>
 
-There are two database connections, ``SQLADatabaseConnection`` and ``PeeWeeDatabaseConnection``, one for each mapping library. Each database connection has two keyword arguments: a user/machine profile, a database name.  The connection will automatically attempt to connect to the specified database with the profile unless the ``autoconnect`` keyword is set to `False`.
-::
+There are two database connections, ``SQLADatabaseConnection`` and ``PeeWeeDatabaseConnection``, one for each mapping library. Each database connection has two keyword arguments: a user/machine profile, a database name.  The connection will automatically attempt to connect to the specified database with the profile unless the ``autoconnect`` keyword is set to `False`. ::
 
     # load a database connection with the Utah manga machine profile and connect to the manga database. To create a Peewee conenction replace with PeeweeDatabaseConnection.
     from sdssdb.connection import SQLADatabaseConnection

python/sdssdb/__init__.py

@@ -1,68 +1,34 @@
 # encoding: utf-8
-# flake8: noqa
 
 from __future__ import absolute_import, division, print_function, unicode_literals
 
-import os
 import warnings
-from pkg_resources import parse_version
 
-import yaml
+from sdsstools import get_config, get_logger, get_package_version
 
 
-# Inits the logging system. Only shell logging, and exception and warning catching.
-# File logging can be started by calling log.start_file_logger(name).
-from .misc import log  # noqa
-
-
-def merge(user, default):
-    """Merges a user configuration with the default one."""
-
-    if isinstance(user, dict) and isinstance(default, dict):
-        for kk, vv in default.items():
-            if kk not in user:
-                user[kk] = vv
-            else:
-                user[kk] = merge(user[kk], vv)
-
-    return user
+warnings.filterwarnings(
+    'ignore', '.*Skipped unsupported reflection of expression-based index .*q3c.*')
 
 
 NAME = 'sdssdb'
 
+__version__ = get_package_version(path='./', package_name=NAME)
 
-yaml_kwds = dict()
-if parse_version(yaml.__version__) >= parse_version('5.1'):
-    yaml_kwds.update(Loader=yaml.FullLoader)
-
+log = get_logger(NAME)
 
 # Loads config
-config_file = os.path.dirname(__file__) + '/etc/{0}.yml'.format(NAME)
-config = yaml.load(open(config_file), **yaml_kwds)
-
-# If there is a custom configuration file, updates the defaults using it.
-custom_config_fn = os.path.expanduser('~/.{0}/{0}.yml'.format(NAME))
-if os.path.exists(custom_config_fn):
-    config = merge(yaml.load(open(custom_config_fn), **yaml_kwds), config)
-
-# from sdssdb.sqlalchemy import db
-# config['db'] = db
-
-warnings.filterwarnings(
-    'ignore', '.*Skipped unsupported reflection of expression-based index .*q3c.*')
-
-
-__version__ = '0.3.3dev'
+config = get_config(NAME, user_path='~/.sdssdb/sdssdb.yml')
 
 
 try:
-    import peewee
+    import peewee  # noqa
     _peewee = True
 except ImportError:
     _peewee = False
 
 try:
-    import sqlalchemy
+    import sqlalchemy  # noqa
     _sqla = True
 except ImportError:
     _sqla = False
@@ -71,9 +37,7 @@ def merge(user, default):
                           'Install at least one of them to use sdssdb.')
 
 
-from .connection import DatabaseConnection
-
 if _peewee:
-    from .connection import PeeweeDatabaseConnection
+    from .connection import PeeweeDatabaseConnection  # noqa
 if _sqla:
-    from .connection import SQLADatabaseConnection
+    from .connection import SQLADatabaseConnection  # noqa

python/sdssdb/connection.py

@@ -14,12 +14,13 @@
 
 import abc
 import importlib
+import re
 import socket
 
 import six
 from pgpasslib import getpass
 
-from sdssdb import config, log, _peewee, _sqla
+from sdssdb import _peewee, _sqla, config, log
 
 
 if _peewee:
@@ -117,7 +118,7 @@ def set_profile(self, profile=None, connect=True):
             # Tries to find a profile whose domain matches the hostname
             for profile in config:
                 if 'domain' in config[profile] and config[profile]['domain'] is not None:
-                    if hostname.endswith(config[profile]['domain']):
+                    if re.match(config[profile]['domain'], hostname):
                         self.profile = profile
                         break
 
@@ -172,19 +173,10 @@ def connect(self, dbname=None, silent_on_fail=False, **connection_params):
         # Gets the necessary configuration values from the profile
         db_configuration = {}
         for item in ['user', 'host', 'port']:
-            if item in connection_params and connection_params[item] is not None:
+            if item in connection_params:
                 db_configuration[item] = connection_params[item]
             else:
                 profile_value = config[self.profile].get(item, None)
-
-                # If the hostname is the same as the current domain,
-                # do not specify the host. This helps with the configuration
-                # of the PSQL security at Utah.
-                if item == 'host':
-                    domain = socket.getfqdn()
-                    if profile_value == domain:
-                        continue
-
                 db_configuration[item] = profile_value
 
         dbname = dbname or self.dbname
@@ -307,7 +299,17 @@ def become_user(self):
 if _peewee:
 
     class PeeweeDatabaseConnection(DatabaseConnection, PostgresqlDatabase):
-        """Peewee database connection implementation."""
+        """Peewee database connection implementation.
+
+        Attributes
+        ----------
+        models : list
+            Models bound to this database. Only models that are bound using
+            `~sdssdb.peewee.BaseModel` are handled.
+
+        """
+
+        models = list()
 
         def __init__(self, *args, **kwargs):
 
@@ -336,10 +338,15 @@ def _conn(self, dbname, silent_on_fail=False, **params):
                 self.dbname = dbname
             except OperationalError:
                 if not silent_on_fail:
-                    log.warning(f'failed to connect to database {self.database!r}.', UserWarning)
+                    log.warning(f'failed to connect to database {self.database!r}.')
                 PostgresqlDatabase.init(self, None)
                 self.connected = False
 
+            if self.is_connection_usable():
+                for model in self.models:
+                    if hasattr(model, 'reflect'):
+                        model.reflect()
+
             return self.connected
 
 

python/sdssdb/etc/sdssdb.yml

@@ -36,9 +36,9 @@ sdssadmin:
 
 operations:
     user: sdss
-    host: operations.sdss.org
+    host: null
     port: 5432
-    domain: operations.sdss.org
+    domain: operations.sdss.*
 
 manga:
     user: sdss