Merge branch 'main' of github.com:sdss/sdssdb into glimpse360

.github/workflows/pythonapp.yml

@@ -5,9 +5,9 @@ name: Python application
 
 on:
   push:
-    branches: [ master ]
+    branches: [ main ]
   pull_request:
-    branches: [ master ]
+    branches: [ main ]
 
 jobs:
   build:

.travis.yml

@@ -26,7 +26,7 @@ notifications:
 
 # repo branches to test
 branches:
-- master
+- main
 
 install:
 - pip install -U pip wheel --quiet

README.rst

@@ -1,7 +1,7 @@
 sdssdb
 ======
 
-|python| |docs| |travis| 
+|python| |docs| |travis|
 
 `SDSS <https://sdss.org>`__ product for database management.
 
@@ -31,7 +31,7 @@ How to use
     :scale: 100%
     :target: https://sdssdb.readthedocs.io/en/latest/?badge=latest
 
-.. |travis| image:: https://travis-ci.org/sdss/sdssdb.svg?branch=master
+.. |travis| image:: https://travis-ci.org/sdss/sdssdb.svg?branch=main
     :target: https://travis-ci.org/sdss/sdssdb
 
 .. |Python application| image:: https://github.com/sdss/sdssdb/workflows/Python%20application/badge.svg

bin/file2db

@@ -12,7 +12,7 @@ import astropy.table
 import click
 
 from sdssdb.connection import PeeweeDatabaseConnection
-from sdssdb.utils.database import file_to_db, table_exists, drop_table
+from sdssdb.utils.ingest import file_to_db, table_exists, drop_table
 
 
 try:

docs/sphinx/contributing.rst

@@ -4,7 +4,7 @@
 Contributing to sdssdb
 ======================
 
-Contributions to ``sdssdb`` are most welcome. Product development happens on its `GitHub repository <https://www.github.com/sdss/sdssdb>`__. For details on how to develop for an SDSS product refer to the `coding style guide <https://sdss-python-template.readthedocs.io/en/latest/standards.html>`__. All contributions to ``sdssdb`` need to be done as pull requests against the master branch.
+Contributions to ``sdssdb`` are most welcome. Product development happens on its `GitHub repository <https://www.github.com/sdss/sdssdb>`__. For details on how to develop for an SDSS product refer to the `coding style guide <https://sdss-python-template.readthedocs.io/en/latest/standards.html>`__. All contributions to ``sdssdb`` need to be done as pull requests against the ``main`` branch.
 
 
 Contributing a new database or schema
@@ -42,7 +42,7 @@ how to fill out those files depending on the library used.
 Peewee
 ^^^^^^
 
-For an example of how to implement a database with Peewee you can look at the `sdss5db <https://github.com/sdss/sdssdb/tree/master/python/sdssdb/peewee/sdss5db>`__ implementation. Let's start with the database connection in the ``__init__.py`` file. The basic structure is quite simple and would look something like ::
+For an example of how to implement a database with Peewee you can look at the `sdss5db <https://github.com/sdss/sdssdb/tree/main/python/sdssdb/peewee/sdss5db>`__ implementation. Let's start with the database connection in the ``__init__.py`` file. The basic structure is quite simple and would look something like ::
 
     from sdssdb.connection import PeeweeDatabaseConnection
     from sdssdb.peewee import BaseModel
@@ -100,7 +100,7 @@ To define a base class with reflection we do ::
             schema = 'stupendous'
             table_name = 'stupendous_table'
 
-When the connection is created this model will be reflected and autocompleted with all the columns that exist in the table. The reflection will include the `foreign keys <peewee:ForeignKeyField>` that have been defined for the table in the database. Sometimes this is not desirable and we'd rather create them manually. In this case we can add the attribute ``reflection_options = {'skip_foreign_keys': True}`` to ``Meta`` in the ``ReflectBaseModel``. You can check the `catalogdb <https://github.com/sdss/sdssdb/blob/master/python/sdssdb/peewee/sdss5db/catalogdb.py>`__ models for an implementation of this type.
+When the connection is created this model will be reflected and autocompleted with all the columns that exist in the table. The reflection will include the `foreign keys <peewee:ForeignKeyField>` that have been defined for the table in the database. Sometimes this is not desirable and we'd rather create them manually. In this case we can add the attribute ``reflection_options = {'skip_foreign_keys': True}`` to ``Meta`` in the ``ReflectBaseModel``. You can check the `catalogdb <https://github.com/sdss/sdssdb/blob/main/python/sdssdb/peewee/sdss5db/catalogdb.py>`__ models for an implementation of this type.
 
 If a model inherits from a base model using `.ReflectMeta`, it is possible to call ``Model.reflect`` manually to trigger a model reflection. This works even if ``use_reflection=False``.
 
@@ -110,7 +110,7 @@ The default Peewee reflection process requires multiple queries against the data
 SQLAlchemy
 ^^^^^^^^^^
 
-Creating a database connection and model classes for SQLALchemy is quite similar to Peewee. As before, refer to the implementation of `sdss5db <https://github.com/sdss/sdssdb/tree/master/python/sdssdb/sqlalchemy/sdss5db>`__ for a good example. In this case the ``__init__.py`` file would look like ::
+Creating a database connection and model classes for SQLALchemy is quite similar to Peewee. As before, refer to the implementation of `sdss5db <https://github.com/sdss/sdssdb/tree/main/python/sdssdb/sqlalchemy/sdss5db>`__ for a good example. In this case the ``__init__.py`` file would look like ::
 
     from sdssdb.connection import SQLADatabaseConnection
     from sqlalchemy.ext.declarative import declarative_base, DeferredReflection

docs/sphinx/installation.rst

@@ -30,6 +30,6 @@ and follow the instructions above to be sure it has been loaded. Remember that `
 
 or for a development version ::
 
-    sdss_install --github sdssdb master
+    sdss_install --github sdssdb main
 
 More instructions on how to install and use ``sdss_install`` are available `here <https://wiki.sdss.org/display/knowledge/sdss_install+bootstrap+installation+instructions>`__.

docs/sphinx/intro.rst

@@ -10,18 +10,22 @@ Getting started with sdssdb
 Making a simple query with ``sdssdb``
 -------------------------------------
 
-Imagine that you want to query ``catalogdb`` (the schema containing all the catalogues used for target selection in SDSS-V) and get all the Gaia targets within a range of magnitudes. In most cases this only requires a couple lines of code ::
+Imagine that you want to query ``catalogdb`` (the schema containing all the catalogues used for target selection in SDSS-V) and get all the Gaia targets within a range of magnitudes. In most cases this only requires a couple lines of code. This example assumes that you're running the code from a machine at Utah which has direct access to the ``operations.sdss.org`` machine. ::
 
-    >>> from sdssdb.peewee.sdss5db.catalogdb import GaiaDR2Source
-    >>> targets = GaiaDR2Source.select().where((GaiaDR2Source.phot_g_mean_mag > 15) & (GaiaDR2Source.phot_g_mean_mag < 16)).limit(10)
+    >>> from sdssdb.peewee.sdss5db.catalogdb import database
+    >>> database.set_profile('operations')
+    True
+    >>> from sdssdb.peewee.sdss5db.catalogdb import Gaia_DR2
+    >>> targets = Gaia_DR2.select().where((Gaia_DR2.phot_g_mean_mag > 15) & (Gaia_DR2.phot_g_mean_mag < 16)).limit(10)
 
-This will returns the first 10 results from Gaia DR2 with g magnitude in the range :math:`(15, 16)`. Simple. The previous example uses peewee but the equivalent for SQLAlchemy is quite similar ::
+This will returns the first 10 results from Gaia DR2 with g magnitude in the range :math:`(15, 16)`. Simple.
 
-    >>> from sdssdb.sqlalchemy.sdss5db.catalogdb import GaiaDR2Source, database
-    >>> session = database.Session()
-    >>> targets = session.query(GaiaDR2Source).filter((GaiaDR2Source.phot_g_mean_mag > 15) & (GaiaDR2Source.phot_g_mean_mag < 16)).limit(10).all()
+A subtlety is that the order of imports is important. Since the ``Gaia_DR2`` model is populated dynamically, it must be imported once a connection to the database has been accomplished. Alternatively, one can do ::
 
-.. warning:: Note that the implementation of ``catalogdb`` in SQLALchemy is very limited and should not be used in general.
+    >>> from sdssdb.peewee.sdss5db import catalogdb
+    >>> catalogdb.database.set_profile('operations')
+    True
+    >>> targets = catalogdb.Gaia_DR2.select().where((catalogdb.Gaia_DR2.phot_g_mean_mag > 15) & (catalogdb.Gaia_DR2.phot_g_mean_mag < 16)).limit(10)
 
 
 .. _available-databases:
@@ -116,14 +120,14 @@ Note that the level of readiness is not necessarily identical in both Peewee and
             <td class="active">catalogdb</td>
             <td class="success"></td>
             <td class="danger"></td>
-            <td align="center"><a class="glyphicon glyphicon-download-alt" href="https://github.com/sdss/sdssdb/raw/master/schema/sdss5db/catalogdb/sdss5db.catalogdb.pdf" alt="catalogdb full version"></a> <a class="glyphicon glyphicon-download-alt" style="color:green" href="https://github.com/sdss/sdssdb/raw/master/schema/sdss5db/catalogdb/sdss5db.catalogdb_lite.pdf" alt="catalogdb reduced version"></td>
+            <td align="center"><a class="glyphicon glyphicon-download-alt" href="https://github.com/sdss/sdssdb/raw/main/schema/sdss5db/catalogdb/sdss5db.catalogdb.pdf" alt="catalogdb full version"></a> <a class="glyphicon glyphicon-download-alt" style="color:green" href="https://github.com/sdss/sdssdb/raw/main/schema/sdss5db/catalogdb/sdss5db.catalogdb_lite.pdf" alt="catalogdb reduced version"></td>
         </tr>
         <tr>
             <td></td>
             <td class="active">targetdb</td>
             <td class="success"></td>
             <td class="danger"></td>
-            <td align="center"><a class="glyphicon glyphicon-download-alt" href="https://github.com/sdss/sdssdb/raw/master/schema/sdss5db/targetdb/sdss5db.targetdb.pdf"></a></td>
+            <td align="center"><a class="glyphicon glyphicon-download-alt" href="https://github.com/sdss/sdssdb/raw/main/schema/sdss5db/targetdb/sdss5db.targetdb.pdf"></a></td>
         </tr>
         <tr>
             <td class="active">archive</td>
@@ -148,6 +152,8 @@ The `~sdssdb.connection.DatabaseConnection` abstract class allows to connect to
     >>> db
     <SQLADatabaseConnection (dbname='manga', profile='local', connected=True)>
 
+(note that this example will only work if you have a local database called ``manga``)
+
 What happened here? `~sdssdb.connection.SQLADatabaseConnection` connected to the ``manga`` database using the ``local`` profile. A profile is simply a set of username, hostname, and port on which to look for a PostgreSQL server. ``sdssdb`` tries to be smart and select a profile that matches the machine on which you are working. That may not always work. For example, imagine that you are working on ``manga.wasatch.peaks`` but trying to connect to ``sdss5db`` which is running on ``operations-test.sdss.utah.edu`` ::
 
     >>> from sdssdb.connection import PeeweeDatabaseConnection
@@ -156,14 +162,14 @@ What happened here? `~sdssdb.connection.SQLADatabaseConnection` connected to the
 
 In this case the profile is not the appropriate for connecting to ``sdss5db`` and the connection fails. We can fix that by connecting with the correct profile ::
 
-    >>> db.set_profile('operations-test')
+    >>> db.set_profile('operations')
     True
     >>> db
-    <PeeweeDatabaseConnection (dbname='sdss5db', profile='operations-test', connected=True)>
+    <PeeweeDatabaseConnection (dbname='sdss5db', profile='operations', connected=True)>
 
 Or we could have connected to the database passing it a full set of parameters ::
 
-    >>> db.connect_from_parameters(user='sdss', host='operations-test.sdss.utah.edu', port=5432)
+    >>> db.connect_from_parameters(user='sdss', host='operations.sdss.org', port=5432)
     True
 
 In other cases you may have several databases running on the same server. You can prepare a connection using the appropriate profile and then connect to a specific database ::
@@ -213,7 +219,7 @@ where ``XXXX``, ``YYYY``, etc are the associated passwords for each set of param
 Supported Profiles
 ------------------
 
-The following `profiles <https://github.com/sdss/sdssdb/blob/master/python/sdssdb/etc/sdssdb.yml>`__ are included with sdssdb. When a :ref:`database connection <conn-db>` is created without an explicit profile, the hostname of the current machine is used to find the best possible profile. Profiles can be added or modified by creating a YAML file in ``~/.config/sdss/sdssdb.yaml`` with the same structure.
+The following `profiles <https://github.com/sdss/sdssdb/blob/main/python/sdssdb/etc/sdssdb.yml>`__ are included with sdssdb. When a :ref:`database connection <conn-db>` is created without an explicit profile, the hostname of the current machine is used to find the best possible profile. Profiles can be added or modified by creating a YAML file in ``~/.config/sdss/sdssdb.yaml`` with the same structure.
 
 * **local**: a generic localhost profile. Used if the hostname does not match any other profile.
 * **apo**: a user on the APO machines.