Merge pull request #683 from spacetelescope/v0.25.0

Release v0.25.0 [into master]
spacetelescope · Dec 31, 2020 · da81597 · da81597
2 parents 78ad809 + 39532e4
commit da81597
Show file tree

Hide file tree

Showing 43 changed files with 1,656 additions and 1,075 deletions.
diff --git a/CHANGES.rst b/CHANGES.rst
@@ -1,3 +1,50 @@
+0.25.0 (2020-12-31)
+===================
+
+New Features
+------------
+
+Project & API Documentation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Added project citation information to ``README``, along with a Zenodo badge
+- Added API Documentation for ``bokeh`` templating software
+
+Web Application
+~~~~~~~~~~~~~~~
+
+- Reorganized and made aesthetic improvements to instrument landing pages to be more user-friendly
+- Enabled more dynamic search options and aesthetic improvements for anomaly query webpage
+- Added web app view for displaying a particular table of the ``jwqldb`` database
+- Added webpage for displaying Bias Monitor results with ``bokeh`` plots
+
+``jwql`` Repository
+~~~~~~~~~~~~~~~~~~~
+
+- Changed ``utils.credentials.py`` to always authenticate a MAST user with a user-identified token in the ``config.json`` file, instead of using a cached token, which was sometimes causes errors
+- Updated software to support the latest versions of ``django`` and ``bokeh``
+- Removed ``affected_tables`` column of ``monitor`` database table, as it stored redundant information
+- Updated the Readnoise Monitor to work for all JWST instruments
+
+
+Bug Fixes
+---------
+
+
+Web Application
+~~~~~~~~~~~~~~~
+
+- Fixed bug in Readnoise Monitor webpage that was causing the web app and ``jwql`` database to hang
+
+``jwql`` Repository
+~~~~~~~~~~~~~~~~~~~
+
+- Fixed bug that was causing the ``test_amplifier_info()`` test in ``test_instrument_properties.py`` to fail; truth values were updated to reflect a change in the format of the returned dictionaries from the ``amplifier_info()`` function
+- Fixed bug in ``get_header_info()`` that was causing ``test_data_containers.py`` to fail; the function expected the filename without the FITS extension, and the returned header info is in a dictionary (not a string)
+- Fixed bug in ``test_utils.py``, and changed ``utils.py`` to make it robust in matching upper and lowercase detector names
+- Updated ``utils.instrument_properties`` fix MIRI amplifier bounds calculation when omitting reference pixels
+
+
 0.24.0 (2020-10-20)
 ===================
 
@@ -15,6 +62,8 @@ Web Application
 - Added webpage that describes how to use the JWQL web app API
 - Added webpage that enables a user to query the ``jwqldb`` database contents
 - Enabled more search options and aesthetic improvements for anomaly query webpage
+- Added webpage for displaying Readnoise Monitor results with ``bokeh`` plots
+- Added webpage for displaying Bad Pixel Monitor results with ``bokeh`` plots
 
 ``jwql`` Repository
 ~~~~~~~~~~~~~~~~~~~
@@ -24,8 +73,6 @@ Web Application
 - Added unit tests for Bias Monitor
 - Added unit tests for Bad Pixel Monitor
 - Added unit tests for ``bokeh`` templating library
-- Added webpage for displaying Readnoise Monitor results with ``bokeh`` plots
-- Added webpage for displaying Bad Pixel Monitor results with ``bokeh`` plots
 
 
 Bug Fixes

diff --git a/README.md b/README.md
@@ -10,6 +10,7 @@
 [![Build Status](https://ssbjenkins.stsci.edu/job/STScI/job/jwql/job/master/badge/icon)](https://ssbjenkins.stsci.edu/job/STScI/job/jwql/job/master/)
 [![Documentation Status](https://readthedocs.org/projects/jwql/badge/?version=latest)](https://jwql.readthedocs.io/en/latest/?badge=latest)
 [![STScI](https://img.shields.io/badge/powered%20by-STScI-blue.svg?colorA=707170&colorB=3e8ddd&style=flat)](http://www.stsci.edu)
+[![DOI](https://zenodo.org/badge/109727729.svg)](https://zenodo.org/badge/latestdoi/109727729)
 [![codecov](https://codecov.io/gh/spacetelescope/jwql/branch/develop/graph/badge.svg)](https://codecov.io/gh/spacetelescope/jwql)
 
 
@@ -106,6 +107,16 @@ The package should now appear if you run `conda list jwql`.
 Much of the `jwql` software depends on the existence of a `config.json` file within the `utils` directory.  This file contains data that may be unique to users and/or contain sensitive information.  Please see the [Config File wiki page](https://github.com/spacetelescope/jwql/wiki/Config-file) for instructions on how to provide this file.
 
 
+## Citation
+
+If you use `JWQL` for work/research presented in a publication (whether directly,
+or as a dependency to another package), we recommend and encourage the following acknowledgment:
+
+  This research made use of the open source Python package JWQL (Bourque et al, 2020).
+
+where (Bourque et al, 2020) is a citation of the Zenodo record available using the DOI badge above. By using the `Export` box in the lower right corner of the Zenodo page, you can export the citation in the format most convenient for you.
+
+
 ## Software Contributions
 
 There are two current pages to review before you begin contributing to the `jwql` development. The first is our [style guide](https://github.com/spacetelescope/jwql/blob/master/style_guide/README.md) and the second is our [suggested git workflow page](https://github.com/spacetelescope/jwql/wiki/git-&-GitHub-workflow-for-contributing), which contains an in-depth explanation of the workflow.

diff --git a/docs/source/bokeh_templating.rst b/docs/source/bokeh_templating.rst
@@ -0,0 +1,21 @@
+****************
+bokeh_templating
+****************
+
+template.py
+-----------------
+.. automodule:: jwql.bokeh_templating.template
+    :members:
+    :undoc-members:
+
+keyword_map.py
+-----------------
+.. automodule:: jwql.bokeh_templating.keyword_map
+    :members:
+    :undoc-members:
+
+factory.py
+-----------------
+.. automodule:: jwql.bokeh_templating.factory
+    :members:
+    :undoc-members:
diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -23,6 +23,7 @@ API documentation
    :maxdepth: 1
    :caption: Contents:
 
+   bokeh_templating.rst
    common_monitors.rst
    database.rst
    edb.rst

diff --git a/environment_python_3_6.yml b/environment_python_3_6.yml
@@ -3,9 +3,9 @@ channels:
 - http://ssb.stsci.edu/astroconda
 dependencies:
 - astroquery=0.4
-- authlib=0.15.1
-- bokeh>=1.0,<1.4
-- django=2.2.5
+- authlib=0.15.2
+- bokeh=2.2.3
+- django=3.1.4
 - drizzle=1.13
 - flake8=3.7.9
 - inflection=0.3.1
@@ -29,12 +29,13 @@ dependencies:
 - sphinx=3.0.3
 - sqlalchemy=1.3.17
 - twine=2.0.0
+- wtforms=2.3.3
 - pip:
   - asdf==2.7.1
-  - astropy==4.0.1
-  - codecov==2.1.9
-  - crds==10.0.0
-  - cryptography==3.1.1
+  - astropy==4.1
+  - codecov==2.1.10
+  - crds==10.1.0
+  - cryptography==3.2.1
   - jwedb>=0.0.3
   - jwst==0.17.1
   - pysqlite3==0.2.2

diff --git a/environment_python_3_7.yml b/environment_python_3_7.yml
@@ -3,9 +3,9 @@ channels:
 - http://ssb.stsci.edu/astroconda
 dependencies:
 - astroquery=0.4
-- authlib=0.15.1
-- bokeh>=1.0,<1.4
-- django=3.0.3
+- authlib=0.15.2
+- bokeh=2.2.3
+- django=3.1.4
 - flake8=3.8.3
 - inflection=0.3.1
 - ipython=7.16.1
@@ -28,12 +28,13 @@ dependencies:
 - sphinx=3.1.2
 - sqlalchemy=1.3.18
 - twine=2.0.0
+- wtforms=2.3.3
 - pip:
   - asdf==2.7.1
-  - astropy==4.0.1.post1
-  - codecov==2.1.9
-  - crds==10.0.0
-  - cryptography==3.1.1
+  - astropy==4.1
+  - codecov==2.1.10
+  - crds==10.1.0
+  - cryptography==3.2.1
   - jwedb==0.0.6
   - jwst==0.17.1
   - pysqlite3==0.4.3

diff --git a/environment_python_3_8.yml b/environment_python_3_8.yml
@@ -2,8 +2,9 @@ channels:
 - defaults
 - http://ssb.stsci.edu/astroconda
 dependencies:
-- authlib=0.15.1
-- django=3.0.3
+- authlib=0.15.2
+- bokeh=2.2.3
+- django=3.1.4
 - flake8=3.8.3
 - inflection=0.3.1
 - ipython=7.16.1
@@ -24,14 +25,14 @@ dependencies:
 - setuptools=49.6.0
 - sphinx=3.2.1
 - twine=2.0.0
+- wtforms=2.3.3
 - pip:
   - asdf==2.7.1
-  - astropy==4.0.1.post1
+  - astropy==4.1
   - astroquery==0.4.1
-  - bokeh>=1.0,<1.4
-  - codecov==2.1.9
-  - crds==10.0.0
-  - cryptography==3.1.1
+  - codecov==2.1.10
+  - crds==10.1.0
+  - cryptography==3.2.1
   - jwedb==0.0.6
   - jwst==0.17.1
   - pysiaf==0.9.0

diff --git a/jwql/database/database_interface.py b/jwql/database/database_interface.py
@@ -219,7 +219,6 @@ class Monitor(base):
     start_time = Column(DateTime, nullable=False)
     end_time = Column(DateTime, nullable=True)
     status = Column(Enum('SUCCESS', 'FAILURE', name='monitor_status'), nullable=True)
-    affected_tables = Column(ARRAY(String, dimensions=1), nullable=True)
     log_file = Column(String(), nullable=False)
 
 
@@ -423,6 +422,12 @@ class : obj
 NIRCamReadnoiseStats = monitor_orm_factory('nircam_readnoise_stats')
 NIRISSReadnoiseQueryHistory = monitor_orm_factory('niriss_readnoise_query_history')
 NIRISSReadnoiseStats = monitor_orm_factory('niriss_readnoise_stats')
+NIRSpecReadnoiseQueryHistory = monitor_orm_factory('nirspec_readnoise_query_history')
+NIRSpecReadnoiseStats = monitor_orm_factory('nirspec_readnoise_stats')
+MIRIReadnoiseQueryHistory = monitor_orm_factory('miri_readnoise_query_history')
+MIRIReadnoiseStats = monitor_orm_factory('miri_readnoise_stats')
+FGSReadnoiseQueryHistory = monitor_orm_factory('fgs_readnoise_query_history')
+FGSReadnoiseStats = monitor_orm_factory('fgs_readnoise_stats')
 
 if __name__ == '__main__':
 

diff --git a/jwql/database/monitor_table_definitions/fgs/fgs_readnoise_query_history.txt b/jwql/database/monitor_table_definitions/fgs/fgs_readnoise_query_history.txt
@@ -0,0 +1,8 @@
+INSTRUMENT, string
+APERTURE, string
+START_TIME_MJD, float
+END_TIME_MJD, float
+ENTRIES_FOUND, integer
+FILES_FOUND, integer
+RUN_MONITOR, bool
+ENTRY_DATE, datetime
diff --git a/jwql/database/monitor_table_definitions/fgs/fgs_readnoise_stats.txt b/jwql/database/monitor_table_definitions/fgs/fgs_readnoise_stats.txt
@@ -0,0 +1,35 @@
+UNCAL_FILENAME, string
+APERTURE, string
+DETECTOR, string
+SUBARRAY, string
+READ_PATTERN, string
+NINTS, string
+NGROUPS, string
+EXPSTART, string
+READNOISE_FILENAME, string
+FULL_IMAGE_MEAN, float
+FULL_IMAGE_STDDEV, float
+FULL_IMAGE_N, float_array_1d
+FULL_IMAGE_BIN_CENTERS, float_array_1d
+READNOISE_DIFF_IMAGE, string
+DIFF_IMAGE_MEAN, float
+DIFF_IMAGE_STDDEV, float
+DIFF_IMAGE_N, float_array_1d
+DIFF_IMAGE_BIN_CENTERS, float_array_1d
+ENTRY_DATE, datetime
+AMP1_MEAN, float
+AMP1_STDDEV, float
+AMP1_N, float_array_1d
+AMP1_BIN_CENTERS, float_array_1d
+AMP2_MEAN, float
+AMP2_STDDEV, float
+AMP2_N, float_array_1d
+AMP2_BIN_CENTERS, float_array_1d
+AMP3_MEAN, float
+AMP3_STDDEV, float
+AMP3_N, float_array_1d
+AMP3_BIN_CENTERS, float_array_1d
+AMP4_MEAN, float
+AMP4_STDDEV, float
+AMP4_N, float_array_1d
+AMP4_BIN_CENTERS, float_array_1d
diff --git a/jwql/database/monitor_table_definitions/miri/miri_readnoise_query_history.txt b/jwql/database/monitor_table_definitions/miri/miri_readnoise_query_history.txt
@@ -0,0 +1,8 @@
+INSTRUMENT, string
+APERTURE, string
+START_TIME_MJD, float
+END_TIME_MJD, float
+ENTRIES_FOUND, integer
+FILES_FOUND, integer
+RUN_MONITOR, bool
+ENTRY_DATE, datetime
diff --git a/jwql/database/monitor_table_definitions/miri/miri_readnoise_stats.txt b/jwql/database/monitor_table_definitions/miri/miri_readnoise_stats.txt
@@ -0,0 +1,35 @@
+UNCAL_FILENAME, string
+APERTURE, string
+DETECTOR, string
+SUBARRAY, string
+READ_PATTERN, string
+NINTS, string
+NGROUPS, string
+EXPSTART, string
+READNOISE_FILENAME, string
+FULL_IMAGE_MEAN, float
+FULL_IMAGE_STDDEV, float
+FULL_IMAGE_N, float_array_1d
+FULL_IMAGE_BIN_CENTERS, float_array_1d
+READNOISE_DIFF_IMAGE, string
+DIFF_IMAGE_MEAN, float
+DIFF_IMAGE_STDDEV, float
+DIFF_IMAGE_N, float_array_1d
+DIFF_IMAGE_BIN_CENTERS, float_array_1d
+ENTRY_DATE, datetime
+AMP1_MEAN, float
+AMP1_STDDEV, float
+AMP1_N, float_array_1d
+AMP1_BIN_CENTERS, float_array_1d
+AMP2_MEAN, float
+AMP2_STDDEV, float
+AMP2_N, float_array_1d
+AMP2_BIN_CENTERS, float_array_1d
+AMP3_MEAN, float
+AMP3_STDDEV, float
+AMP3_N, float_array_1d
+AMP3_BIN_CENTERS, float_array_1d
+AMP4_MEAN, float
+AMP4_STDDEV, float
+AMP4_N, float_array_1d
+AMP4_BIN_CENTERS, float_array_1d
diff --git a/jwql/database/monitor_table_definitions/nirspec/nirspec_readnoise_query_history.txt b/jwql/database/monitor_table_definitions/nirspec/nirspec_readnoise_query_history.txt
@@ -0,0 +1,8 @@
+INSTRUMENT, string
+APERTURE, string
+START_TIME_MJD, float
+END_TIME_MJD, float
+ENTRIES_FOUND, integer
+FILES_FOUND, integer
+RUN_MONITOR, bool
+ENTRY_DATE, datetime
diff --git a/jwql/database/monitor_table_definitions/nirspec/nirspec_readnoise_stats.txt b/jwql/database/monitor_table_definitions/nirspec/nirspec_readnoise_stats.txt
@@ -0,0 +1,35 @@
+UNCAL_FILENAME, string
+APERTURE, string
+DETECTOR, string
+SUBARRAY, string
+READ_PATTERN, string
+NINTS, string
+NGROUPS, string
+EXPSTART, string
+READNOISE_FILENAME, string
+FULL_IMAGE_MEAN, float
+FULL_IMAGE_STDDEV, float
+FULL_IMAGE_N, float_array_1d
+FULL_IMAGE_BIN_CENTERS, float_array_1d
+READNOISE_DIFF_IMAGE, string
+DIFF_IMAGE_MEAN, float
+DIFF_IMAGE_STDDEV, float
+DIFF_IMAGE_N, float_array_1d
+DIFF_IMAGE_BIN_CENTERS, float_array_1d
+ENTRY_DATE, datetime
+AMP1_MEAN, float
+AMP1_STDDEV, float
+AMP1_N, float_array_1d
+AMP1_BIN_CENTERS, float_array_1d
+AMP2_MEAN, float
+AMP2_STDDEV, float
+AMP2_N, float_array_1d
+AMP2_BIN_CENTERS, float_array_1d
+AMP3_MEAN, float
+AMP3_STDDEV, float
+AMP3_N, float_array_1d
+AMP3_BIN_CENTERS, float_array_1d
+AMP4_MEAN, float
+AMP4_STDDEV, float
+AMP4_N, float_array_1d
+AMP4_BIN_CENTERS, float_array_1d