Newcandidate 370rc0 (#1747)

Co-authored-by: Steve Goldman <32876747+s-goldman@users.noreply.github.com> Co-authored-by: Joseph Hunkeler <jhunkeler@users.noreply.github.com> Co-authored-by: Nadia Dencheva <nadia.dencheva@gmail.com> Co-authored-by: codesee-maps[bot] <86324825+codesee-maps[bot]@users.noreply.github.com>
spacetelescope · Feb 21, 2024 · f9a8d7c · f9a8d7c
1 parent 61f998c
commit f9a8d7c
Show file tree

Hide file tree

Showing 143 changed files with 647 additions and 607 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -54,13 +54,11 @@ jobs:
       cache-path: ${{ needs.crds.outputs.path }}
       cache-key: crds-${{ needs.crds.outputs.context }}
       envs: |
-        - linux: test-xdist
-          python-version: 3.9
         - linux: test-xdist
           python-version: 3.10
         - linux: test-xdist
           python-version: 3.11
         - macos: test-xdist
           python-version: 3.11
-        - linux: test-xdist-cov
+        - linux: test-xdist-cov-devdeps
           coverage: codecov
diff --git a/.github/workflows/ci_cron.yml b/.github/workflows/ci_cron.yml
@@ -43,8 +43,8 @@ jobs:
       cache-path: ${{ needs.crds.outputs.path }}
       cache-key: crds-${{ needs.crds.outputs.context }}
       envs: |
-        - macos: test-xdist
-          python-version: 3.9
         - macos: test-xdist
           python-version: 3.10
+        - macos: test-xdist
+          python-version: 3.11
         - linux: test-devdeps-xdist
diff --git a/.gitignore b/.gitignore
@@ -20,3 +20,8 @@ coverage.xml
 *.egg-info
 *.swp
 *~
+astrodrizzle.log
+hapcut_utility.log
+headerlet.log
+svm_poller_file.txt
+.vscode/settings.json
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -18,14 +18,48 @@ number of the code change for that issue.  These PRs can be viewed at:
     https://github.com/spacetelescope/drizzlepac/pulls
 
 
-3.6.2rc2 (unreleased)
-=====================
+3.7.0rc0 (21-Feb-2024) Infrastructure Build
+===========================================
+
+- Update project.toml file to specify numpy>=1.18,  <2.0 [#1743]
+
+- Update project.toml file to specify python_requires>=3.10 [#1737]
+
+- Github branch "master" renamed to main. [#1725]
+
+- Clean up spacing in toml file to eliminate improper spacing to
+  avoid decprecation warning [#1731]
+
+- Clean up YAML diagram in of workflows area [#1728]
+
+- Updated installation instructions and small text changes [#1727]
+
+- Remove outdated references of Pyraf and change to Python [#1726]
+
+- Fix to add "stregion" to the requirements-dev.txt file to fix the build
+  error under Python 3.12. [#1714]
+
+- Reorganized the readthedocs documentation with the help of various STScI
+  staff. [#1717] 
+
+- Updates requirements-dev.txt to not install eggs that cause problems
+  for the regression tests [#1721]
+
+- Regression Testing: allow "dev" jobs to fail [#1718]
+
+- Initial setup for Architectural Design Records used to keep track of top-level
+  thinking behind the code. [#1697]
+
+3.6.2 (27-Nov-2023)
+===================
+
+>>>>>>> 04cf04c1 (HLA-1175: New Readthedocs structure (#1717))
 - At this time pin Astrocut to versions <=0.9 to avoid conflicts with urllib3
-  package.  [#1145]
+  package.  [#1689]
 
 - Added functionality to allow the use of a two-column poller file. This is used
   to update the WFPC2 SVM aperture header keywords from the values in the poller 
-  file. [#1683, #1150]
+  file. [#1683]
 
 - Removed the version restriction on matplotlib. [#1649]
 

diff --git a/JenkinsfileRT b/JenkinsfileRT
@@ -30,9 +30,20 @@ bc1.test_cmds = ["pytest --cov=./ --basetemp=tests_output --junitxml=results.xml
 bc1.test_configs = [data_config]
 bc1.failedFailureThresh = 0
 
-bc2 = utils.copy(bc1)
+bc2 = new BuildConfig()
 bc2.name = '3.9-dev'
-bc2.build_cmds[1] = "pip install -r requirements-dev.txt --upgrade -e '.[test]'"
+bc2.nodetype = 'linux'
+bc2.env_vars = ['TEST_BIGDATA=https://bytesalad.stsci.edu/artifactory']
+bc2.conda_packages = ['python=3.9']
+bc2.build_cmds = ["pip install numpy astropy codecov pytest-cov ci-watson || true",
+                 "pip install -r requirements-dev.txt --upgrade -e '.[test]' || true",
+                 "pip freeze || true"]
+bc1.test_cmds = ["pytest --cov=./ --basetemp=tests_output --junitxml=results.xml --bigdata || true",
+                "codecov || true"]
+bc1.test_configs = [data_config]
+// Apply a large failure threshold to prevent marking the pipeline job failed
+// when xunit ingests any test results
+bc1.failedFailureThresh = 1000
 
 bc3 = new BuildConfig()
 bc3.runtime.add('CFLAGS=-std=gnu99')

diff --git a/LICENSE.txt b/LICENSE.txt
@@ -1,6 +1,6 @@
 BSD 3-Clause License
 
-Copyright (c) 2021, Association of Universities for Research in Astronomy (AURA)
+Copyright (c) 2024, Association of Universities for Research in Astronomy (AURA)
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without

diff --git a/README.md b/README.md
@@ -24,6 +24,18 @@ can be found at:
 
 `Drizzlepac` is installed when you install the `stenv` conda environment (a replacement for `astroconda`). Select your desired release and follow the instructions on the [`stenv` installation page](https://stenv.readthedocs.io/en/latest/getting_started.html). 
 
+## Install with pip
+
+```bash
+$ pip install git+https://github.com/spacetelescope/drizzlepac.git
+```
+The option `--no-use-pep517` MAY be required in order to correctly build 
+the C extensions with `pip` versions up to 22.2, after commenting out 
+the `build-backend` from the `pyproject.toml` config file.
+
+Support for installing using `pip` is still evolving, so use of this 
+command is provided on an experimental basis for now.
+
 ## From Source
 
 ### Clone this repository
@@ -39,26 +51,12 @@ an interactive `python` or `ipython` session, we recommend you do not skip
 this step.
 
 ```bash
-$ python setup.py build_sphinx
+$ cd doc/
+$ make html
 ```
 
-### Install drizzlepac
+### Install DrizzlePac
 
 ```bash
 $ python setup.py install
 ```
-
-##### SUPPORT for PIP Installation:
-Installation tools are evolving to rely on commands such as `pip` 
-to build and install software.  This package can now be installed 
-using the following command:
-
-```bash
-$ pip install .
-```
-The option `--no-use-pep517` MAY be required in order to correctly build 
-the C extensions with `pip` versions up to 22.2, after commenting out 
-the `build-backend` from the `pyproject.toml` config file.
-
-**Support for installing using `pip` is still evolving, so use of this 
-command is provided on an experimental basis for now.**
diff --git a/doc/ADR_readme.md b/doc/ADR_readme.md
@@ -0,0 +1,35 @@
+# Architectural Decision Records
+
+This directory is used to host architectural decision records (ADRs). These are used to outline high-level decisions about the codebase. 
+
+## Suggestions for writing good ADRs
+
+Characteristics of a good ADR:
+
+  * Rationale: Explain the reasons for doing the particular AD. This can include the context (see below), pros and cons of various potential choices, feature comparions, cost/benefit discussions, and more.
+
+  * Specific: Each ADR should be about one AD, not multiple ADs.
+
+  * Timestamps: Identify when each item in the ADR is written. This is especially important for aspects that may change over time, such as costs, schedules, scaling, and the like.
+
+  * Immutable: Don't alter existing information in an ADR. Instead, amend the ADR by adding new information, or supersede the ADR by creating a new ADR.
+
+Characteristics of a good "Context" section in an ADR:
+
+  * Explain your organization's situation and business priorities.
+
+  * Include rationale and considerations based on social and skills makeups of your teams.
+
+  * Include pros and cons that are relevant, and describe them in terms that align with your needs and goals.
+
+Characteristics of good "Consequences" section in an ADR:
+
+  * Explain what follows from making the decision. This can include the effects, outcomes, outputs, follow ups, and more.
+
+  * Include information about any subsequent ADRs. It's relatively common for one ADR to trigger the need for more ADRs, such as when one ADR makes a big overarching choice, which in turn creates needs for more smaller decisions.
+
+  * Include any after-action review processes. It's typical for teams to review each ADR one month later, to compare the ADR information with what's happened in actual practice, in order to learn and grow.
+
+A new ADR may take the place of a previous ADR:
+
+  * When an AD is made that replaces or invalidates a previous ADR, then a new ADR should be created
diff --git a/doc/ADRs.md b/doc/ADRs.md
@@ -0,0 +1,28 @@
+# Updating WCS in Headers
+
+## Context
+
+In updating WCS information there was a worry that information about datasets used in research would be lost.
+
+## Decision
+
+In order to ensure reproducibility and the ability to share consistent datasets "headerlets" were created. These hold the wcs inforamtion without any data.  
+
+## Consequences
+
+Headerlets can now be used to easily update and select WCS for datasets. Headerlets are files with the _hlet.fits suffix
+
+
+# Updating APERTURE keyword through poller
+
+## Context
+
+Two types of poller files exist. Full poller files, that include all of the necessary information, and simple poller files that include only the filenames; additional information is taken from the header. We needed a way to update a header keyword ("APERTURE") and it was found that the easiest way to do this was to pass the keyword through the second column of the poller file.
+
+## Decision
+
+The second column of a poller file is now reserved (for WFPC2) for passing the aperture keyword through the code to update the header keyword. For other instruments, the code treats a two-column poller file as a simple (filename-only) poller file. 
+
+## Consequences
+
+Caution must be taken is using variations of the poller file while processing WFPC2 data.
diff --git a/doc/source/CHANGELOG.rst b/doc/source/CHANGELOG.rst
@@ -1,4 +1,2 @@
-Release Notes
--------------
 
 .. include:: ../../CHANGELOG.rst
diff --git a/doc/source/conf.py b/doc/source/conf.py
@@ -132,7 +132,7 @@ def check_sphinx_version(expected_version):
 
 # General information about the project.
 project = u'DrizzlePac'
-copyright = u'2021, Warren Hack, Nadia Dencheva, Chris Sontag, Megan Sosey, Michael Droettboom, Mihai Cara, Michael Dulude, Michele De La Pena, Steve Goldman'
+copyright = u'2024, Steve Goldman, Michele De La Pena, Warren Hack, Mihai Cara, Nadia Dencheva'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

diff --git a/doc/source/drizzlepac_api.rst b/doc/source/drizzlepac_api.rst
@@ -0,0 +1,27 @@
+DrizzlePac Interface
+====================
+.. _drizzlepac_api:
+
+DrizzlePac is written in a hierarchical manner with top-level code in python, and some lower-level code in C. The primary user interface for drizzling individual frames is the astrodrizzle.py ``Astrodrizzle()`` function, which accepts input parameters as variables or as a user-specified configuration (.cfg) file. Examples of how to use these functions can be found in the DrizzlePac `Notebook Tutorials <https://github.com/spacetelescope/notebooks/tree/master/notebooks/DrizzlePac>`_. 
+
+The main steps of the drizzling process are listed below. For more information on these steps please refer to the DrizzlePac `Handbook <https://www.stsci.edu/scientific-community/software/drizzlepac.html>`_. Additional information on the Hubble Advanced Product (HAP) Single- and Multi-visit mosaics can be found in the MAST Data Products section.
+
+.. toctree:: 
+    :maxdepth: 1
+
+    drizzlepac_api/astrodrizzle
+    drizzlepac_api/process
+    drizzlepac_api/static
+    drizzlepac_api/sky
+    drizzlepac_api/adrizzle
+    drizzlepac_api/median
+    drizzlepac_api/ablot
+    drizzlepac_api/drizcr
+    drizzlepac_api/outimage
+
+
+Index
+-----
+
+* :ref:`genindex`
+* :ref:`modindex`
diff --git a/doc/source/reference/ablot.rst → doc/source/drizzlepac_api/ablot.rst b/doc/source/reference/ablot.rst → doc/source/drizzlepac_api/ablot.rst
diff --git a/doc/source/reference/adrizzle.rst → doc/source/drizzlepac_api/adrizzle.rst b/doc/source/reference/adrizzle.rst → doc/source/drizzlepac_api/adrizzle.rst
diff --git a/doc/source/reference/astrodrizzle.rst → doc/source/drizzlepac_api/astrodrizzle.rst b/doc/source/reference/astrodrizzle.rst → doc/source/drizzlepac_api/astrodrizzle.rst
diff --git a/doc/source/reference/drizcr.rst → doc/source/drizzlepac_api/drizcr.rst b/doc/source/reference/drizcr.rst → doc/source/drizzlepac_api/drizcr.rst
diff --git a/doc/source/reference/median.rst → doc/source/drizzlepac_api/median.rst b/doc/source/reference/median.rst → doc/source/drizzlepac_api/median.rst
diff --git a/doc/source/reference/outimage.rst → doc/source/drizzlepac_api/outimage.rst b/doc/source/reference/outimage.rst → doc/source/drizzlepac_api/outimage.rst
diff --git a/doc/source/reference/process.rst → doc/source/drizzlepac_api/process.rst b/doc/source/reference/process.rst → doc/source/drizzlepac_api/process.rst
@@ -1,8 +1,8 @@
 .. _processinput:
 
-*************
-Process Input
-*************
+*******************************
+Process Input (data validation)
+*******************************
 
 This module supports the interpretation and initial verification of all the
 input files specified by the user. These functions:
@@ -19,13 +19,4 @@ input files specified by the user. These functions:
 
 .. automodule:: drizzlepac.processInput
    :members:
-   :undoc-members:
-
-
-ResetBits Update of Input
-=========================
-
-This module provides the capability to set a specific set of bit values in the input DQ arrays to zero. This allows a user to reset pixels previously erroneously flagged as cosmic-rays to good for reprocessing with improved alignment or detection parameters.
-
-.. automodule:: drizzlepac.resetbits
-   :members:
+   :undoc-members:
diff --git a/doc/source/reference/sky.rst → doc/source/drizzlepac_api/sky.rst b/doc/source/reference/sky.rst → doc/source/drizzlepac_api/sky.rst
diff --git a/doc/source/reference/static.rst → doc/source/drizzlepac_api/static.rst b/doc/source/reference/static.rst → doc/source/drizzlepac_api/static.rst
diff --git a/doc/source/explanation.rst b/doc/source/explanation.rst