Merge 1da907c into a53174f

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.5.3dev
+current_version = 3.0.0dev
 commit = False
 tag = False
 tag_name = {new_version}
@@ -14,7 +14,7 @@ values =
 	dev
 	alpha
 
-[bumpversion:file:python/mangadap/__init__.py]
+[bumpversion:file:mangadap/__init__.py]
 
 [bumpversion:file:setup.py]
 

.gitignore

@@ -10,4 +10,5 @@ examples/data/*.fits.gz
 examples/output/HYB10-GAU-MILESHC/
 examples/output/VOR10-GAU-MILESHC/
 examples/output/common/
+mangadap/data/remote
 

.travis.yml

@@ -7,11 +7,9 @@ cache:
   pip: true
   timeout: 1000
 
-sudo: false
-
 python:
-    - '3.6'
     - '3.7'
+    - '3.8'
 
 os:
     - linux
@@ -44,7 +42,7 @@ install:
     - python setup.py install
 
 script:
-    - pytest python/mangadap/tests --cov python/mangadap --cov-report html
+    - pytest mangadap/tests --cov mangadap --cov-report html
 
 after_success:
     - coveralls

CHANGES.md

@@ -15,23 +15,94 @@ TODO
  - Define an emission-line FOM flag that identifies poorly fit lines
  - Documentation of config examples are out of date!
  - Get rid of the filter keyword from the output headers...
- - Abstract input datacube reading
- - Minimize the hurdles to running `manga_dap`; i.e., allow it to run
-   without the input and plan parameter files
- - Stop transposing the datacube data!
- - Get rid of optional `dapsrc` keyword; now defined by the package
  - Fix ELMREBIN in 2nd round of moment calculations (not sure this is
    still a problem...)
  - Keep all the reference files to the common directory?
  - Make the package pip installable
  - Change format of emission-line module reference file?
     - Make more independent of stellar-continnuum module reference file
     - Change `BASE` extension to `CONTINUUM` extension.
+ - Enable composite indices
+ - Allow construction of DAP MAPS headers to indicate masks that don't
+   have the same prepended extenstion name... (e.g., quality mask of
+   `SPECINDEX_CORR` should be `SPECINDEX_MASK`).
+ - Differentiate between mask provided for BF index and mask for main
+   continuum value? (e.g., latter only cares about the side bands)
+ - Deprecate `mangadap.survey.mangampl`
 
-
-2.5.3dev
+3.0.0dev
 --------
 
+ - Directory restructuring:  Removed "python" top-level directory and
+   moved "data" into "mangadap".  This should make package distribution
+   via pip easier.
+ - Remove "default" prefix from methods in `mangadap.config.defaults`.
+ - Remove obsolete `dapsrc` keyword arguments
+ - Some general clean-up of commented code and docs.
+ - Import clean-up, including removal of any `from __future__` imports.
+ - Added `bin/dap_status` to check the status of a batch DAP run.
+ - `__credits__` now include all authors of the DAP papers
+ - Added new DataCube and RowStackedSpectra classes, beginning the
+   generalization of the interface with the input data for use with
+   non-MaNGA data.
+ - For instantiating a MaNGADataCube, changed from using a yanny par
+   file to a configuration (ini) file.  Included code that can write
+   these files using data from the DRPall or DRPComplete files.
+ - Included a script that will download data into a new
+   `mangadap/data/remote` directory for testing.  The directory is not
+   included in the repo and has been added to .gitignore to prevent it
+   from being accidentally added.
+ - Included a number of tests that use the new remote data.  These will
+   be skipped if the remote data is not available.
+ - Significant improvements and testing of `mangadap.util.covariance`.
+   Ensuring that the correlation matrices provided by the DRP can be
+   read by `MaNGADataCube` and are effectively identical to the
+   calculation performed by `MaNGARSS`.
+ - Moved all core script code from `bin` to `mangadap/scripts`.  Code in
+   `bin` now make simple calls to these scripts.  Moved `rundap.py` from
+   `mangadap.survey` to `mangadap.scripts`.
+ - General clean-up of the executables in `bin`.
+ - Integrated use of `MaNGADataCube` instead of `DRPFits` when executing
+   the pipeline.  Usage of `DRPFits` is now largely obsolete.  Not
+   removed yet.  Interface is still not abstracted enough for a general
+   `DataCube` (because of naming conventions), but getting there.
+ - Tests now include a nominal run of `manga_dap` using the `ALL`
+   binning scheme.
+ - Usage of `ObsInputPar` is now obsolete.
+ - Spectral resolution is no longer chosen by the spectral binning
+   module; instead the spectral resolution is selected when reading the
+   datacube (makes much more sense!).  Led to some clean-up of the
+   binning config files.
+ - To select different spectral resolution extensions, use command-line
+   arguments in `rundap` or `write_dap_config`.
+ - Started doc edits.
+ - Example scripts now use `MaNGADataCube` instead of `DRPFits`, but
+   further clean-up of the `examples` directory is needed.
+ - Include a default analysis plan, so that executions of `manga_dap`
+   don't require an analysis plan yanny parameter file.
+ - Docstring updates for modules up through `StellarContinuumModel`, but
+   still lots to do.
+ - Consolidates the common functionality in `MaNGADataCube` and
+   `MaNGARSS` into a new, repurposed `DRPFits` base class.
+ - Global data from unique DAPTYPEs now put in separate extensions of
+   the DAPall file.
+ - Incorporate change from 'PREDISP' to 'LSFPRE' and 'DISP' to 'LSFPOST'
+   in the spectral resolution extensions produced by the DRP.
+ - Added DataCube metadata.
+ - Added Legendre polynomial fitter
+ - Added MaStar HC V2 library
+ - Include additional spectral index measurements and weights to use
+   when combining indices; leads to changes in MAPS data model.
+ - Updated versions of required packages, including incrementing version
+   dependency to ppxf==7.2.0; code still uses default `linear_method`
+   pending testing of `linear_method='lsqlin'`.
+ - Added photometry fix file; only adjusts 8083-12702 pending further
+   assessments.
+ - Added redshift fixes for IC342, Coma, and globular cluster cubes.
+ - Fixed minor bug in coordinates used in HYB binning cases for some
+   DAPall calculations.
+
+
 2.5.2 (14 Feb 2020)
 -------------------
 

MANIFEST.in

@@ -0,0 +1,10 @@
+include *.md
+include setup.cfg
+
+recursive-include mangadap/data *.fits*
+recursive-include mangadap/data *.par
+recursive-include mangadap/data README
+recursive-include python/mangadap/config *.ini
+
+global-exclude *.pyc *.o
+

bin/construct_dapall

@@ -1,65 +1,7 @@
 #!/usr/bin/env python3
 
-from __future__ import print_function
-from __future__ import division
-from __future__ import absolute_import
-from __future__ import unicode_literals
-
-import os
-import time
-
-from argparse import ArgumentParser
-
-import numpy
-
-from mangadap.util.log import init_DAP_logging, module_logging
-from mangadap.par.analysisplan import AnalysisPlanSet
-from mangadap.survey.dapall import DAPall
-
-#-----------------------------------------------------------------------------
-
-#-----------------------------------------------------------------------------
+from mangadap.scripts import construct_dapall
 
 if __name__ == '__main__':
-    t = time.perf_counter()
-
-    parser = ArgumentParser()
-
-    parser.add_argument('plan', type=str, help='SDSS parameter file with analysis plan')
-
-    parser.add_argument('--drpver', type=str, help='DRP version', default=None)
-    parser.add_argument('-r', '--redux_path', type=str,
-                        help='Top-level directory with the DRP products; defaults to '
-                             '$MANGA_SPECTRO_REDUX/$MANGADRP_VER', default=None)
-    parser.add_argument('--dapver', type=str, help='DAP version', default=None)
-    parser.add_argument('-s', '--dap_src', type=str,
-                        help='Top-level directory with the DAP source code; defaults to '
-                            '$MANGADAP_DIR', default=None)
-    parser.add_argument('-a', '--analysis_path', type=str,
-                        help='Top-level output directory for the DAP results; defaults to '
-                             '$MANGA_SPECTRO_ANALYSIS/$MANGADRP_VER/$MANGADAP_VER', default=None)
-    parser.add_argument('-m', '--methods', type=str, nargs='+', default=None,
-                        help='Only include output from this DAP method designation in the output')
-    parser.add_argument('-v', '--verbose', action='count',
-                        help='Set verbosity level; can be omitted and set up to -vv', default=0)
-    parser.add_argument('--quiet', action='store_true', default=False,
-                        help='suppress all terminal output')
-    parser.add_argument('--double', dest='single_precision', action='store_false', default=True,
-                        help='Output the floating-point data in double precision '
-                             '(default is single precision)')
-
-    arg = parser.parse_args()
-    analysisplan = AnalysisPlanSet.from_par_file(arg.plan)
-
-    # Initialize the logging objects and start the log
-    init_DAP_logging(None)#, simple_warnings=False)
-    loggers = module_logging(__name__, arg.verbose)
-
-    DAPall(analysisplan, methods=arg.methods, drpver=arg.drpver, redux_path=arg.redux_path,
-           dapver=arg.dapver, dapsrc=arg.dap_src, analysis_path=arg.analysis_path,
-           loggers=loggers, quiet=arg.quiet, single_precision=arg.single_precision)
-
-    print('Elapsed time: {0} seconds'.format(time.perf_counter() - t))
-
-
+    construct_dapall.main(construct_dapall.parse_args())
 

bin/dap_calculate_covariance

@@ -0,0 +1,6 @@
+#!/usr/bin/env python3
+
+from mangadap.scripts import calculate_covariance
+
+if __name__ == '__main__':
+    calculate_covariance.main(calculate_covariance.parse_args())

bin/dap_find_repeat_observations

@@ -0,0 +1,6 @@
+#!/usr/bin/env python3
+
+from mangadap.scripts import find_repeat_observations
+
+if __name__ == '__main__':
+    find_repeat_observations.main(find_repeat_observations.parse_args())