more fringe-related doc updates

docs/commands.rst

@@ -29,7 +29,8 @@
 .. |ol-ltrans|   replace:: computes transforms to align frames
 .. |ol-makebias| replace:: combine a run to make a bias frame
 .. |ol-makedark| replace:: combine a run to make a dark frame
-.. |ol-makeflat| replace:: combine a list of frames into a flat
+.. |ol-makeflat| replace:: combine a set of frames into a flat
+.. |ol-makefringe| replace:: combine a set frames into a fringe map
 .. |ol-makemovie| replace:: makes stills for movies from a run
 .. |ol-mstats|   replace:: list stats of multiple frames from a run
 .. |ol-mul|      replace:: multiply two frames
@@ -41,6 +42,7 @@
 .. |ol-rtplot|   replace:: plot frames as they come in [pgplot]
 .. |ol-setaper|  replace:: define the photometric apertures
 .. |ol-setdefect| replace:: define a file of CCD defects
+.. |ol-setfringe| replace:: define peak/trough pairs for fringe measurement
 .. |ol-splice|   replace:: splice two frames together
 .. |ol-stats|    replace:: report statistics of a frame
 .. |ol-sub|      replace:: subtract two frames
@@ -128,6 +130,8 @@ useful.
    +--------------+----------------+----------+----------+---------+-----------+------------+
    | |makeflat|   | |ol-makeflat|  |          | Yes      |         |           |            |
    +--------------+----------------+----------+----------+---------+-----------+------------+
+   | |makefringe| | |ol-makefringe|| Yes      | Yes      |         |           |            |
+   +--------------+----------------+----------+----------+---------+-----------+------------+
    | |makemovie|  | |ol-makemovie| |          |          | Yes     |           | Yes        |
    +--------------+----------------+----------+----------+---------+-----------+------------+
    | |mstats|     | |ol-mstats|    |          |          |         |           | Yes        |
@@ -150,6 +154,8 @@ useful.
    +--------------+----------------+----------+----------+---------+-----------+------------+
    | |setdefect|  | |ol-setdefect| | Yes      |          |         |           |            |
    +--------------+----------------+----------+----------+---------+-----------+------------+
+   | |setfringe|  | |ol-setfringe| | Yes      | Yes      |         |           |            |
+   +--------------+----------------+----------+----------+---------+-----------+------------+
    | |splice|     | |ol-splice|    |          |          |         |   Yes     |            |
    +--------------+----------------+----------+----------+---------+-----------+------------+
    | |stats|      | |ol-stats|     |          |          |         |           | Yes        |
@@ -176,16 +182,17 @@ you get up to speed.
 Basic parameter input
 ---------------------
 
-The command ``rtplot`` has more parameters than most others and is a good one
-to start with. Suppose then that we have a raw |hiper| file,
-:file:`run0076.fits`, that we want to plot. If we type 'rtplot' and follow
-the prompts, the first few lines might be::
+The command |rtplot| has more parameters than most others and is a
+good one to start with (see also its replacement |nrtplot|). Suppose
+then that we have a raw |hiper| file, :file:`run0076.fits`, that we
+want to plot. If we type 'rtplot' and follow the prompts, the first
+few lines might be::
 
   rtplot
   run - run name [run0064]: run0076
   first - first frame to plot [10]: 1
 
-This shows that the last time ``rtplot`` was invoked, it was used to look
+This shows that the last time |rtplot| was invoked, it was used to look
 as :file:`run0064.fits` starting with frame 10. Note that the extension
 '.fits' is not needed: |hiper| makes significant use of extensions to
 differentiate between different forms of file, all of which could be
@@ -208,7 +215,7 @@ will do, and the first two parameters could be similarly specified::
   rtplot first=1 run=run0076
 
 Note that by naming the parameters, the order becomes immaterial. Now,
-assuming that the command was completed, if you run ``rtplot`` again you
+assuming that the command was completed, if you run |rtplot| again you
 might get::
 
   rtplot
@@ -337,8 +344,8 @@ Global vs local parameters
 
 All |hiper| pipeline parameters fall into one of two classes, either being
 'local' to a command or 'global' to multiple commands. The ``run`` parameter
-of ``rtplot`` for instance also appears in ``grab`` and if you change it in
-``rtplot``, it will be changed in ``grab``. This is very useful when running
+of |rtplot| for instance also appears in |grab| and if you change it in
+|rtplot|, it will be changed in |grab|. This is very useful when running
 a series of commands on the same file as the commands almost 'know' what you want,
 saving much typing.
 
@@ -452,6 +459,7 @@ extension '.hcm' to distinguish them, although they are also FITS-format files.
 .. autofunction:: hipercam.scripts.makebias
 .. autofunction:: hipercam.scripts.makedark
 .. autofunction:: hipercam.scripts.makeflat
+.. autofunction:: hipercam.scripts.makefringe
 .. autofunction:: hipercam.scripts.makemovie
 .. autofunction:: hipercam.scripts.mstats
 .. autofunction:: hipercam.scripts.mul
@@ -463,6 +471,7 @@ extension '.hcm' to distinguish them, although they are also FITS-format files.
 .. autofunction:: hipercam.scripts.rupdate
 .. autofunction:: hipercam.scripts.setaper
 .. autofunction:: hipercam.scripts.setdefect
+.. autofunction:: hipercam.scripts.setfringe
 .. autofunction:: hipercam.scripts.splice
 .. autofunction:: hipercam.scripts.stats
 .. autofunction:: hipercam.scripts.sub

docs/files.rst

@@ -17,7 +17,7 @@ The |hiper| CCDs have a number of defects, dust and the like, and, as
 of 13 May 2021, a hair in the lower-left quadrant of CCD 4 and
 something similar in the upper-right quadrant of CCD 2. When acquiring
 targets these may not be visible so you are **very strongly advised**
-to plot a file of defects. You can also create your own using the
+to over plot defects. You can also create your own using the
 pipeline command |setdefect| and you are encouraged to do so if you
 see a defect not already marked in whatever file you use.
 
@@ -30,7 +30,7 @@ I have decided to define defects according to the following prescription:
 ``moderate``: 10 to 25% deviation from the norm; ``severe``: greater than 25%
 from the norm. I mark each pixel in these categories, so a really bad feature
 might appear in multiple pixels (there is a very nasty feature in the
-lower-left quadrant of CCD 4 for example). I plan to update
+lower-left quadrant of |hiper|'s CCD 4 for example). I plan to update
 the file with time, and retain old ones for monitoring purposes.  I
 identified the bad pixels using a flat field divided by a smoothed version of
 itself (smoothed with a 40 pixel FWHM 2D gaussian filter).
@@ -65,10 +65,11 @@ Files:
 Fringe maps and peak/trough pairs
 =================================
 
-Here are some pre-prepared files for de-fringing. There are two types,
-fringe maps as generated by |makefringe| (these are otherwise standard
-hcm files), and sets of peak/trough pairs as generated by |setfringe|
-with extension ".frng".
+Two types of files are needed to implement defringing.  Fringe maps as
+generated by |makefringe| (these are otherwise standard hcm files),
+and sets of peak/trough pairs as generated by |setfringe| with
+extension ".frng". Here are some pre-prepared examples to get you
+going.
 
   #. 2018-04-15 (:download:`hipercam z-band fringe map
      <hipercam_fmap_2018_04_15.hcm>`)

docs/organisation.rst

@@ -2,8 +2,8 @@
 
 .. include:: globals.rst
 
-Organisation
-************
+File organisation
+*****************
 
 This page contains advice on how to organise |hiper| reduction in terms of
 directory structure and file names.
@@ -84,27 +84,36 @@ available. Then a name like 'flat_ugr_caution.hcm' might be warranted.
 Reduction file names
 ====================
 
-Once all set with calibrations, then for a given run, I use the root name
-('run023' or 'run0034' or whatever) for all files associated with that
-run. The pipeline defines a set of standard extensions that will be added to
-any given file to allow this. Thus you will end up with multiple files of the
-form 'run023.XXX', with 'XXX' being the file extension. A full set of such
-files and their likely meanings are:
+Once all set with calibrations, then for a given run, I use the root
+name ('run023' or 'run0034' or whatever) for all files associated with
+that run. The pipeline defines a set of standard extensions that will
+be added to any given file to allow this. Thus you will end up with
+multiple files of the form 'run023.XXX', with 'XXX' being the file
+extension. This makes it much easier to keep track of what a given
+file is associated with. Many of the pipeline scripts are designed
+with this in mind, e.g. |averun| will set the default output name
+according to the run name. A full set of such files and their likely
+meanings (including a few ancilliary files not associated with
+specific runs) are:
 
 ===========  =========== =======================================================================
 Name         Type        Meaning
 ===========  =========== =======================================================================
 run023.hcm   Binary FITS Representative image of the run, probably from its start using |averun|
-run023.ape   JSON text   File of photometric apertures (JSON text file)
+run023.ape   JSON text   File of photometric apertures
 run023.red   ASCII text  Reduction driver file for |reduce|
 run023.log   ASCII text  Output log file from a run of |reduce|
 run023.fits  Binary FITS FITS version of a |reduce| log file made with |hlog2fits|
+defect.dft   JSON text   File of CCD defects
+fringe.frng  JSON text   File of peak/trough pairs for fringe measurement
+files.lis    ASCII text  Column of file names (source='hf' option in some commands)
 ===========  =========== =======================================================================
 
 If you find such a set, within a directory, then assuming the pipeline version
 used to make the log file still applies, you should be able to re-reduce using
 the aperture ('.ape') and reduction ('.red') files. A run of |setaper| with
 the hcm ('.hcm') and ('.ape') files will show the apertures selected. If the
 reduction fails, then you may want to re-generate the reduction driver file
-using |genred|.
+using |genred|; see also |rupdate| which attempts to bring old reduction files
+up to date.
 

hipercam/scripts/makefringe.py

@@ -24,7 +24,8 @@ def makefringe(args=None):
     """``makefringe [source] (run first last [twait tmax] | flist) (bias
     flat dark) fpair ([nhalf]) ccd fwhm [clobber] output``
 
-    Averages a set of images to make a frame for defringing.
+    Averages a set of images to make a frame for defringing (referred
+    to elsewhere as a "fringe map").
 
     At long wavelengths, CCDs suffer what is known as "fringing",
     which in terms of structure looks something like the coloured
@@ -34,16 +35,25 @@ def makefringe(args=None):
     additive in nature.
 
     De-fringing in |hiper| is implemented according to the method
-    suggested by Snodgrass & Carry (2013Msngr.152...14S). The idea
-    here is to create a set of pairs of points the ratios of which
-    will be used to estimate the level of fringing compared to a
-    reference frame. ``makefringe'' is used to make the reference
-    frame. It works as follows: given an input list of files (or
-    optionally a single run), it reads them all in, debiases,
+    suggested by Snodgrass & Carry (2013Msngr.152...14S). The idea is
+    to create a set of pairs of points marking the peak and troughs of
+    fringes. The difference in intensity between these in the data to
+    be corrected is compared with the difference in a reference
+    "fringe map" by taking their ratio. This is done many times so
+    that the median can be taken to eliminate ratios affected by
+    celestial targets or cosmic rays. the ratios of which will be used
+    to estimate the level of fringing compared to a reference
+    frame. ``makefringe`` is used to make the reference fringe map. It
+    works as follows: given an input list of files (or optionally a
+    single run), it reads them all in, optionally debiases,
     dark-subtracts and flat-fields them, calculates a median count
-    level of each one which is subtracted from and (optionally)
-    divided into each frame individually. The pixel-by-pixel median of
-    all frames is then calculated.
+    level of each one which is subtracted from each CCD
+    individually. The pixel-by-pixel median of all frames is then
+    calculated. It is also possible to apply the fringe pair ratio
+    measurement to scale each frame before combining which allows
+    frames of similar pattern but varying amplitude to be
+    combined. Finally, the output can be smoothed because fringes are
+    usually a medium to large scale pattern.
 
     Parameters: