/
sun139.tex
19546 lines (17766 loc) · 804 KB
/
sun139.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\documentclass[twoside,11pt]{starlink}
% ? Specify used packages
% ? End of specify used packages
% -----------------------------------------------------------------------------
% ? Document identification
% Fixed part
\stardoccategory {Starlink User Note}
\stardocinitials {SUN}
\stardocsource {sun\stardocnumber}
% Variable part - replace [xxx] as appropriate.
\stardocnumber {139.18}
\stardocauthors {Peter W. Draper, Mark Taylor, Alasdair Allan}
\stardocdate {7 July 2011}
\stardoctitle {CCDPACK -- CCD data reduction package}
\stardocversion {Version 4.0}
\stardocmanual {Users' Manual}
\stardocabstract {
CCDPACK is a package of programs for reducing CCD-like data. They
allow you to debias, remove dark current, flatfield, register,
resample and normalize data from single- or multiple-CCD instruments.
CCDPACK is designed to help you to reduce your data easily.
The basic reduction stages can be set up using an X based GUI that
controls an automated reduction system.
The GUI is designed to allow you to start working without any
detailed knowledge of the package (or indeed of CCD reduction).
Registration is performed using graphical, script based or automated
techniques that help reduce the amount of work to a minimum.
This document is intended for all users of CCDPACK.
It provides instruction in how to use CCDPACK, describes CCD reduction
in general and contains complete descriptions of the individual
programs.
}
% ? End of document identification
% -----------------------------------------------------------------------------
% ? Document specific \providecommand or \newenvironment commands.
% Size of \tt text - looks better smaller in LaTeX but not HTML.
\providecommand{\ttsize}{\latexhtml{\small}{}}
% Environment for indenting and using a small font.
\newenvironment{myquote}{\begin{quote}}{\end{quote}}
% In-line typed text, buttons and menu items.
\providecommand{\butt}[1]{{\ttsize \textbf{\texttt{#1}}}}
\providecommand{\menu}[1]{{\ttsize \textbf{\emph{#1}}}}
\providecommand{\wlab}[1]{{\ttsize \textbf{#1}}}
\renewcommand{\text}[1]{\texttt{#1}}
% Quick routine descriptions
\providecommand{\quickdes}[3]{
\parbox{1.1in}{\textbf{#1}}
\parbox{4.4in}{\raggedright #2 \dotfill}
\parbox{0.6in}{\pageref{#3}}
\vspace*{0.2in}}
% Quotes for SST.
\providecommand{\qt}[1]{\texttt{"}#1\texttt{"}}
\providecommand{\qs}[1]{\texttt{'}#1\texttt{'}}
% Routines with descriptions in the appendix.
\providecommand{\routine}[1]{\textsc{#1}}
\providecommand{\xroutine}[1]{\htmlref{\textsc{#1}}{#1}}
% Reference to a section of this document.
\providecommand{\secref}[2]{\latexhtml{Section \ref{#1} ``#2''}{``\htmlref{#2}{#1}''}}
% Latex only sections, subsections etc. Surround these with a latexonly
% environment.
\providecommand{\latexonlysection}[1]{\section{#1}}
% ? End of document specific commands
% -----------------------------------------------------------------------------
% Title Page.
% ===========
\begin{document}
\scfrontmatter
\section{\xlabel{introduction}Introduction}
CCDPACK is a package of programs for reducing CCD-like data. They
allow you to debias, remove dark current, pre-flash, flatfield,
register, resample, normalize and combine your data.
Using CCDPACK it is possible to automatically reduce CCD data (as far
as the flatfielding stage). This uses a scheduling system that only
requires knowledge of the frame types (bias, flatfield etc.) and
important CCD geometric features.
Using this information it can decide how to reduce your data and may
then run the necessary programs.
The frame types and detector characteristics can be obtained from FITS
headers, for certain telescopes/CCDs, so your job could be reduced to
identifying the telescope/detector used and the frames you want
reducing.
The automated reduction system can be controlled from an X based GUI
(Graphical User Interface) that has been specifically designed to
help novice and/or occasional users of CCD data (although it is
expected to appeal to the more experienced as well).
The necessary ease of use is achieved by limiting what could
potentially be a large range of options to those of immediate concern,
by providing a selection of known detectors and by having a context
sensitive help system.
It also aims to be complete by allowing you to define the necessary
geometric characteristics of your data interactively (if they cannot
be obtained elsewhere).
For those who prefer it an equivalent command-line interface is also
available.
The core of CCDPACK is a suite of programs that have been designed to
help in processing large amounts of data. Consequently {\em all\/}
CCDPACK routines process {\em lists\/} of data (these can be identified
using wildcards, \text{*,?,[a-z],\{X,Y,Z\}}, or from lists contained in
text files) and also record the progress of a reduction using an
integral log system.
As well as performing the usual instrumental corrections you can
also do defect removal (using keyword descriptions stored in a text file)
and generate and propagate data errors (these are derived from a knowledge
of the detector noise and the Poissonian nature of the detected
electron count).
Debiassing can be performed using only the bias strips as well as
using bias frames (these are combined to reduce noise levels).
Calibration data can be combined using many different techniques
(mean, median, trimmed mean etc.), so that you can pick a method that
makes most efficient use of your data.
Data registration (the determination of geometric transformations that
map the same positions in each of your data frames) is primarily based
on the {\em linear} transformation (this allows offsets, scalings,
rotation and shear), although more general transformations can be
used.
General linear transforms can be easily determined using an
interactive procedure for displaying and selecting image features.
Alternatively if your datasets are just shifted (offset) with respect
to each other, then you may be able to register them by using a series
of commands which locate all the objects in all the frames,
determine the object-object correspondence and then derive the
transforms.
A graphical application is also provided (for offset frames) that allows
you to select the objects to be used by identifying image pairs that
overlap and have some objects in common.
Other facilities are provided for using external information about
alignment of images; these may be of particular use for mosaic
cameras, or when pointing information is available in FITS headers
of the data products being reduced.
Data resampling uses World Coordinate System information stored
within the data, which removes the need to remember the
appropriate transformations.
As well as a wide selection of general purpose combination methods
such as median estimators, the more specialized `drizzling'
algorithm is also available.
Normalisation and combination (often also called mosaicing)
are done in a single step, which is
designed to deal with very large datasets.
This uses robust methods for determining scale and/or zero point
corrections.
To help when processing images from a multiple-CCD mosaic camera,
or a CCD with readout from multiple amplifiers,
this version of CCDPACK introduces the concept of a
Set of data-files: this makes it easy to match
corresponding data and calibration frames for reduction tasks,
and to keep track of known alignments during registration.
\section{\xlabel{startup}Starting up CCDPACK}
CCDPACK commands are made available from the C shell using the command:
\begin{terminalv}
% ccdpack
\end{terminalv}
(note that the \text{\%} represents the C-shell prompt and shouldn't be typed).
CCDPACK commands can also be used from the Starlink/\xref{ICL}{sg5}{}
and IRAF/CL command languages (IRAF users should consult section
\S\ref{IRAFNOTES}).
\subsection{\xlabel{gettinghelp}Getting Help}
Help is available in two forms; from the command-line and via a
hypertext version of this document.
Just type:
\begin{terminalv}
% ccdhelp
\end{terminalv}
to get the command-line version, and:
\begin{terminalv}
% ccdwww
\end{terminalv}
to load the hypertext version. The hypertext version has one optional
parameter, the name of the WWW browser to use. The default is
\htmladdnormallink{the Netscape Navigator}{http://home.netscape.com/}
although
\htmladdnormallink{NCSA Mosaic}
{http://www.ncsa.uiuc.edu/SDG/Software/Mosaic/Docs/help-about.html}
can also be used. If you prefer Mosaic try the command:
\begin{terminalv}
% ccdwww Mosaic
\end{terminalv}
If this doesn't work then replace the word \text{Mosaic} with the
command that starts the browser for you (this shouldn't be an alias).
If you want to always use Mosaic then set the environment variable
\text{HTX\_BROWSER} to \text{Mosaic} in your \text{.login} file.
Help is also available in the reduction GUI menus or from program
prompts (by responding with a `?').
If you come across any bugs or problems when using CCDPACK then e-mail
a description to: \texttt{starlink@jiscmail.ac.uk}.
\subsection{Running from the C-shell}
When using CCDPACK from the C-shell (or any other UNIX shell) care needs to
be taken with some special characters.
Wildcard characters \text{*,?,[a-z],\{X,Y,Z\}}, quoted strings \text{""}
and vector braces \text{[ ]} must be protected by using either the
escape character {\verb+\+} or by single quotes (wildcard characters
must be protected as these are expanded internally by CCDPACK, rather than by
the shell).
So for instance to pass the names of several data frames and the extent
of the useful part of a CCD you might use a command like:
\begin{terminalv}
% debias in='"datar*,ffr?"' extent='[51,1094,1,1024]'
\end{terminalv}
If in doubt, it is a good idea to put single quotes around arguments
in this way -- it can't hurt.
\section{\xlabel{demos}\label{demos}Demonstrations and test data}
If you want to get a feel for what CCDPACK can do before reading
further, you might like to run one or more of the three
demonstration scripts:
\begin{description}
\item[\texttt{ccdexercise}]
emphasises the data reduction aspects of CCDPACK (debiassing,
dark calibration, flatfielding)
\item[\texttt{wcsexercise}]
shows more of the advanced image registration capabilities
\item[\texttt{setexercise}]
organises data frames into Sets as if obtained from a mosaic camera
and demonstrates CCDPACK's Set-handling abilities
\end{description}
To run any of these scripts, change to an empty directory,
start up CCDPACK:
\begin{terminalv}
% ccdpack
\end{terminalv}
and then type the name of the demonstration script you want to see,
either
\begin{terminalv}
% ccdexercise
\end{terminalv}
or
\begin{terminalv}
% wcsexercise
\end{terminalv}
or
\begin{terminalv}
% setexercise
\end{terminalv}
You will then be prompted for a graphics device:
answer `xw' to see the progress graphically on an Xwindows display,
or `!' to run without graphics.
Each of these scripts generates test data files (small fictional
starfields) and various intermediate files in the current directory
when it runs. These can be deleted when the script has finished.
However, you may find them useful as test data if you want to
see how the scripts have operated, or to experiment with the CCDPACK
commands. You can also use the scripts themselves as examples or
templates for reducing your own data.
\section{Basic principles of CCD data reduction\xlabel{CCDprinciples}}
The primary aim of CCD data reduction is to remove any effects that are
due to the nature of the detector and telescope -- the `instrumental
signature'.
This is so that measurements (of intensity and possibly error) can be
made that do not require any knowledge about how the data was taken.
CCD data requires several corrections to attain this state.
The most fundermental of these corrections is the `bias level' subtraction.
The bias level is an electronic offset added to the signal from the
CCD that makes sure that the Analogue-to-Digital Converter (ADC)
always receives a positive value.
The ADC is responsible for converting the signal representing
the amount of charge accumulated in a CCD pixel to a digital value.
A pixel is one of the discrete elements of the CCD where electrons
accumulate during exposure to a light source (you see these as a
picture element on your image display).
The bias level has an intrinsic noise (induced by the signal
amplification) known as the `readout noise' (this one of the features
which limits the usefulness of CCDs at very low light levels).
Usually the bias level is removed by the subtraction of specifically
taken `bias-frames' ($0$ second exposure readouts) or by using
estimates derived from bias data that is added in regions around
the real data. These regions are known as the bias strips or
over/under-scan regions (see \slhyperref{this figure}{Figure
\S}{}{CCDPICCY}).
After bias subtraction the data values are now directly related to the
number of photons detected in each CCD pixel.
The relation between the units of the data and the number of photons
is a scale factor known as the gain.
In this document the gain factor is referred to as the ADC factor.
The units of CCD data before being multiplied by the ADC factor are
known as ADUs (Analogue-to-Digital Units).
CCD data when calibrated in electrons has a Poissonian noise distribution
(if you exclude the readout noise).
Other corrections which are occasionally made to CCD data are
dark count subtraction and pre-flash subtraction.
These are only usual in older CCD data (but for IR array data the dark
current correction is essential).
Dark correction is the subtraction of the electron count which
accumulates in each pixel due to thermal noise.
Modern CCDs usually have dark counts of less than a few ADUs per pixel
per hour, so this correction can generally be ignored.
Pre-flashing of CCDs has been used to stop loss of signal in CCDs with
poor across-chip charge transfer characteristics, the reasoning being
that if signal is entered in a pixel before the main exposure, then
subsequent losses are less likely to affect the data --- note however
that this also means that a higher signal to noise level is required
for detection.
The final stage in the correction of CCD data for instrument signature
is `flatfielding'.
The sensitivity of CCDs varies from point to point (i.e.\ the recorded
signal per unit of incident flux -- photons -- is not uniform), so if
the data is to be relatively flux calibrated (so that comparison from
point to point can be made) this sensitivity variation must be
removed.
To achieve this correction exposures of a photometrically flat source
must be taken, these are known as flatfields.
The basic idea of flatfield correction is then to divide the data by a
`sensitivity map' created from the calibrations, although in real life
noise considerations, together with others (see appendix \ref{app:glos}),
mean that particular care needs to be taken at this stage.
After all these corrections have been made your data is
usually\footnote{Usually because another correction may also be
necessary -- the removal of fringing see appendix \ref{app:glos}.}
ready for analysis.
Other processes which are frequently undertaken before analysis are
registration, alignment, normalisation and combination. Registration is
the process of determining the transformations which map the same
positions on different datasets. This is essential if measurements, say
with different filters, are to be made. In this case registration may be
informal and just consists of identifying the same objects on different
datasets. However, very accurate measures are often also required;
certainly this is the case when data combination is to be performed.
`Data combination' is just when aligned datasets are combined by a
process of taking the mean or some other estimator at each pixel, this
is also frequently referred to as `mosaicing'. Aligning datasets
means achieving pixel-to-pixel correspondence (in real data it is
unlikely that this state is true, even if it was intended).
Alignment uses the registering transforms to `resample' the data onto
a new pixel grid. If the exposure times, atmospheric transparency or
sky brightness have varied, then data must be `normalised' before
combination. Normalisation is the determination of the zero points and
scale factors which correct for these changes.
\section{How to reduce your data now}
If you're now wondering how you should go about reducing your data and
don't want to read this very long document, then this is the place to
start. Later sections are rather more technical or describe data
registration and related topics and should really only be consulted
when the need arises.
\subsection{Using the CCDPACK data reduction GUI
\label{usingxreduce}\xlabel{usingxreduce}}
If you are sitting at an X display then type the commands:
\begin{terminalv}
% ccdpack
% xreduce &
\end{terminalv}
These commands will start the CCDPACK GUI. The GUI will lead you
through the stages necessary for organising your data so that it can
be scheduled for processing and will start a background job to do the
reduction. The GUI has a context sensitive help system, so pull down
the \menu{Help} menu and choose \menu{On Window} to get information
about your immediate concerns (the help will actually appear in a
hypertext browser -- netscape by default). Hopefully this should be
enough to get you started but read the next few paragraphs if you have
time.
The underlying purpose of \xroutine{XREDUCE} is to get sufficient
information gathered about your data so that it can plan a reduction
schedule. It can do this in one of two ways. Either you can select
from a list of known telescope/detector combinations, or enter
all the information required yourself (with lots of help of course).
\routine{XREDUCE} just really needs to know what package options to
use, where a few geometric features of the CCD are (if needed) and the
types of the various input frames (it needs to know which frames are
bias, flatfields, darks and the real astronomical ones, called
``target frames'').
If you're fortunate then your data will contain a description of
itself in a form that \routine{XREDUCE} can understand. In this case
all you need to do is identify the detector and then go on to list all
the frames you want to process. If your data is from a detector with
limited information available (such as the geometries of the various
parts of the CCD) then you can select this and all you need to do is
then inform \routine{XREDUCE} which of the input frames are which frame
type. Check the known detectors under the \menu{Options} menu in the
main window.
If neither of these options is available to you then all isn't lost as
configuring \routine{XREDUCE} is easy. All you need to do is set some
general options (press button \butt{General Options}), define the CCD
characteristics (there's graphical help available if you need to
define any of the CCD geometries, check under \menu{Options} in the
\butt{CCD Characteristics} window) and then go on to organize your
frames into their types (button \butt{Manual Organization}). To set
the reduction running look under \butt{Setup and Run}.
No matter how you inform \routine{XREDUCE} about your data you still
have to follow pretty similar routes:
\begin{enumerate}
\item set up the package (probably using both the \butt{General Options}
and \butt{CCD Characteristics} windows)
\item ``import'' your data frames (probably using one of the
\butt{Manual Organization} or \butt{Using FITS Headers} windows)
\item say how you want to debias your data (this is done in the
\butt{Setup and Run} window, where your options will be restricted
to those possible).
\end{enumerate}
Debiassing is done by two basic methods, either by subtracting bias frames
or using the bias strips. If you have bias frames then use them, if
you don't look for dark strips down the sides of your data; these are
the bias strips (have a look at \slhyperref{this figure}{Figure
\S}{}{CCDPICCY}).
If you have bias frames and bias strips then use both (this is the
\text{zeroed master, offsetting to bias strips} option).
The reduction is run independently of the GUI and its output is kept
in a log file \text{xreduce.log}, so exit \routine{XREDUCE} when you
are finished (this is after you have accepted the options under
\butt{Setup and Run}) and the reduction will continue.
One final piece of information concerns the nature of the ``data
frames'' as they have been called so far. These may be of any kind (as
described \slhyperref{later}{in section \S}{}{datatypes}) providing you
have initialized the \xref{CONVERT}{sun55}{} package before starting
up \routine{XREDUCE}. If you have not initialized CONVERT then you
will only be able to use data frames that are stored in the Starlink
\xref{NDF}{sun33}{} format and their names will consequently have a
file type of `\text{.sdf}'. When using these names in the GUI do
include the `\text{.sdf}' extension, this is contrary to how you would
deal with these names in ``normal'' NDF processing programs where you
would not include the file type.
\subsubsection{GUI problems}
If you have a ungraceful exit from \xroutine{XREDUCE} you may have
some CCDPACK and \xref{KAPPA}{sun95}{} processes left running. You
will now have to kill these by hand. How you do this is system
dependent. On BSD type systems (like Digital UNIX and Linux) try:
\begin{terminalv}
% ps ux
\end{terminalv}
and on system V machines (Solaris) try:
\begin{terminalv}
% ps -ef | grep $USER
\end{terminalv}
and look for the processes named \text{ccdpack\_res} and
\text{kapview\_mon} (and perhaps others, such as
\text{ndfpack\_mon}).
Get rid of these using the \text{kill} command followed by the
process identifiers (usually the number in the second column).
\subsection{Command-line reduction}
There is a command-line interface with similar abilities to
\xroutine{XREDUCE} if you do not have access to an X display (or prefer
a command-line anyway) it is started using the command:
\begin{terminalv}
% reduce
\end{terminalv}
\xroutine{REDUCE} it isn't quite as capable as \routine{XREDUCE} so
you'll have to work a bit harder. Consult the description in the
appendix for a little more about this routine, but it's probably
worth trying it out and failing first.
\section{Data\label{datatypes}\xlabel{datatypes} formats}
The ``native'' data format that CCDPACK uses is the Starlink NDF (the
N-dimensional data format (\xref{SUN/33}{sun33}{}), which is based on
HDS, the Hierarchical Data System \xref{(SUN/92)}{sun92}{}). This is
portable between the operating systems that CCDPACK runs on (Digital
UNIX, Solaris and Linux at this time) so it can be copied, accessed
via NFS and ftp'd (using binary transfer) between these systems.
If you have your data stored on tapes in FITS format then you can use
the \xref{KAPPA}{sun95}{} application \xref{FITSIN}{sun95}{FITSIN} to
read them into NDFs. You can also get a complete list of all the FITS
headers of all the files on a tape (or list of files) using the KAPPA
application \xref{FITSHEAD}{sun95}{FITSHEAD}.
If you already have (or want to keep) your data in another
astronomical format such as IRAF, disk FITS or old FIGARO then CCDPACK
can also process these, but with an additional processing overhead. To
use data in these formats you should initialise the CONVERT package
(\xref{SUN/55}{sun55}{}):
\begin{terminalv}
% convert
\end{terminalv}
and the necessary facilities will be set up.
Having done this, the simplest way to proceed is to use the
full name of the unconverted files (i.e.\ including an extension
such as `.fits') when giving filenames to CCDPACK; it will
transparently convert them to NDF format as required without
any further effort from you.
An alternative is to convert the files explicitly to NDF format
before using CCDPACK commands on them.
Depending on how you are using the commands, this may save processing
time by preventing CCDPACK from doing the same conversion more than once.
A full description of how to do this is given in \xref{SUN/55}{sun55}{},
but normally it just consists of running a command called `$<$TYPE$>$2NDF'.
For instance converting all the FITS files in the current directory
to NDF files with the same name can be done like this:
\begin{terminalv}
% fits2ndf in='*.fits' out='*'
\end{terminalv}
One useful tip for \xref{FITS2NDF}{sun55}{FITS2NDF}
is to set the CONTAINER parameter to true if you are using Multi-Extension
FITS files (MEFs), i.e.\ type instead
\begin{terminalv}
% fits2ndf in='*.fits' out='*' container=true
\end{terminalv}
This will convert an MEF into a single HDS container file
containing each of the HDUs in the MEF --- the upshot of this is that
you can pass a single file name to CCDPACK programs and it will
process all the images contained therein.
It may also make Set processing (see section \ref{ccdpack_sets}) easier.
One point that you should take note of is that not all formats are as
flexible as NDF and cannot therefore store all the information that
can be generated by CCDPACK. In particular IRAF data files cannot
store additional data arrays such as variance and quality, so for
instance you cannot gain any useful information about how errors
propagate in your data. Also the extension information stored by
CCDPACK has to be converted into native headers, this makes it less
obvious what header information CCDPACK is using.
If you are unfortunate enough to have data in a format not supported by the
CONVERT package then you will need to consult \xref{SSN/20}{ssn20}{}
about how to proceed. You should bear in mind that the requirements
of CCDPACK are that your format supports the storage of an image and
some associated header information (this is essential for registration).
\section{Using the data reduction programs\xlabel{DIY}}
The following sections lead you through the CCD reduction process as
you would do it without the aid of \xroutine{XREDUCE} or
\xroutine{REDUCE}.
The initial part details how the automated processing works and the
later parts show you how to do the same things using the core programs.
\subsection{Using the automated processing facilities directly}
There are three stages that you must go though to use the automated
processing facilities:
\begin{enumerate}
\item Configure the package.
\item Record the reduction information in your data.
\item Schedule the reduction.
\end{enumerate}
\subsubsection{Package configuration\xlabel{configuration}
\label{configuration}}
Package configuration is performed using the \xroutine{CCDSETUP}
program.
This sets the values of a sequence of global parameters, that are used
by the other CCDPACK programs.
Starting with this routine serves as a useful reminder of what values
etc.\ will be required to perform the reduction.
Note, however, that none of the parameters are compulsory (indeed
\routine{CCDSETUP} itself is not compulsory) and may be returned as \text{`!'}
(this is the null-value symbol).
\routine{CCDSETUP} asks for the following values (together with some others
which are best accepted until more experience with the package is
gained):
\begin{itemize}
\item The ADC factor which converts the
ADUs of the input data frames into detected electrons (this also
used to generate errors).
\item The bias strip positions (used for offsetting the master bias
frame or for bias estimation).
\item The readout direction (defines bias strip positions and the
bias interpolation direction).
\item The typical readout-noise (also used to generate errors).
\item The useful CCD area.
\end{itemize}
The routine also initialises the CCDPACK logging system (see
\slhyperref{later}{\S}{}{logsystem}).
Which of this information you'll have to supply depends on whether or
not your data has the correct information stored with it.
CCDPACK can ``import'' FITS headers and use these to supply the
necessary information.
A typical CCD geometry is shown in Figure \ref{CCDPICCY}; note how the
bias strips and useful CCD areas are defined. The coordinate values
for these regions are always defined in pixel indices (i.e.\ row or
column numbers, usually starting at 1,1 for the lower left-hand corner
of the data array).
A typical set up command is:
\begin{terminalv}
% ccdsetup adc=1.5 bounds='[2,10,400,416]' rnoise=10 extent='[11,399,1,576]'
\end{terminalv}
In this example certain required characters (the \text{[ ]}) are also
special to the C shell so must be protected.
\xroutine{CCDSETUP} also allows you to define which parts of the CCD are
corrupt or unreliable (due to hot spots, bad columns etc.); see
\slhyperref{the section on data-masks}{\S}{}{datamasks}, if you need to do this.
\myfig{sun139_geo}{}{}{CCDPICCY}{Typical CCD geometries.}{Typical CCD geometries.
In the figure on the left the readout direction is `Y', the bias
strips are located with bounds I,J,K,L and the useful CCD area is
M,J$+1$,N,K$-1$ (ish, you should probably use more than $\pm 1$). In
the figure on the right the readout direction is `X', the bias strips
are located with bounds I,J,K,L and the useful CCD area is
N,J$+1$,K$-1$,M$-1$ (N.B. some observatories recommend that you only
use the left-hand strip, if you use the right-hand one too, check that
it isn't contaminated by residual charge).}
\subsubsection{Setting reduction information \label{settingreductioninfo}}
Before you can ask CCDPACK to schedule a reduction it is necessary to
put your data frames through a process known as ``importing''. What
this means is that all the necessary descriptions about the type of
data (target, bias, flat etc.), filter, where the CCD bias strips are
etc. are put into the data files (in a part known as the CCDPACK
extension; note that for non-NDF formats this means the image
headers).
There are two ways to enter this information into your data frame
extensions, use either the \xroutine{PRESENT} or \xroutine{IMPORT} programs.
If your data has the correct FITS headers then you use \routine{IMPORT}
to interpret these and if it has none (or the FITS headers do not
give a complete description) you use \routine{PRESENT}. Unless you have
an existing FITS ``import control table'' for your detector (some of these
are available with CCDPACK, check \xroutine{XREDUCE} or \xroutine{REDUCE}
for a list of these) using \routine{IMPORT}
isn't a trivial task and you should probably opt for using
\routine{PRESENT}, even if you have some FITS information available.
A description of how to create a FITS import control table is given in
the \routine{IMPORT} \slhyperref{description}{description in \S}{}{IMPORT}.
Using \routine{PRESENT} is fairly trivial as long as you've
run \routine{CCDSETUP} and answered all the relevant parts.
\routine{PRESENT} just requires lists of the
data frames of each type and a filter type (any old value will do if this
isn't relevant). So an invocation of \routine{PRESENT} might be:
\begin{terminalv}
% present bias='bias*' target='ngc891*' flat='ff*'
\end{terminalv}
although you'd generally run it and respond to the prompts
interactively. CCDPACK programs can generally accept lists of data
frames, so it this case it processes all the frames with names
starting with \text{bias} as bias frames etc. Note that the
expansion of the wildcard symbol ``\text{*}'' happens inside the program
(and not by the shell as is usual) so it is protected by single quotes.
When running \routine{PRESENT} it's a good idea to check the output to
make sure that your frames are given the correct frame type. The known frame
types are:
\begin{itemize}
\item \texttt{TARGET}: These are the frames with the real data. The
``target'' of your observations (or perhaps the target that the
telescope points at).
\item \text{FLAT}: The flatfields.
\item \text{BIAS}: The bias frames
\item \text{DARK}: Any dark count frames (not usual).
\item \text{FLASH}: Any pre-flash frames (very unusual).
\end{itemize}
There are also several extra types that are used to define the
calibration masters. These are \text{MASTER\_BIAS},
\text{MASTER\_DARK}, \text{MASTER\_FLAT} and \text{MASTER\_FLASH}.
\routine{PRESENT} can be used to import foreign masters (such as
spectral flatfields).
\subsubsection{Reduction extension items \xlabel{reductionitems}}
When running \xroutine{PRESENT} you'll notice some strange names are
given under the ``\text{Item}'' column. These are the names of the
extension items stored in your data frames (at least they are for NDFs
anyway, other formats use other names). Most of their meanings are
fairly obvious as they correspond to the \xroutine{CCDSETUP}
parameters with similar names.
\begin{itemize}
\item \text{FTYPE}: The frame type (\text{TARGET}, \text{BIAS},
\text{FLAT}, \text{DARK} or \text{FLASH}).
\item \text{FILTER}: The filter name (any unique string).
\item \text{TIMES.DARK}: The dark count exposure time.
\item \text{TIMES.FLASH}: The pre-flash exposure time.
\item \text{DIRECTION}: The readout ``direction''.
\item \text{BOUNDS.START1}: The first row/column of the first bias strip.
\item \text{BOUNDS.END1}: The last row/column of the first bias strip.
\item \text{BOUNDS.START2}: The first row/column of the second bias strip.
\item \text{BOUNDS.END2}: The last row/column of the second bias strip.
\item \text{EXTENT.MINX}: Lower X value of the useful CCD section.
\item \text{EXTENT.MAXX}: Upper X value of the useful CCD section.
\item \text{EXTENT.MINY}: Lower Y value of the useful CCD section.
\item \text{EXTENT.MAXY}: Upper Y value of the useful CCD section.
\item \text{ADC}: The analogue to digital conversion factor.
\item \text{RNOISE}: The readout noise (ADUs).
\item \text{DEFERRED}: The deferred charge value.
\item \text{SATURATION}: The saturated pixel count (ADUs).
\end{itemize}
\subsubsection{\label{scheduling}Scheduling a reduction}
After you've performed the first two tasks this part is easy. Just run the
application \xroutine{SCHEDULE}:
\begin{terminalv}
% schedule in='*' execute=true
\end{terminalv}
This requires a list of the data frames to reduce and will show what
processing it reckons is required (including a range of possible
debiassing options), write a script to perform this and (optionally)
execute it.
It also has some nifty features such as picking up reduces that fail
(just give it all the names of the files produced as well as the
originals) and deleting intermediary frames to save on disk space.
\subsubsection{Scheduling from a script}
If you expect to regularly process large amounts of data from
a particular detector (so that using automated reduction is especially
attractive), it is fairly straight forward to write a script that will
do the reductions.
If you have an import control table (check the
\menu{Set detector...} item in the \menu{Options} menu of the
\xroutine{XREDUCE} program to see what CCDPACK has already)
then you would just need a script that contained something like:
\newpage
\begin{center}
\fbox{Example 1}
\end{center}
\begin{terminalv}
#!/bin/csh
#
# Initialize ccdpack
#
ccdpack
#
# Clear any existing global parameters.
#
ccdclear reset accept
#
# Do the general configuration.
#
ccdsetup logto=terminal genvar=false mask=defects.ard reset accept
#
# Import the FITS headers into CCDPACK.
#
import in='*' table=$CCDPACK_DIR/WHTFLAT.DAT reset accept
#
# Schedule and run the reduction.
#
schedule in='*' execute=true debias=1 spacesave=some reset accept
exit
\end{terminalv}
assuming that only the frames to be processed are in the current
directory. A copy of this file is available as
\text{\$CCDPACK\_DIR/ccdpack\_ex1.csh}
If you do not have an import control table then you will need to adopt
some method of differentiating your data into its various frame
types. Section \ref{backgroundprocessing} has some ideas about this.
Assuming you have adopted a simple naming scheme then a reduction
script might be like:
\begin{center}
\fbox{Example 2}
\end{center}
\begin{terminalv}
#!/bin/csh
#
# Initialize ccdpack
#
ccdpack
#
# Clear any existing global parameters.
#
ccdclear reset accept
#
# Restore the general and CCD configuration.
#
ccdsetup restore=true restorefile=CCDPACK_SETUP.DAT reset accept
#
# Present the data to CCDPACK (note different filters).
#
present target='DATAV*' bias='BIAS*' flat='FFV*' onefilter=true \
filter=V reset accept
present target='DATAR*' flat='FFR*' onefilter=true filter=R \
reset accept
#
# Schedule and run the reduction.
#
schedule in='"DATA*,BIAS*,FF*"' execute=true debias=1 spacesave=some \
reset accept
exit
\end{terminalv}
The file \text{CCDPACK\_SETUP.DAT}
is a \xroutine{CCDSETUP} restoration file and has been created by
running \routine{CCDSETUP} and saving its configuration.
A copy of this file is available as \\
\text{\$CCDPACK\_DIR/ccdpack\_ex2.csh}
\subsection{Using the CCDPACK programs to reduce data}
CCDPACK reductions are based around a suite of programs (that the
automated and GUI facilities rely on) that you can use directly. These
programs are:
\begin{itemize}
\item \xroutine{MAKEBIAS} - combines bias frames into a `master' bias calibration
frame.
\item \xroutine{DEBIAS} - debiasses lists of data frames either by master bias
subtraction or by estimation from the bias strips, applies bad data
masks, extracts a subset of the data area, produces errors and
detects saturated pixels values.
\item \xroutine{MAKECAL} - combines pre-flash or dark count frames into a `master'
calibration frame.
\item \xroutine{CALCOR} - performs dark or flash count corrections on a list of
frames.
\item \xroutine{MAKEFLAT} - combines flat fields into a `master' calibration
flatfield.
\item \xroutine{FLATCOR} - performs the flatfield correction on a list of frames.
\end{itemize}
If you want to process your data completely by hand (or if the
limitations of the automated processing are a problem) then the
following sections will lead you through the various options. At the
end of this section reduction scripts are shown as examples.
\subsubsection{Step 1 - Setting up}
The first step in starting a CCDPACK reduction sequence is to set up
the device characteristics using the routine:
\begin{itemize}
\item \xroutine{CCDSETUP}.
\end{itemize}
\routine{CCDSETUP} is described \slhyperref{elsewhere}{in \S}{}{configuration}.
\subsubsection{Step 2 - Making a bias calibration frame \xlabel{masterbias}
\label{masterbias}}
If you intend to debias your CCD data using bias frames, then the next
move is to combine all these into a `best bet' low noise frame; a
`master bias'. There are probably only two ways in which you'd like to
do this:
\begin{itemize}
\item Just combine them.
\item Zero the mean level first and then combine them (leaving the
mean at zero).
\end{itemize}
The second option may seem strange (if you're not used to it), but it
has a good rationale behind it and is the default method. Using this
method requires that your data have bias strips.
These are used as a monitor of the bias level at readout time and the
master bias is offset to them so that any small variations in the zero
point are tracked.
You make a master bias by running the program:
\begin{itemize}
\item \xroutine{MAKEBIAS}
\end{itemize}
If you want to make a master bias using the first method, then use a
command like:
\begin{terminalv}
% makebias in='bias/*' out=master_bias rnoise=10 zero=false
\end{terminalv}
for the second method use:
\begin{terminalv}
% makebias in='bias/*' out=master_bias rnoise=10
\end{terminalv}
The IN specification \text{bias/*} means get all the frames
in the subdirectory \text{bias/}.
The RNOISE parameter specifies the readout-noise (in ADUs) of
the CCD you're using (if you've set up a global value for this using
\xroutine{CCDSETUP} then this need not be supplied).
\routine{MAKEBIAS} shows an estimate of the readout-noise which it derives from
the data, use this to check your value, or use this value if none
other exists.
The nominal readout-noise value can usually be found in the technical
descriptions issued by the observatories.
CCDPACK uses the readout noise value to generate error estimates, you
may specify GENVAR=FALSE to disable this option if your
destination analysis package does not make use of data errors, your
data format doesn't support the storage of this information, or if
disk space is tight (the addition of error components to your data
will double the disk space needed).
\subsubsection{\xlabel{debiassing}\label{debiassing}Step 3 - Debiassing}
The next stage (or the first stage, if you're not using a master bias)
is to debias all your data frames; flatfields, flash frames, dark
frames, and the targets.
Debiassing can be done in two basic ways --- with and without a bias
frame (well actually three methods exist, the third being subtraction
of a constant; this is well worth avoiding unless there's nothing else
for it).
Let's tackle these methods one at a time.
\subsubsection{\label{withbiasframes}With a master bias}
If you have made a master bias frame using \xroutine{MAKEBIAS} then how you
debias depends on how you made it.
If your master bias has been combined to give a mean of zero then it will
require offsetting to the `zero' level in the bias strips.
\xroutine{DEBIAS} will require the values of the rows or columns that the strip(s)
are found within.
You tell \routine{DEBIAS} whether the values are rows or columns by specifying a
``readout direction'' `X' or `Y' (see Figure \ref{CCDPICCY}).
The bias strip ranges must be supplied in pairs; the column or row
number on which it starts and the column or row number on which it ends.
There are usually two strips on each side of the data, so this
requires 4 values.
If your data has three ``strips'' (probably as part of a region
running around the data) then choose the two parallel ones (but make
sure that the overscan strip, usually the one on the right, isn't
contaminated by residual charge), if it has only one then the choice
is obvious.
To subtract a zeroed master bias frame type something like:
\begin{terminalv}
% debias in='"rdata/*,ffr/*"' out='*_debias' bounds='[2,10,400,416]' rnoise=10
adc=1 bias=master_bias
\end{terminalv}
or conversely let \routine{DEBIAS} prompt you. If you meet any questions which you
do not understand hit return to accept the default, or respond with a
`?' to get some help. If things are really bad then `!!' (abort) will
always terminate the application immediately. (Note that ADC,
BOUNDS and RNOISE need not be given if you've used
\xroutine{CCDSETUP}.)
If your master bias frame has a non zero mean (if you've selected
the ZERO=FALSE option in \routine{MAKEBIAS}) you just want to subtract
it so use:
\begin{terminalv}
% debias in='"rdata/*,ffr/*"' out='*_debias' rnoise=10 adc=1 bias=master_bias
offset=false
\end{terminalv}
The ADC -- analogue-to-digital conversion -- factor is
required to generate error estimates from the number of ADUs recorded in
each pixel, as is the RNOISE value.
To avoid this just use GENVAR=FALSE and leave out the
ADC and RNOISE parameters.
\subsubsection{\xlabel{nobiasframes}\label{nobiasframes}Without a bias frame}
Debiassing of CCD data can be performed reasonably well by the
subtraction of values derived from the bias strips.
If the data has two bias strips then an interpolation using a straight
line or constant for each line is used.
Alternatively a plane can be fitted to the whole of the bias strip data.
If only one bias strip is present then extrapolation across the data
is used. In this case a single value is derived for each line or
one global value for the whole frame.
To subtract the bias using interpolation type something like:
\begin{terminalv}
% debias in='"rdata/*,ffr/*"' out='*_tmp' bounds='[2,10,400,416]' rnoise=10
adc=2
\end{terminalv}
This will interpolate between each pair of lines in the bias strips
using a constant. Before the interpolation occurs the bias strips are
smoothed using a box filter (this aims to reduce the variation {\em along}
the strips rather than across them, thus reducing the inter-line noise).
If you do not have any bias frames or bias strips then it is still
possible to debias the data provided you know what the debias level
actually is (this is also useful when the debiassing has already been
done for you and you want to add estimates of the errors, such as in
IR data).
To debias using a constant try something like:
\begin{terminalv}
% debias in='"rdata/*,ffr/*"' out='*_tmp' usecon zero=100.0
\end{terminalv}
This subtracts \text{100.0} from all the data before making error
estimates applying data masks etc.
IR data that are bias subtracted using combination bias and sky frames
should only be `debiassed' using this method if you actually know what
the bias level is (this is usually an observing option) otherwise any