notes.txt

2013-09-17
- this is my initial notes file
- lots of it is outdated/depracated
- it is kept for archival puroposes you shouldn't need it for anything
  unless you are very desperate


######################################################################
# TODO

General:

- there is a discrepancy in img binning between iter and qspec

- need a final test for T and lx using the 2 Andersson clusters

- check in iter spec the linking of rmf/arf so that it does what you
  want

- luminosity measurement need core excised and not excised + error
  estimation (see below, might be better to do this from images also
  given the area correction)

- is there a systematic shift to lower T in A11 -  see also below need
  to model background

- problem with fcarith? (not working?)

- rmf/arf problem:

  the most advanced versions:
  5		unknown		!Users!rs!data1!sw!esaspi!extract-arf.sh.~5~
  3		unknown		!Users!rs!data1!sw!esaspi!extract-rmf.sh.~3~

  currenntly the state is reverted. The advanced version has extraction
  from source region but 1. is slow (1 hour iter spec), 2. fails more
  often than simpler approach. The difference was quite small between
  the different approaches

- improve geometric refil factor by using the profile information

- think about the xspec lumin error estimation for unabsorbed flux - I
  think the current solution is pretty good though not completely
  ideal. ultimately we can rather the outputs of the imaging pipe

- iter spec speadup: error estimation for lx could be done only for
  the final step/ speedups also possible if the rmf/arf is not
  constantly recalculated since it makes only little difference

- refactor the ds9 usage (use the new DS9_BINARY everywhere)

- remask has different sets for det and wcs mask - is that ok?

- add model background not only annulus extraction

- explore why the dataset version of rmf/arf gives different
  normalizaiton while the same remaining parameters - comapre if esas
  rmf/arf is used instead

- in spec-*.sh the binned spectra use the old naming convetion without
  the region ID (in qspec)

- try increaseing the ps removal fraction (0.95-0.98)?

- from pn-spectra extract part that adds the proper ps regions to
  images like *-obj-im-500-2000.fits (ideally automatize extraction
  and coordinate conversion for the reg) - now EXTRACT_SPEC_ESAS_ has
  to be run which takes too much time if the esas background part is
  not yet used

- in 0205 the pn-man-mask.fits.cheese are merged all 3 (same for m1
  and m2) - rahter than individual... why?

- imaging pipeline - PSF fitting

- find out what causes the eveselct change - go through the parameters

- centralize cosmology (see py and xspec part)

- dedicated source detection module

- would be nice to cleanup the namespace (sometime variables are
  duplicated) and the capitalization convention should be better

Imaging:
- distmatrix needs to pass only shape not full im

Iterator:
- core PSF spillover?
- how to stop before running into low s/n?
- use andersson optimized band for spectroscopy?
- what about the python deprecation warning

Profiles:

Snowden:
- do the actual fitting
- annulus support for converter
- finalize/test proton
- finalize/test combine-smooth.sh

######################################################################
# dependencies

SAS v. >=11.0.0
ftools

python:
- scipy, numpy, pyfits, matplolib

quick-spec:
- requires vikhlinins calculator # FIXME: refactor to remove this dependency
- it has to be in your $PATH

iter-spec:
- requires cosmo_dist.py in the python path
- requires vikhlinins calculator # FIXME: refactor to remove this dependency

sb fitter:
- requires lmfit

- requires minuit as the fitting package (deprecated - not needed
  anymore)

######################################################################
# running on a dir - how to setup an analysis directory

1. download tar from XSA
2. untar 1 time: should have OBSID/odf/xxxxxx.tar.gz, copy to your
   work dir
3. get-templates.sh 060401099
   - standalone script to do the prep work

old/manual way:
have a .conf and .modules file in configs/ (e.g. copy templates):

cp ~/data1/sw/esaspi/templates/template.conf configs/SPT-XXXXX.conf
cp ~/data1/sw/esaspi/templates/template.modules configs/SPT-XXXXX.modules

- edit them by setting the config options and pick the first module

4. there should be a notes/ dir for analysis notes
5. there has to be a soft link OBSID -> your analysis dir : you can
use /Users/rs/w/xspt/data/link-obsid-clname.sh
6. initialize heasoft, sas11
7. run

dir=test0205 ; run-esas.sh configs/${dir}.conf configs/${dir}.modules 2>&1 | tee log-${dir}-TESTING-prefixes.log

NOTES:
3. -> path can be set from running run-esas.sh

3. + 4. -> or change the setups in run-esas.sh

5. -> should be changed to allow multiple reductions per OBSID

######################################################################
# calls to the pipeline

basic main call example:

dir=SPT-CL-J9999-9999 ; task=QSPEC-01 ; run-esas.sh configs/${dir}.conf configs/${dir}.modules 2>&1 | tee log-${dir}-${task}.log

quick-spec call in cluster/spec/ subdir:

ID=001 ; rm -r $ID; spec-m1m2pn-ta.sh SPT-CL-J9999-9999 ${ID} conf/SPT-CL-J9999-9999-par-qspec-001.conf 2>&1 | tee spec.log

######################################################################
# file naming etc.

possible prefixes are:

[pn,mos1,mos2][U,S]???

- prefix list is now automatically added to analysis notes files (and
  should be copied to the .conf file) in the RUN_FILTERS module

analysis:
pnS003-ori.fits    - raw event list
pnS003-clean.fits  - cleaned event list

abbreviations:

bg - background
gc - growth curve = cumulative profile
oot - out of time events
ps - point source(s)
uv - unvignetted
woot - *with* oot (i.e. pre-subtraction)

######################################################################
# create region - manual solution - CLOSED

cd 0097820101/analysis
xmmselect table=pnS005-clean.fits &

select DETX DETY
click image
click run
create region
click 2D region
copy to pn-reg.txt
prepend &&
e.g.:

&&((DETX,DETY) IN circle(69.7556,-86.7555,600))


######################################################################
# solution for region files - CLOSED (add annulus support)

- look into /code/convert-reg-wcs-to-det.sh

- standard regions (circular) can be created in ds9 in wcs and
automatically converted with: /code/convert-reg-wcs-to-det.sh


- manual handling:

cd 0097820101/analysis

make wcs region file:

# Region file format: DS9 version 4.1
# Filename: pnS005-obj-im.fits
global color=green dashlist=8 3 width=1 font="helvetica 10 normal" select=1 highlite=1 dash=0 fixed=0 edit=1 move=1 delete=1 include=1 source=1
fk5
circle(207.22025,26.590618,30")

ecoordconv imageset=pnS005-obj-im.fits   x=207.22025 y=26.590618 coordtype=EQPOS
ecoordconv imageset=mos1S003-obj-im.fits x=207.22025 y=26.590618 coordtype=EQPOS
ecoordconv imageset=mos2S004-obj-im.fits x=207.22025 y=26.590618 coordtype=EQPOS

~/data1/sbox/esas/0097820101/analysis> ecoordconv imageset=pnS005-obj-im.fits   x=207.22025 y=26.590618 coordtype=EQPOS
ecoordconv:- Executing (routine): ecoordconv imageset=pnS005-obj-im.fits srcexp='' x=207.22025 y=26.590618 coordtype=EQPOS withcoords=yes  -V 4
ecoordconv:- ecoordconv (especget-1.31.2)  [xmmsas_20110223_1801-11.0.0] started:  2011-09-05T16:09:16.000
ecoordconv:-  Region Centre:
 Theta: Phi: 65.8815 3.53639
 X: Y: 26421 27821
 DETX: DETY: 67.2793 -123.994
 RA: DEC: 207.22 26.5906
 RAWX: RAWY: 38 189
 CCD(s):  4 centred on CCD: 4

ecoordconv:- ecoordconv (especget-1.31.2)  [xmmsas_20110223_1801-11.0.0] ended:    2011-09-05T16:09:17.000
[4]+  Done                    ds9 mos2S004-im-det.fits pnS005-im-det.fits
~/data1/sbox/esas/0097820101/analysis> ecoordconv imageset=mos1S003-obj-im.fits x=207.22025 y=26.590618 coordtype=EQPOS
ecoordconv:- Executing (routine): ecoordconv imageset=mos1S003-obj-im.fits srcexp='' x=207.22025 y=26.590618 coordtype=EQPOS withcoords=yes  -V 4
ecoordconv:- ecoordconv (especget-1.31.2)  [xmmsas_20110223_1801-11.0.0] started:  2011-09-05T16:09:35.000
ecoordconv:-  Region Centre:
 Theta: Phi: 5.59933 4.38992
 X: Y: 26421 27821
 DETX: DETY: 205.322 -244.603
 RA: DEC: 207.22 26.5906
 RAWX: RAWY: 310 289
 CCD(s):  1 centred on CCD: 1

ecoordconv:- ecoordconv (especget-1.31.2)  [xmmsas_20110223_1801-11.0.0] ended:    2011-09-05T16:09:35.000
~/data1/sbox/esas/0097820101/analysis> ecoordconv imageset=mos2S004-obj-im.fits x=207.22025 y=26.590618 coordtype=EQPOS
ecoordconv:- Executing (routine): ecoordconv imageset=mos2S004-obj-im.fits srcexp='' x=207.22025 y=26.590618 coordtype=EQPOS withcoords=yes  -V 4
ecoordconv:- ecoordconv (especget-1.31.2)  [xmmsas_20110223_1801-11.0.0] started:  2011-09-05T16:09:45.000
ecoordconv:-  Region Centre:
 Theta: Phi: 66.8417 5.07716
 X: Y: 26421 27821
 DETX: DETY: 62.5219 -16.9727
 RA: DEC: 207.22 26.5906
 RAWX: RAWY: 303 300
 CCD(s):  1 centred on CCD: 1

- dmmakereg in ciao might be relevant

######################################################################
# point source removal - CLOSED

# FIXME: adding more elegant solution at the remask step

###############
# update: Mon Apr 16 15:17:52 2012

- using emllist*reg keep only spurious ps - save list
- then use the saved list and do in tpc (topcat) all matches in 1 not 2 with
  emllist, save as emllist.fits

###############
ds9 *cheese*

compare this to image - if removal is insufficient:

fcat2reg.sh emllist.fits 10.0 ML_ID_SRC

fv emllist.fits &

find the ML_ID_SOURCE you want to remove and edit it DIST_NN so that
it is > 40 (possibly but unlikely also edit other columns so that it
makes the cut of the region task, see below) columns to change:

# Mon Jan  2 15:35:54 2012: the default is now 20", it's set in cheese

DET_ML
FLUX
DIST_NN

cuts depending on what is in remask.sh

- save the change in the fits file
- run the remask.sh task
- check everything again

Further trick:

ds9 *cheese* &

create contours, 2 levels 0/1, lowest smoothness and save this way
you'll see which of the all emldetect sources get actually masked out

this is how cheese calls region/make mask it:

region eventset=pnS005-clean.fits operationstyle=global srclisttab=emllist.fits:SRCLIST expression='(DIST_NN >= 40.0)&&(FLUX >= 1e-14)&&(ID_INST == 0)&&(ID_BAND == 0)&&(DET_ML >= 15)' bkgregionset=pnS005-bkg_region-det.fits energyfraction=0.5 radiusstyle=enfrac outunit=detxy verbosity=1

region eventset=pnS005-clean.fits operationstyle=global srclisttab=emllist.fits:SRCLIST expression='(DIST_NN >= 40.0)&&(FLUX >= 1e-14)&&(ID_INST == 0)&&(ID_BAND == 0)&&(DET_ML >= 15)' bkgregionset=pnS005-bkg_region-sky.fits radiusstyle=contour bkgratestyle=col nosrcellipse=yes bkgfraction=0.5 outunit=xy verbosity=1

make_mask inimage=pnS005-obj-im.fits inmask=pnS005-mask-im.fits outmask=pnS005-cheese.fits reglist=pnS005-bkg_region-sky.fits


- region is undocumented
- region has a expression - can take emldetect output and post-select
-> this will be clearly useful later

- note the radiusstyle differes, is this significant or bug?


######################################################################
# rass background spectrum - CLOSED

go here:
http://heasarc.gsfc.nasa.gov/cgi-bin/Tools/xraybg/xraybg.pl

1. put in source coords
2. region annulus: inner 1 deg, outer 2 deg
3. save spectrum as rass.pi and response as pspcc.rsp in analysis/

######################################################################
# images

ESAS part:

for *obj-im*  900x900
for *obj-det* 780x780

RS part:
900x900

DCP pipe:
648x648
1pix=4"
1"=20phypix

e.g. in iter-spec:

export r=$r_init
export r_phy=$(echo "scale=6;$r*20.0" | bc)

-> these are sky coordinates so the conversion is independent of
   binning

pprof pipe:
648x648

kapi:
?x? - unlisted, seems no way to get it (no headers, maybe paper?)


######################################################################
# quick spectroscopy

1. run esaspi in a standard manner, until the end

2. open e.g. pnS003-obj-im-400-1250.fits
(for background annulus mos is better to contain it in a single chip)

3. adjust regions to needs, define core and bgann - keep the ps
regions roughly confined to the spectroscopic regions - in order not
to have too many regions which conflicts with SAS tasks (still so with
SASv11?). For ps you can load in the *bkg_region.fits to ds9. The
exclusion red bar can be (un-)set in the "properties" menu (select all
first)

4. save the regions in wcs and phys, standard naming:
SRC_REGION=cluster-man-01.phy.reg
BG_REGION=bg-ann-01.phy.reg
PS_REGION=ps-man.phy.reg

5. manually adjust quick-spec-locbg.sh if needed

6. run quick-spec-locbg.sh via run-esaspi.sh (set it in the modules)

7. fitting:

7.1 cd to spec dir
7.2 check conf/*.conf file

7.3 make sure you don't have a DYLD_LIBRARY_PATH conflict, if it is
overtaken by SAS, run:

export DYLD_LIBRARY_PATH=/Users/rs/data1/sw/heasoft-6.11/i386-apple-darwin10.7.0/lib

7.3 run a fitter from the quick-spec dir, e.g.

spec-m1m2pn-ta.sh abc-0205 001 abc-0502-par-001.conf

where

cluster=$1
fitid=$2
parfile=$3

######################################################################
# analysis-template.txt

- the filename can be set in run-esas.sh
- some scripts can use it to load in parameters automatically
- for a given parameter the *first upper case* instance is read in
  automatically
- i.e. alternative parameters are supported, and this is the mechanism
  to load them in

######################################################################
# xspec sas11 conflict

xspec opened in a shell with sourced SAS will quickly crash with bus
error.

Reason likely is library conflict:

original heainit (xspec works):

DYLD_LIBRARY_PATH=/Users/rs/data1/sw/heasoft-6.11/i386-apple-darwin10.7.0/lib

after sourcing sas11 (xspec crashes):

DYLD_LIBRARY_PATH=/Users/rs/data1/sw/sas-11.0.0/xmmsas_20110223_1803/libsys:/Users/rs/data1/sw/sas-11.0.0/xmmsas_20110223_1803/libextra:/Users/rs/data1/sw/sas-11.0.0/xmmsas_20110223_1803/lib:/Users/rs/data1/sw/heasoft-6.11/i386-apple-darwin10.7.0/lib

fix it:
before running xspec reset to:

export DYLD_LIBRARY_PATH=/Users/rs/data1/sw/heasoft-6.11/i386-apple-darwin10.7.0/lib

######################################################################
# timing
# times are for ~40-60 ks observations

export EXTRACT_SPEC_ESAS_PN=0
export EXTRACT_SPEC_ESAS_M1=0                   # !!!! not tested yet
export EXTRACT_SPEC_ESAS_M2=0                   # !!!! not tested yet
export EXTRACT_BACK_ESAS_PN=0                   # !!!! not tested yet
export EXTRACT_BACK_ESAS_M1=0                   # !!!! not tested yet
export EXTRACT_BACK_ESAS_M2=0                   # !!!! not tested yet
export GROUP_SPEC=0                             # FIXME - prefixes the
# EXTRACT_* block takes about 5-6 hours - if ran on FoV: quite
# demanding - run only during night and close unecessary windows (can
# lead to kernel panic if overload)

export QUICK_SPEC_LOCBG=1
# roughly 20 mins if extracting both source and background, arf, rmf
# and backscale

export PREP_ODF_DIR=0           # ~ 1min
export MAKE_CCF_ODF=0           # ~ 1min
export RUN_EV_CHAINS=0          # ~ 8min
export RUN_FILTERS=0            # ~ 1min
export CHEESE_1B=0              # ~ 2min
export CHEESE_2B=0                              # !!!! not tested yet
export REMASK=0
export REMASK_MANUAL_MASK=0
export DO_EVIGWEIGHT=1          # ~ >1min
export CORRECT_PROTON=0            # !!!! not finished/tested yet
export COMBINE_SMOOTH=0            # !!!! not finished/tested yet
export RENAME_SPEC_PRODUCTS=0      # should be used after CORRECT_PROTON and COMBINE_SMOOTH
export PIPE_TEST=0
export PLT_LC_HIST=0
export PLT_DIAGNOSTICS=0
export RELINK_SPEC_PRODUCTS=0
export MAKE_QUICK_VIEW_IMS=0

for iterative spectroscopy:

- 1 iteration including arf/rmf calculation is ~10 mins (for small
  apertures <~100")

- convergence typically in <5 iterations (hard limit 10 at the moment)

######################################################################
# libraries with internal settings/options variables
# i.e. stuff that can/should be tweaked locally manually  in the file

remask.sh - can change options whether or not add manually selected ps

quick-spec-locbg.sh
iter-spec/iter-spec-driver.sh
py/t_to_r.py

/imaging/make-model-bg-spline.sh
USE_OOT=0                       # use the oot correction for pn? using
                                # it causes some small (maybe
                                # negligible) artefacts

######################################################################
# currently deprecated scripts (debris, early draft no used

# anymore/not useful at the moment)

quick-spec/setrsp.sh
see also ../aux-esaspi/ (not under git)

######################################################################
# arf/rmf

my old rmf/arf parameter setup vs. the esas setup gives agreement to
better than 0.5%

so I currently use the badpixel location, psf weighted rmf and arf
calculated with a detmap - likely you cand do better then this though
the difference is totally negligible

FIXME: add this arf/rmf setup also to iter-spec

######################################################################
# standalone/external scripts
# i.e. small utility scripts that can be run outside run-esaspi.sh

utils/get-coords-from-reg.sh

- extracts coordinates for analysis script
- INPUT: region ID (bar wcs id and suffix, standard naming)

utils/parse-spec-reg.sh

- converts a region to .desc file: a SAS compliant selection
  expression
- INPUT:
  1. ds9 region file
  2. 0/1 - remove/keep these regions? i.e. put &&! or &&

######################################################################
# devel notes

0. random

- dmfilth for refiling ps/gaps

- oot subbed spectra give slightly (tiny) difference in arf but rmf is
  exactly same (spec results are unaffected by this)


1. SB analyisis

- 1D vs 2D fitting

- binning - 3sigma + logarithmic min? or better not at all

- arnaud02 vs mohr99
- further: pratt & arnaud02 (2d fit), pratt & arnaud05
- final refinement croston05

- XMM-LSS:
- alshino10
- claim arnaud02, not clear
- get a test case from here?

arnaud02
- RX J1120.1+4318 - another test case (start here)

other references - helpful:

- lorenzo - no
- yu-ying thesis - no
- desiree - no
- karl - ??
- SAS ??
- snowden no
- sanders ??
- maughan vids - ??

2D:

- needs to extract sub-image for speed -> maybe not: fftconvolve is
  significantly faster then convolve

- but still creating the beta image takes some time

- 900x900 takes 3s with fftconvolve:
  beta creation took:  0.778832  s
  PSF creation took:  0.717411  s
  convolve took:  1.190855  s

- could me made faster - decrease to about 600x600 or less + possibly
  a vecrorication of the image creation is possible? (currently for
  loops)

Other speedup possibilities:
can remove optional keyword and test like DO_ZERO_PAD and APPLY_PSF
(can be always true)

What works (validated)
Mon Sep 10 14:50:08 2012

1. beta model 1d fitter (2d image via its profile)
   test_lmfit_beta_1d

2. 2d beta x PSF image creator in lmfit format
   test_create_beta_psf_image
   make_2d_beta_PSF
   - validate in both modes  - with and without psf

2. background

- in imaging pipe need to improve the masking for 2comp/spline: too much exclusion?
- also the chipgaps are too larger - check snowden parameters on it - they are the same
- other methods: arnaud, xmm-LSS, yu-ying, snowden, karl


######################################################################
# moving to desproc:

1. DONE setting paths at run-esas
2. DONE currently manual flag notebook/desproc, possibility is to use
   automatic switch based on host name
3. DONE the perl path in xspec scripts
4. DONE the calc path needs to be refactored
5. DONE ideally setup softlinked dirs to mirror notebook...
6. check across code ~ vs. hard linked home paths

######################################################################
# running on depsroc

# sync esaspi
source ~/bin/setup_desproc_paths.sh
heainit
sas11
# should be now able to run everything
# path settings are in run-esaspi.sh and should be automatically
# recognize


######################################################################
# tmp code dump

# Tue Aug 21 16:35:34 2012 - fitter
    ######################################################################
    # # setup sb model
    # model_fit =  minuit.Minuit(minuit_sb_model_likelihood,
    #                            #
    #                            norm=norm0,
    #                            fix_norm=False,
    #                            limit_norm=limit_norm,
    #                            #
    #                            rcore=rcore0,
    #                            fix_rcore=False,
    #                            limit_rcore=limit_rcore,
    #                            #
    #                            beta=beta0,
    #                            fix_beta=False,
    #                            limit_beta=limit_beta,
    #                            #
    #                            xsize=xsize,
    #                            fix_xsize=True,
    #                            # limit_xsize=None,
    #                            #
    #                            ysize=ysize,
    #                            fix_ysize=True,
    #                            # limit_ysize=None,
    #                            #
    #                            xsize_obj=xsize_obj,
    #                            fix_xsize_obj=True,
    #                            # limit_xsize_obj=None,
    #                            #
    #                            ysize_obj=ysize_obj,
    #                            fix_ysize_obj=True,
    #                            # limit_ysize_obj=None,
    #                            #
    #                            xcen=xcen,
    #                            fix_xcen=True,
    #                            # limit_xcen=None,
    #                            #
    #                            ycen=ycen,
    #                            fix_ycen=True,
    #                            # limit_ycen=None,
    #                            #
    #                            instrument=instrument,
    #                            fix_instrument=True,
    #                            # limit_instrument=None,
    #                            #
    #                            theta=theta,
    #                            fix_theta=True,
    #                            # limit_theta=None,
    #                            #
    #                            energy=energy,
    #                            fix_energy=False,
    #                            # limit_energy=None
                               # )


# results

beta true:  0.666666666667
rcore true:  10.0

beta:  0.674940199813 0.187876144339
rcore:  10.1513293223 5.33226054116
norm:  3.31932610478 1.00144036795