Code for the analysis of 21 cm intensity mapping data.
Python IDL Other
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
cal Lots of work on beam fits. Feb 12, 2013
core Separate logic if only measureing the variance. Jul 22, 2014
foreground bug where weights was cast as int. Jul 5, 2012
foreground_clean added kperp and kpar hard cuts, pure signal line sim Jan 4, 2013
history moved subtraction/datalike to pwr operations; working autopower pipel… Oct 29, 2012
input Merge branch 'kiyo_branch' Apr 8, 2013
kiyopy added kiyopy Jul 10, 2012
map Merge branch 'eric_switzer' into kiyo_branch Apr 8, 2013
noise Separate logic if only measureing the variance. Jul 22, 2014
old_code Moving stuff around. Aug 9, 2011
pipeline added mode cleaning to bulk sim, pipeline manager leaves a lock file,… Jun 28, 2012
plots cleaning up source tree; added trailing ini files; moved more to history Jun 28, 2012
plotting first pass at improved SVD weights; 1hr run scripts; trans func with … Nov 11, 2012
postprocess small formatting changes to plots. Apr 12, 2013
profile Moved more stuff and updated unit tests to point to test data. Aug 9, 2011
pwrspec_plots quick autopower code running; starting to make bulk sims for autopower Oct 14, 2012
quadratic_products added truncation of 2d k plane for noise realizations Feb 4, 2013
scripts Rough bandpass script Jul 24, 2014
simulations making cspline import rrobust; ncpu=1 rather than 24 for single auto Oct 26, 2012
testdata worked on polarized beam models. made script to extract noise_inv dia… Nov 28, 2012
time_stream Started adding hdf5 support ofr Also started 2D ortho_pol… Nov 14, 2012
utils Merge branch 'eric_switzer' into kiyo_branch Apr 8, 2013
.gitignore_global modified .gitignore Feb 20, 2013
README Moved more stuff and updated unit tests to point to test data. Aug 9, 2011 Moved more stuff and updated unit tests to point to test data. Aug 9, 2011


This Kiyo's code for processing GBT data.  This code depends on the Kiyo's
personal utilities package: kiyopy (available from github, user: kiyo-masui) as
well as several external packages: scipy, numpy, ephem, pyfits.

Note that for the paths to the different packages in the subdirectories to work
out, the python interpreter should be invoked from this directory. i.e.:
python core/
not :
cd core

This will make a difference for code that depends on code in a different

Ideally, any code pushed back to the github repository should be tested and
pass the test suit (python test_*).  Also please be conscious of backward

Environment Variables:

Environment variable are used in the input files so that the same input file
can be used on different systems (and by different users) without changing the
input file (pain in the ass if this input file is versioned!).  Currently I am
using two environment variables which any user should set in thier .bashrc or

$GBT10B_DATA - Points to raw GBT data. Must be readable but doesn't have to be
$GBT10B_OUT - Points to the directory where outputs and intermediate files are
stored.  The programs will build subdirectories in this one.  Must be writable.

In addition your $PYTHONPATH variable MUST
begin with a ':'.  The ':' is like adding a blank string
to your python path and tells the interpreter that the current working
directory is to be added to the path.

Overview and Notes on Code Design :

The philosophy is to have a modular pipeline where different components on this
pipeline can be dropped in and out in any order without crashing the whole
thing.  To that effect, the data is stored to disk after every step of the
pipeline and it is always stored in the same format: COMPATIBLE WITH THE
ORIGINAL GBT FITS FILES.  This should be true right up to the map making step,
where obviously the data format changes.

For example: lets say we have a steps on the pipeline, one that applies some 
filter to the time stream data and another that flags bad data.  The pipeline 
looks as follows.

raw gbt fits data -> flags bad data -> fits data -> filter -> fits data 
         -> map making

Then, without changing any code, we also want to be able to do the following:
raw gbt fits data -> filter -> fits data -> map making
raw gbt fits data -> flags bad data -> fits data -> map making
or even:
raw gbt fits data -> map making

The advantages are as follows:
  - Can add and remove modules at will to see what effect they have.
  - Can replace modules, make better ones and compare results trivially.
  - Can write modules without understanding or breaking the other ones.
  - Can independently test modules.
  - Don't have to wait for a monolithic code to be written before we start
    testing and evaluating algorithms.

Note that with this modular set up there is no reason that a module would have
to be written in a particular language.  However, if you do decide to use
python for your module, I have written some infrastructure that will
facilitate things.

Directories :

1. core - These are core utilities for reading, writing and storeing data.

The DataBlock class in  This is the vessel for holding time
stream data.  It holds a single IF and a single scans worth of data.  It
contains ALL the information needed to make a valid fits file.

Reader Class in  This reads a fits file and returns DataBlocks.

Writer Class in  This takes a bunch of DataBlocks and writes them
to a fits file.

The system probably isn't air tight, but as long as you do all your IO with
these, you should be forced to conform to the proper standards.  For examples
of how to use these classes, take a look at the unit tests (test_*.py).

2. time_stream - Any module of the analysis where both the inputs and outputs
are time stream fits files compatible with the the raw data taken at GBT.

3. inifile_km - Input files for created by Kiyo Masui for variouse modules.
Feel free to make you own inifile_your_initials directory or to use my input

4. map - where I will be putting map makers.

5. pipeline - Code that strings a bunch of modules together for covieniece.

Module Layout :

To be compatible with the pipeline, a module has to have a specific layout.
More on this to come.