Skip to content
Code for the analysis of 21 cm intensity mapping data.
Python IDL Other
Latest commit eacf84b @kiyo-masui Merge pull request #10 from kiyo-masui/fix_measure_noise
Separate logic if only measureing the variance.
Failed to load latest commit information.
cal Lots of work on beam fits.
core Separate logic if only measureing the variance.
foreground bug where weights was cast as int.
foreground_clean added kperp and kpar hard cuts, pure signal line sim
history moved subtraction/datalike to pwr operations; working autopower pipel…
input Merge branch 'kiyo_branch'
kiyopy added kiyopy
map Merge branch 'eric_switzer' into kiyo_branch
noise Separate logic if only measureing the variance.
old_code Moving stuff around.
pipeline added mode cleaning to bulk sim, pipeline manager leaves a lock file,…
plots cleaning up source tree; added trailing ini files; moved more to history
plotting first pass at improved SVD weights; 1hr run scripts; trans func with …
postprocess small formatting changes to plots.
profile Moved more stuff and updated unit tests to point to test data.
quadratic_products added truncation of 2d k plane for noise realizations
scripts Rough bandpass script
simulations making cspline import rrobust; ncpu=1 rather than 24 for single auto
testdata worked on polarized beam models. made script to extract noise_inv dia…
time_stream Started adding hdf5 support ofr Also started 2D ortho_pol…
utils Merge branch 'eric_switzer' into kiyo_branch
.gitignore_global modified .gitignore
README Moved more stuff and updated unit tests to point to test data. Moved more stuff and updated unit tests to point to test data.


This Kiyo's code for processing GBT data.  This code depends on the Kiyo's
personal utilities package: kiyopy (available from github, user: kiyo-masui) as
well as several external packages: scipy, numpy, ephem, pyfits.

Note that for the paths to the different packages in the subdirectories to work
out, the python interpreter should be invoked from this directory. i.e.:
python core/
not :
cd core

This will make a difference for code that depends on code in a different

Ideally, any code pushed back to the github repository should be tested and
pass the test suit (python test_*).  Also please be conscious of backward

Environment Variables:

Environment variable are used in the input files so that the same input file
can be used on different systems (and by different users) without changing the
input file (pain in the ass if this input file is versioned!).  Currently I am
using two environment variables which any user should set in thier .bashrc or

$GBT10B_DATA - Points to raw GBT data. Must be readable but doesn't have to be
$GBT10B_OUT - Points to the directory where outputs and intermediate files are
stored.  The programs will build subdirectories in this one.  Must be writable.

In addition your $PYTHONPATH variable MUST
begin with a ':'.  The ':' is like adding a blank string
to your python path and tells the interpreter that the current working
directory is to be added to the path.

Overview and Notes on Code Design :

The philosophy is to have a modular pipeline where different components on this
pipeline can be dropped in and out in any order without crashing the whole
thing.  To that effect, the data is stored to disk after every step of the
pipeline and it is always stored in the same format: COMPATIBLE WITH THE
ORIGINAL GBT FITS FILES.  This should be true right up to the map making step,
where obviously the data format changes.

For example: lets say we have a steps on the pipeline, one that applies some 
filter to the time stream data and another that flags bad data.  The pipeline 
looks as follows.

raw gbt fits data -> flags bad data -> fits data -> filter -> fits data 
         -> map making

Then, without changing any code, we also want to be able to do the following:
raw gbt fits data -> filter -> fits data -> map making
raw gbt fits data -> flags bad data -> fits data -> map making
or even:
raw gbt fits data -> map making

The advantages are as follows:
  - Can add and remove modules at will to see what effect they have.
  - Can replace modules, make better ones and compare results trivially.
  - Can write modules without understanding or breaking the other ones.
  - Can independently test modules.
  - Don't have to wait for a monolithic code to be written before we start
    testing and evaluating algorithms.

Note that with this modular set up there is no reason that a module would have
to be written in a particular language.  However, if you do decide to use
python for your module, I have written some infrastructure that will
facilitate things.

Directories :

1. core - These are core utilities for reading, writing and storeing data.

The DataBlock class in  This is the vessel for holding time
stream data.  It holds a single IF and a single scans worth of data.  It
contains ALL the information needed to make a valid fits file.

Reader Class in  This reads a fits file and returns DataBlocks.

Writer Class in  This takes a bunch of DataBlocks and writes them
to a fits file.

The system probably isn't air tight, but as long as you do all your IO with
these, you should be forced to conform to the proper standards.  For examples
of how to use these classes, take a look at the unit tests (test_*.py).

2. time_stream - Any module of the analysis where both the inputs and outputs
are time stream fits files compatible with the the raw data taken at GBT.

3. inifile_km - Input files for created by Kiyo Masui for variouse modules.
Feel free to make you own inifile_your_initials directory or to use my input

4. map - where I will be putting map makers.

5. pipeline - Code that strings a bunch of modules together for covieniece.

Module Layout :

To be compatible with the pipeline, a module has to have a specific layout.
More on this to come.

Something went wrong with that request. Please try again.