Python IDL Other
|Failed to load latest commit information.|
This Kiyo's code for processing GBT data. This code depends on the Kiyo's personal utilities package: kiyopy (available from github, user: kiyo-masui) as well as several external packages: scipy, numpy, ephem, pyfits. Note that for the paths to the different packages in the subdirectories to work out, the python interpreter should be invoked from this directory. i.e.: python core/test_data_block.py not : cd core python test_data_block.py This will make a difference for code that depends on code in a different directory. Ideally, any code pushed back to the github repository should be tested and pass the test suit (python test_*). Also please be conscious of backward compatibility. Environment Variables: Environment variable are used in the input files so that the same input file can be used on different systems (and by different users) without changing the input file (pain in the ass if this input file is versioned!). Currently I am using two environment variables which any user should set in thier .bashrc or .tcshrc. $GBT10B_DATA - Points to raw GBT data. Must be readable but doesn't have to be writable. $GBT10B_OUT - Points to the directory where outputs and intermediate files are stored. The programs will build subdirectories in this one. Must be writable. In addition your $PYTHONPATH variable MUST begin with a ':'. The ':' is like adding a blank string to your python path and tells the interpreter that the current working directory is to be added to the path. Overview and Notes on Code Design : The philosophy is to have a modular pipeline where different components on this pipeline can be dropped in and out in any order without crashing the whole thing. To that effect, the data is stored to disk after every step of the pipeline and it is always stored in the same format: COMPATIBLE WITH THE ORIGINAL GBT FITS FILES. This should be true right up to the map making step, where obviously the data format changes. For example: lets say we have a steps on the pipeline, one that applies some filter to the time stream data and another that flags bad data. The pipeline looks as follows. raw gbt fits data -> flags bad data -> fits data -> filter -> fits data -> map making Then, without changing any code, we also want to be able to do the following: raw gbt fits data -> filter -> fits data -> map making or: raw gbt fits data -> flags bad data -> fits data -> map making or even: raw gbt fits data -> map making The advantages are as follows: - Can add and remove modules at will to see what effect they have. - Can replace modules, make better ones and compare results trivially. - Can write modules without understanding or breaking the other ones. - Can independently test modules. - Don't have to wait for a monolithic code to be written before we start testing and evaluating algorithms. Note that with this modular set up there is no reason that a module would have to be written in a particular language. However, if you do decide to use python for your module, I have written some infrastructure that will facilitate things. Directories : 1. core - These are core utilities for reading, writing and storeing data. The DataBlock class in data_block.py: This is the vessel for holding time stream data. It holds a single IF and a single scans worth of data. It contains ALL the information needed to make a valid fits file. Reader Class in fitsGBT.py: This reads a fits file and returns DataBlocks. Writer Class in fitsGBT.py: This takes a bunch of DataBlocks and writes them to a fits file. The system probably isn't air tight, but as long as you do all your IO with these, you should be forced to conform to the proper standards. For examples of how to use these classes, take a look at the unit tests (test_*.py). 2. time_stream - Any module of the analysis where both the inputs and outputs are time stream fits files compatible with the the raw data taken at GBT. 3. inifile_km - Input files for created by Kiyo Masui for variouse modules. Feel free to make you own inifile_your_initials directory or to use my input files. 4. map - where I will be putting map makers. 5. pipeline - Code that strings a bunch of modules together for covieniece. Module Layout : To be compatible with the pipeline, a module has to have a specific layout. More on this to come.