No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
cps_data Merge in changes from PR278 Aug 29, 2018
cps_stage1 Set targets precision to '%.4f' Jul 1, 2018
cps_stage2 Add re-zipping logic to Makefile Jul 5, 2018
cps_stage3 Add benefits tests May 29, 2018
doc
puf_data Add comment describing HIWAGE_PROB_SF calibration Aug 31, 2018
puf_stage1
puf_stage2 update weights Aug 10, 2018
puf_stage3 update weights Aug 10, 2018
tests Move taxdata to using Python 3.6 Sep 10, 2018
.gitignore
.travis.yml Revise .travis.yml to run GitHub tests using Python 3.6 Sep 10, 2018
LICENSE Update LICENSE Oct 20, 2015
Makefile Add impute_pencon.py to Makefile and puf_data/finalprep.py Aug 28, 2018
README.md More edits to README.md file Jul 3, 2018
csvcopy.sh Remove stage4 logic from csvcopy.sh script Aug 10, 2018
datasets.md readable links Mar 20, 2018
environment.yml Avoid Python 3.7 for the time being Sep 11, 2018
gitpr
gitsync Steamline bash scripts to do complex git operations Dec 21, 2017

README.md

About taxdata Repository

This repository prepares data used in the Tax-Calculator repository.

The data files produced here, all of which have CSV format, provide two different sets of data files for Tax-Calculator:

  • A set based on a recent IRS-SOI Public Use File (PUF)

  • A set based on recent Census Current Population Survey (CPS) data

Because the PUF data are restricted in their use, the IRS-SOI-supplied PUF file and the puf.csv data file produced here are not part of the taxdata repository or the Tax-Calculator repository.

Each of these two sets of data files contains several types of files:

  1. a sample data file containing variables for each tax filing unit;

  2. a factors file containing annual variable extrapolation factors;

  3. a weights file containing annual weights for each filing unit;

  4. a ratios file containing annual adjustment ratios for some variables (currently only the PUF data set includes a ratios file);

  5. a benefits file containing extrapolated benefits for each filing unit (currently only the CPS data set includes a benefits file).

Note that the factors file is the same in both sets of data files because the variable extrapolation factors are independent of the sample data being used. But the weights, ratios, and benefits files do depend on the data file, so they are different in the two sets of data files.

Data-Preparation Documentation and Workflow

The best documentation of the data-preparation workflow is the taxdata Makefile. The Makefile shows the input files and the Python script that generates each made file. The files made in early stages of the workflow serve as input files in later stages, which means there is a cascading effect of changes in the scripts and/or input files. The Makefile automates this complex workflow in an economical way because it executes scripts to make new versions of made files only when necessary. Start exploring the Makefile by running the make help command. If you want more background on the make utility and makefiles, search for Internet links with the keywords makefile and automate.

Note that the stage2 linear program that generates the weights file is very long-running, taking five or more hours depending on your computer's CPU speed. We are considering options for speeding up this stage2 work, but for the time being you can execute make puf-files and make cps-files in separate terminal windows to have the two stage2 linear programs run in parallel. (If you try this parallel execution approach, be sure to wait for the make puf-files job to begin stage2 work before executing the make cps-files command in the other terminal window. This is necessary because the CPS stage1 work depends on output from PUF stage1.) If you are generating the taxdata made files in an overnight run, then simply execute the make all command.

You can copy the made files to your local Tax-Calculator directory tree using the csvcopy.sh bash script. Use the dryrun option to see which files would be copied (because they are newer than the corresponding files in the Tax-Calculator directory tree) without actually doing the file copies. At the terminal command-prompt in the top-level taxdata directory, execute ./csvcopy.sh to get help.

Contributing to taxdata Repository

Before creating a GitHub pull request, on your development branch in the top-level directory of the taxdata repository, run make cstest to make sure your proposed code is consistent with the repository's coding style and then run make pytest to ensure that all the tests pass.

Contributors

  • Anderson Frailey
  • John O'Hare
  • Amy Xu
  • Martin Holmer