Replication code — The Missing Value of Data

This repository reproduces every figure and table of The Missing Value of Data (Bhutani, Ordoñez, Veldkamp). One command runs the whole pipeline from raw inputs to the paper-ready figures, tables, and inline numbers.

If you use this code, please cite the paper — see License and citation.

Quick start

1. Get the code. Clone the repository, or download a ZIP (the green Code ▸ Download ZIP button) and unzip it:

   cd BOV_missing_value_data

2. Install Miniconda if you don't already have conda. Miniconda is a small, free program that provides Python and the conda environment manager, so you do not need to install or configure Python yourself. Download it from https://www.anaconda.com/docs/getting-started/miniconda/install and run the installer for your system (on macOS, download the .pkg, double-click it, and follow the prompts). Then open a new terminal window so the conda command is available. If you already have Anaconda or Miniconda, skip this.

3. Create the environment. Run these once to build the tested environment; afterwards, just conda activate value_data at the start of each session. The first time, conda may ask you to accept a couple of agreements — follow the prompt to accept.

   conda create -n value_data python=3.11 -y
   conda activate value_data
   pip install -r requirements.txt

4. Place the raw data in data/raw/. The data is not included — you obtain each input yourself from its provider. See Input data for the full list of files and where to get them, and Data format requirements for the columns each file must have.

5. Run the pipeline:

   python run_pipeline.py

   This produces all the figures and regression tables used in the paper. To also write the inline prose numbers (outputs/numbers.csv), add --emit-numbers to the command above (python run_pipeline.py --emit-numbers). A plain run takes about 5–8 minutes depending on the machine.

Stata is optional but recommended. The pipeline produces every table and number without it. Installing Stata reproduces the exact standard errors reported in the paper — see Stata (recommended).

A command isn't found, or an install error? See Troubleshooting.

What we used. All results in the paper were produced with Python 3.11 and the package versions pinned in requirements.txt; the steps above reproduce that environment. A different Python version or another way of installing the packages may well work too; if something does not, the environment is the first place we would look. It does not matter which Python version your base Miniconda happens to use (it may be 3.13, for example) — conda create … python=3.11 builds its own Python 3.11 just for this project.

Input data

The pipeline reads its inputs from data/raw/, so the data must be in place before you run it. The data is not included — every file below is one you obtain yourself from its provider and place in data/raw/:

File	Source
compustat.csv	Compustat North America — Fundamentals Annual (WRDS), USA only. We recommend pulling fiscal years 1999–2025 (the pipeline filters to its 2001–2022 study window internally). Required columns: gvkey, datadate, fyear, sale, at, emp, ib, ppegt, sic, cogs, indfmt, naics
link.dta	CRSP-Compustat link table (WRDS)
ibes_data.csv	I/B/E/S Management Guidance (Detail table) (WRDS)
crsp_ibes_link.csv	CRSP-I/B/E/S ticker link (WRDS)
CPIAUCSL.csv	FRED — CPI all items, monthly: https://fred.stlouisfed.org/series/CPIAUCSL
employment_CE16OV.csv	FRED — Employment Level: https://fred.stlouisfed.org/series/CE16OV (annual frequency)
GDPC1.csv	FRED — Real GDP: https://fred.stlouisfed.org/series/GDPC1. The code assumes that the GDP numbers are in Billions of Chained 2017 Dollars which they were at the time of download. Please adjust the input if this has changed at the time you download.
OEWS oews/all_data_M_*.xlsx	BLS Occupational Employment and Wage Statistics national files — see the dedicated note below

Drop each single-file input directly into data/raw/ (e.g. data/raw/compustat.csv).

OEWS files (data/raw/oews/)

The OEWS occupational-wage inputs are a special case: they are many files that go in their own folder. Create the folder data/raw/oews/ and place each year's national file in it as an individual .xlsx, named exactly as the pipeline expects (all_data_M_2012.xlsx … all_data_M_2022.xlsx):

data/raw/oews/
  all_data_M_2012.xlsx
  all_data_M_2013.xlsx
  …
  all_data_M_2022.xlsx

To obtain them, open the BLS OEWS data-tables page https://www.bls.gov/oes/tables.htm. For each May release from 2012 through 2022, download the national "All data" file — a ZIP named oesmYYYYall.zip (for example, https://www.bls.gov/oes/special-requests/oesm2022all.zip for 2022) — unzip it, and copy the all_data_M_YYYY.xlsx it contains into data/raw/oews/. Place only the individual .xlsx files there — not the ZIP and not a per-year subfolder. The folder must be named oews exactly (this is the path run_pipeline.py looks in). code/06_oews_prep.py builds data/derived/oews_ibes_compustat.csv from these files.

Data vintage. The Compustat and I/B/E/S extracts used in the paper were downloaded in February 2025. WRDS revises these datasets over time, so results can drift slightly between vintages. A replication with data re-downloaded in June 2026 showed the drift to be minor — less than 1% for most numbers reported in the paper — and all figures, tables, and qualitative results were unchanged.

Optional inputs

You can reproduce the core results without obtaining every dataset. If an optional input is absent, the pipeline prints a notice, skips just the parts that need it, and completes:

• OEWS files missing → Tables 7, 8, 9 are not generated, and the calibration uses the hard-coded data-worker wage-share target 0.004 (the value the OEWS computation rounds to). When the OEWS files are present, that target is instead computed from the data and used exactly as before.
• GDP or employment missing (GDPC1.csv / employment_CE16OV.csv) → Figure 5 (the economy-wide missing-GDP share) is not generated; Figure 4 and everything before it still are.

A missing core input (compustat.csv, link.dta, ibes_data.csv, crsp_ibes_link.csv, CPIAUCSL.csv) stops the run up front with a clear message naming the missing file. Inline numbers that depend on an absent optional input are simply left blank in outputs/numbers.csv; the run still completes.

Data format requirements

Each stage validates its raw inputs the moment they are read (lib/data_checks.py) and stops with a clear error if a file is missing a required column, has a non-numeric value in a numeric column, or has a date that does not parse in the expected layout. Supply the files in these formats:

File	Required columns (consumer-referenced)	Numeric	Dates (format)
compustat.csv	gvkey, datadate, fyear, sale, at, emp, ib, ppegt, sic, cogs, indfmt, naics	gvkey, fyear, sale, at, emp, ib, ppegt, sic, cogs	datadate (YYYY-MM-DD)
link.dta	gvkey, LPERMNO, LINKPRIM, LINKDT, LINKENDDT, FIC	gvkey, LPERMNO	— (Stata dates)
ibes_data.csv	ticker, prd_yr, measure, pdicity, range_desc, val_1, val_2, actdats, anndats	prd_yr, range_desc, val_1, val_2	actdats, anndats (YYYY-MM-DD)
crsp_ibes_link.csv	TICKER, PERMNO, SCORE, sdate, edate	PERMNO, SCORE	sdate, edate (YYYYMMDD)
CPIAUCSL.csv	observation_date, CPIAUCSL	CPIAUCSL	observation_date (YYYY-MM-DD)
employment_CE16OV.csv	fyear, total_emp	fyear, total_emp	—
GDPC1.csv	observation_date, GDPC1	GDPC1	observation_date (YYYY-MM-DD)
OEWS all_data_M_*.xlsx	NAICS, OCC_CODE, A_MEAN, TOT_EMP (case-insensitive; presence only)	— (BLS top-codes A_MEAN with #/*, handled in cleaning)	—

Missing/blank cells are allowed (the pipeline drops or fills them); only a present value of the wrong type fails. Columns beyond those listed are ignored.

What a run produces

Everything under outputs/ is regenerated on every run: the figures (outputs/figures/*.png), the state-space estimates (outputs/estimates/), the regression tables (outputs/tables/), and the calibrated-model pickle (outputs/model_run.pkl). The inline prose numbers (outputs/numbers.csv) are produced only with --emit-numbers, and the per-figure data CSVs only with --emit-figure-numbers.

python run_pipeline.py --help lists the options. By default the screen shows only which stage is running and the figures/numbers each one produces; pass --verbose to mirror the full per-stage output to the screen (it is always written in full to a timestamped log under logs/). --full-calibration runs the full calibration grid search instead of the fast default and takes longer (~30 minutes).

Stata (recommended)

The regressions behind Tables 3, 4, 7, 8, 9 are run in Python (code/05_gp_regs.py, code/09_oews_regs.py, via pyfixest) on every pipeline run, so you get these tables — and every paper number — whether or not you have Stata. Installing Stata is recommended for consistency with the published tables: in the paper we report the standard errors produced by Stata rather than those from Python's pyfixest. The original Stata .do files are included and produce the .tex tables, and run_pipeline.py runs them automatically if it can find Stata:

• It looks for Stata in the standard place the installer puts it (on macOS, /Applications/Stata*/…; on Windows, C:\Program Files\Stata*\…; on Linux, /usr/local/stata*/…), so a typical GUI install is detected automatically.
• It also accepts a stata / stata-mp executable on PATH.
• If your Stata lives somewhere non-standard, set the STATA_EXE environment variable to its executable (e.g. STATA_EXE=/path/to/stata-mp python run_pipeline.py).

Required Stata packages. The .do files use the user-contributed commands reghdfe (high-dimensional fixed-effects regressions, which depends on ftools and require) and estout (eststo / esttab, which write the .tex tables). If you have Stata, install these once from within Stata before running (skip any you already have):

ssc install ftools
ssc install require
ssc install reghdfe
ssc install estout

If a .do file runs but a required package is missing, it aborts partway and run_pipeline.py reports it explicitly (e.g. Stata .do FAILED (r(199); …)) but does not stop the run — the Stata stages are non-fatal because the Python stages already produce every one of these tables. If Stata cannot be found at all, the .do files are simply skipped with a notice. Either way nothing is lost. The Stata batch logs are written under logs/.

Pipeline overview

Tracing inputs → outputs, the paper's results come from this chain (run in order by run_pipeline.py):

01_data_prep.py ──────────► data/derived/capped_ibes_compustat.csv  + Figure 1 (SFE distribution)
                            + data/derived/{real_sale_…, ibes_window_panel}.csv  (state-space-estimation panels)
02_state_space_mle.py ────► outputs/estimates/state_space_params.csv (Table 1: ρ, σ²_ε, σ²_η, σ²_α)
03_prior_fe_predict.py ───► outputs/estimates/prior_fe_params.csv    (prior fixed-effects + calibration constants)
                            + outputs/estimates/predicted_theta_log.csv (predicted θ series)
04_data_decomp.py ────────► Figures 2, 3
05_gp_regs.py(.do) ───────► Tables 3, 4
06_oews_prep.py ──────────► data/derived/oews_ibes_compustat.csv        (optional; needs OEWS data)
07_calibration.py ────────► outputs/model_run.pkl                    (Tables 5, 6)
08_value_calcs.py ────────► Figures 4, 5
09_oews_regs.py(.do) ─────► Tables 7, 8, 9                              (optional; needs OEWS data)
10_bought_data.py ────────► Appendix-E numbers
aggregate_paper_numbers.py► outputs/numbers.csv      (inline paper numbers; only with --emit-numbers)

Importable library code (not pipeline stages) lives in lib/: lib/common.py (shared calibration constants + the per-year aggregator), lib/pooled_kalman.py (the Kalman filter / RE-MLE primitives), and lib/plotting.py (every figure helper). 02_state_space_mle.py estimates the state-space model by maximum likelihood (a Kalman filter); 03_prior_fe_predict.py reuses those parameters to form the per-firm prior fixed-effects and the predicted-θ series the value-of-data calculation consumes.

All pipeline scripts below live in code/.

Paper artefact	Producing script	Notes
Figure 1	01_data_prep.py	Python (SFE-distribution summary of the prepared data)
Figures 2, 3	04_data_decomp.py	Python
Figures 4, 5	08_value_calcs.py	Python (Figure 5 needs GDP + employment)
Table 1	02_state_space_mle.py	Python (state-space MLE)
Table 2	multiple sources	Python; combines several scripts with judgments drawn from the literature
Tables 5, 6	07_calibration.py	Python
Appendix-E numbers	10_bought_data.py	Python
Tables 3, 4	05_gp_regs.py (+ .do)	Python (+ Stata, recommended)
Tables 7, 8, 9	09_oews_regs.py (+ .do)	Python (+ Stata, recommended; needs OEWS data)

Paper numbers (--emit-numbers)

Paper-number generation is opt-in. A plain run_pipeline.py produces the figures and tables; add --emit-numbers to also write outputs/numbers.csv — the paper's inline prose numbers as live values, each rounded to the precision the paper printed, with a paper_locator column saying where each appears. Numbers that simply restate a regression-table cell (Tables 3, 4, 7, 8, 9) are not included — read those directly from the tables in outputs/tables/.

Troubleshooting

• conda: command not found. Miniconda is not installed yet, or you need to open a new terminal window after installing it (the installer only takes effect in shells opened afterward). See Quick start.
• A pyfixest or numpy "Could not find a version that satisfies…" error, or a Python-version message from run_pipeline.py. You are almost certainly not in the value_data environment. Run conda activate value_data, confirm python --version reports 3.11, and try again. (These errors mean the command ran against a different Python — usually the system one.)

Cross-platform notes

The Python pipeline uses pathlib throughout and is tested on macOS / Linux. It should run unchanged on Windows under any modern Python 3.10+. The Stata .do files use relative paths and assume the working directory is the repo root, which run_pipeline.py ensures.

License and citation

The code in this repository is released under the BSD 3-Clause License (see LICENSE).

No data is included with this repository. You must obtain every input yourself from its original provider — both the licensed extracts (Compustat, I/B/E/S, CRSP, obtained through a WRDS subscription) and the public series (FRED, BLS OEWS) — and place it under data/raw/ (see Input data).

If you use this code, please cite the paper (machine-readable metadata is in CITATION.cff):

Ankit Bhutani, Guillermo Ordoñez, and Laura Veldkamp, "The Missing Value of Data," NBER Working Paper 35266 (2026), https://doi.org/10.3386/w35266.