Working together on one analysis — a git and GitHub exercise

About

This repository is a training exercise.

Three people split one household-survey analysis between them, work at the same time on separate branches, and then have to put the pieces back together. The analysis itself is small and the Stata is ordinary. The difficulty is entirely in the last step: three branches that all changed the same four files, and one working pipeline that has to come out of the merge.

You will practise: branching, committing, opening a pull request, reviewing someone else's code, hitting a merge conflict, and resolving it without breaking what the other two wrote.

The pipeline is the MML template pipeline: three stages, each reading only what the previous stage produced. 00_master.do is the only entry point.

The data

The practice survey is the Republic of Examplandia Household Budget Survey 2024 (EXC-HBS-2024).

It is synthetic: the country, the survey, the people and the prices are all invented. It was built to behave like a real household budget survey, so poverty, inequality and Engel-curve calculations give sensible answers, but nothing in it may ever be quoted as a real statistic.

It lives in the shared read-only archive, in the folder EXAMPLE-COUNTRY, and consists of four linked files:

File	Rows	One row is	Key
exc_hbs2024_household.dta	10,000	a household	hhid
exc_hbs2024_individual.dta	44,950	a person	hhid pid
exc_hbs2024_consumption.dta	60,000	a household and one of six goods	hhid citem
exc_hbs2024_income.dta	19,685	an employed person aged 15-64	hhid pid

The archive folder holds its own README.md with the full data dictionary, the deliberate imperfections that were built in, and the statistics the data should produce. Read it before you start: it is the answer key for "is my number right?".

00_init.do names the four files as $fRawHH, $fRawInd, $fRawCons and $fRawInc, so no script has to spell out a path.

Before you start

1. Open code/00_init.do and add a block for your own Windows username, copying the one that is already there. Set gdProject to your clone of this repository and gdArchive to your synced copy of the shared data archive. This is the only machine-specific edit in the whole repository.
2. Run code/00_master.do. It stops in 02_sim.do and says it could not find any pre-simulation product. That is the intended starting point, not a bug: the pre-sim block in 00_master.do is empty, so nothing has been produced yet.
3. Add your own do line to that block and run again. Now your own pre-simulation file stops and prints the list of variables it could not find. That list is your to-do list, and it shrinks as you work.

Stata 16 or newer is required, because the pipeline keeps parameters in a frame.

---

The assignment

Three roles

Each person owns one file, reads two of the four raw files, and produces one household-level table. Every product has one row per household and exactly 10,000 rows, keyed on hhid.

	Person A	Person B	Person C
Topic	Demography	Labour income	Consumption
Your file	code/01a_presim_demographics.do	code/01b_presim_income.do	code/01c_presim_consumption.do
You read	$fRawHH, $fRawInd	$fRawHH, $fRawInc	$fRawHH, $fRawCons
You write	$gdPresim/hh_demographics.dta	$gdPresim/hh_income.dta	$gdPresim/hh_consumption.dta

All three of you read the household file. It is the frame: it decides which households exist, and starting from it is what guarantees 10,000 rows.

Then, together, you merge the three tables in code/02_sim.do.

code/03_postsim.do is given, finished, and nobody edits it. It produces two tables and seven figures, and it is written against the variable names below and nothing else. If your names are right it runs untouched; if they are not it stops and tells you which name is wrong.

The variable contract

This is the part that makes parallel work possible. Three people cannot check each other's code while they are writing it, so they agree the names up front instead.

Each pre-simulation file ends with a call to mml_check_vars, which refuses to save a file that does not deliver every name below.

Person A — hh_demographics.dta

This file also carries the survey design variables for the whole pipeline. 02_sim.do may only read $gdPresim, so if hhweight does not come through here it never reaches the outputs at all.

Variable	What it is
hhid	household identifier, unique
region urban stratum psu hhweight interview_month	carried through from the household file, unchanged
hhsize	household size as the household reported it
n_members	people actually on the roster
n_male n_female	members by sex
n_age_0_14 n_age_15_24 n_age_25_44 n_age_45_64 n_age_65p	members by age band
n_working_age	members aged 15-64, that is the three middle bands
n_dependents	members under 15 or 65 and over
dep_ratio	n_dependents / n_working_age, missing when there is no working-age member
n_head	number of heads, which must be 1
head_age head_male head_educ	characteristics of the head

The five age bands must add up to n_members, and so must n_male + n_female.

Person B — hh_income.dta

All values are monthly EXS, summed over every earner in the household. 02_sim.do turns them into annual figures; do not do it here, or the total gets multiplied by twelve twice.

Variable	What it is
hhid	household identifier
n_earners	people in the household who work
inc_job1 inc_job2	household income from main jobs and from second jobs
inc_agri inc_manu inc_serv	household income by industry of the main job
inc_formal inc_informal	household income by formality of the main job
inc_total	total household labour income

Both splits are of the same total, so inc_agri + inc_manu + inc_serv and inc_formal + inc_informal must each equal inc_total.

Person C — hh_consumption.dta

All values are annual EXS.

Variable	What it is
hhid	household identifier
c_cereals c_meatfish c_fruitveg	the three food items
c_clothing c_transport	the two non-food items
c_rentutil	the housing item
c_food c_nonfood c_housing	the three groups the six items nest in
c_total	total annual household consumption
n_citems	goods with non-zero spending, out of 6

The six items must add up to c_total, and so must the three groups.

What 02_sim.do adds

You do not have to build these, but 03_postsim.do needs them, so the merge is not finished until they are there: inc_total_year, c_pc, inc_pc, ln_c_pc, ln_inc_pc, share_food, share_nonfood, share_housing, share_agri, share_formal, share_job2, dec_c_pc, poor, inc_total_sim, inc_pc_sim.

dec_c_pc is the decile of per capita consumption that every table and figure is grouped by. It is weighted by hhweight * n_members, so each decile holds a tenth of the population, not a tenth of the households.

02_sim.do also uses the three parameters you each add to inputs/parameters.csv: p_n_deciles (Person A), p_months_per_year (Person B) and p_poverty_line (Person C). A row left out of the csv stops the run there and names the parameter, so all three rows have to survive the merge.

Three traps, one per person

Each role has one thing in the data that gives a wrong answer quietly if you miss it. None of them produces an error message.

Person A — a missing value is bigger than any number. Writing age >= 65 puts every person with a missing age into the top band, because in Stata a missing value sorts above every number. Write (age >= 65) & !missing(age).

Person B — 310 households have nobody in work. The earnings file only holds households where somebody works, so it has 9,690 households, not 10,000. Those 310 households are real, they have zero labour income, and dropping them deletes the poorest part of the distribution. Separately, income_job2 is missing, not zero, for the 82% of earners with only one job.

Person C — 45 consumption entries are exactly zero. Zero is a real answer: the household spent nothing on that good. It is not a missing value, and dropping those rows makes n_citems meaningless.

The four shared files

These four files are edited by all three of you, in the same place. That is deliberate. Whoever merges first has an easy time; the second and third will get conflicts, and resolving them is the exercise.

File	What each person adds
code/00_master.do	one do line, inside the marked pre-sim block
code/02_sim.do	one merge block, inside the marked merge block
inputs/parameters.csv	one parameter row at the end
README.md	one row in the table below

When you hit a conflict in one of these, the answer is almost always keep both sides, in the documented order. It is almost never "take mine".

Outputs produced

Add your row when your part works.

Person	File produced	Rows	Key variables
TODO: Person A
TODO: Person B
TODO: Person C

The git workflow

1. git switch -c yourname-demographics — one branch per person, off main.
2. Write your pre-simulation file. Commit early and often; small commits are easier to merge than one big one.
3. Add your line to each of the four shared files.
4. git push -u origin yourname-demographics, then open a pull request on GitHub.
5. Ask one of the other two to review it. Reviewing someone else's code is half the exercise.
6. Merge it.
7. The other two now pull main and merge it into their branch. This is where the conflicts appear. Resolve them, check the pipeline still runs, push again.

Do not merge your own pull request without a review, and do not force-push a shared branch.

How to check your work

Run code/00_master.do. If it finishes, compare your numbers against the archive README.md, which lists what the data should produce. These are the ones worth checking:

Statistic	Should be
Households / people represented	4,199,769 / 19,364,350
Urban share, sample then weighted	44% then 34%
Median annual consumption per capita, unweighted	249,861 EXS
Gini of consumption per capita, weighted	0.414
Food share, poorest decile then richest	0.50 then 0.28
Mean household size, rural then urban	5.0 then 3.8

If the weighted and unweighted columns of tab_summary.csv are identical, you lost hhweight somewhere.

The worked answer is on the solutions branch. Look at it after you have tried, not before.

---

How the pipeline works

00_master.do is the only entry point. It initialises the environment, empties the run folders, and calls each stage in order.

Script	Reads	Writes
code/00_init.do	inputs/	paths, $suser run identity, $p_* parameters
code/00_ado.do	—	installs external ado, puts code/ado/ on the adopath
code/01a_presim_demographics.do	the raw archive	$gdPresim/hh_demographics.dta
code/01b_presim_income.do	the raw archive	$gdPresim/hh_income.dta
code/01c_presim_consumption.do	the raw archive	$gdPresim/hh_consumption.dta
code/02_sim.do	$gdPresim	$gdSim/hh_analysis.dta
code/03_postsim.do	$gdSim	$gdPostsim, $gdOut (two tables, seven figures)

Each stage reads only its predecessor's product folder, so any stage can be re-run on its own from what is already on disk. Every script starts with the same self-initialising block, so stages also run standalone. The *-temp folders are scratch space: the master empties them on every run and nothing downstream may read them.

Produced data lands in runs/<username>/, which is gitignored and disposable. Raw data lives outside the repository and is never written to.

Helper commands (code/ado/)

00_ado.do puts code/ado/ on the adopath, so these are available in every stage. help mml_mkdir and friends document each one.

Command	What it does
mml_mkdir	Creates a folder and every missing parent above it.
mml_clean	Erases every file below a folder but keeps the folders; refuses any path outside $gdRun.
mml_check_dir	Reports every missing folder at once; warns by default, fail to stop.
mml_check_global	Reports every global in a list that is not set; warns by default, fail to stop.
mml_check_vars	Reports every variable missing from the data in memory. This is what enforces the contract above.
mml_csv_to_globals	Turns a name,value csv into globals, optionally with prefix().

Analysis inputs (inputs/)

Growth rates, elasticities, equivalence scales, poverty lines and scenario definitions are assumptions about the analysis, not data. They live in inputs/, they are committed to git, and git log inputs/ is the record of which assumptions produced which results. 00_init.do loads both files below once, so every stage sees the same values.

inputs/parameters.csv — flat scalars. Columns name,value,description, and every row becomes a global ${p_<name>}. This is one of the four shared files: each person adds one row, and that is where the conflict happens.

inputs/parameters.do — when a parameter needs code. Use it when the value is derived from another parameter, is a list rather than a scalar, or is a table that a name,value file cannot express. The example sets the active scenario $p_scenario and builds a scenario × multiplier × label table into the Stata frame params, which 02_sim.do and 03_postsim.do read. Switching p_scenario from high to low changes the results without editing a stage.

Data does not go here. Raw microdata lives in $gdRaw, a read-only archive outside the repository, and is never committed.

One Stata trap worth knowing before you start

Never write a slash immediately followed by a star inside a comment, not even in a path such as a folder name followed by a wildcard. Stata reads it as the opening of a block comment wherever it appears, and with no closing pair the rest of the file is silently ignored. The stage then runs, prints nothing, writes nothing and reports no error at all.

License

This project is licensed under the MIT License together with the World Bank IGO Rider. The Rider is purely procedural: it reserves all privileges and immunities enjoyed by the World Bank, without adding restrictions to the MIT permissions. Please review both files before using, distributing or contributing.