ECV — Encuesta de Condiciones de Vida: data processing pipeline (Python/Polars)

Builds harmonised person-year, household-year and longitudinal panel tables, plus a validated series of poverty and inequality indicators, from the INE Encuesta de Condiciones de Vida (ECV) microdata — Spain's component of EU-SILC. Cross-sections cover 2004–2025; the longitudinal panel covers 2008–2025.

Same design philosophy as mvazcar/MCVL and mvazcar/panel-de-hogares: a single config.py, a readers.py of era-aware readers, numbered stepNN_*.py modules each writing a parquet checkpoint, a pipeline.py orchestrator running the independent steps in parallel subprocesses, and a thin run.py CLI.

Two things are different here. First, ECV microdata is public, so download.py fetches all 50 archives straight into raw_zipped/ and the whole pipeline is reproducible from a clean checkout — no data request, and no external system dependency (py7zr handles the three .7z archives in pure Python, where Panel de Hogares needs UnRAR). Second, INE ships its own derived indicators, which gives us something rare: the pipeline recomputes them from the raw items and checks its answers against INE's, so a methodology change surfaces as a failing check rather than a quietly different number.

Pipeline: input to output

raw_zipped/T2013__2020.zip                 50 archives, exactly as INE ships them
      │
      │  normalize_filenames.py            unzip (zip-in-zip, 7z), rename to one convention
      ▼
raw/T2013/2020/ECV2020D.csv                200 data files: 4 per archive, D H R P
      │        ECV2020H.csv                +5 companions (2014/15 AROPE, 2021 RK, 2005 income)
      │        ECV2020R.csv
      │        ECV2020P.csv
      │
      │  step01_household   D ⋈ H                  step03_indicators  AROP/AROPE/Gini/S80-S20
      │  step02_person      R ⋈ D ⋉ P              step04_panel       longitudinal releases
      ▼
temp/*.parquet                             one checkpoint per step, steps 01-04 run in parallel
      │
      │  step05_merge
      ▼
output/ecv_person_year.parquet   ecv_household_year.parquet   ecv_panel_person.parquet
       ecv_indicators.parquet    ecv_indicators_ca.parquet

After normalization nothing downstream ever guesses a filename: config.raw_path(key, letter) names the file directly, the way MCVL's raw_path(year, filetype, part) does. Every piece of INE's naming chaos is absorbed once, in normalize_filenames.py.

What is the ECV?

An annual survey of ~15,000–30,000 Spanish households, published by INE since 2004 as Spain's EU-SILC instrument. It is a four-year rotating panel: each household stays in the sample for four waves. Income refers to the calendar year before the interview, so ECV 2025 reports 2024 income. Four files per release, following the EU-SILC convention:

file	contents
D	datos básicos del hogar — household register: ids, region, weights
H	datos detallados del hogar — income (HY*), deprivation items, tenure, derived HX*
R	datos básicos de la persona — every member: age, sex, weights, INE's derived vr* flags
P	datos detallados de los adultos (16+) — activity, education, personal income (PY*)

INE publishes four families, all handled here: transversal and longitudinal, each under base 2004 (2004–2012) and base 2013 (2008–2025).

Repository structure

ECV/
  config.py                archive catalogue + URLs, year ranges, raw_path(), NUTS2 -> CA
  download.py              fetch the 50 archives from INE      -> raw_zipped/  (public, ~1 GB)
  normalize_filenames.py   unzip + rename to one convention    -> raw/         (pure Python)
  parse_disenos.py         parse the "Diseño de Registro" workbooks -> catalog/
  build_catalog.py         walk raw/ -> catalog/ inventory + variable presence matrix
  readers.py               one reader per file letter; lossless casting
  harmonise.py             recode variables across the 2009/2011/2021/2023 breaks
  indicators.py            AROP, SMD, SMSD, LWI, AROPE, Gini, S80/S20, poverty gap

  step01_household.py      D join H            -> temp/household_year_<base>.parquet
  step02_person.py         R join D, left P    -> temp/person_year_<base>.parquet
  step03_indicators.py     indicators + validation against INE's own columns
  step04_panel.py          longitudinal person panel from the 15 base-2013 releases
  step05_merge.py          final assembly      -> output/

  pipeline.py              orchestrator: steps 01-04 in parallel subprocesses, then merge
  run.py                   CLI: --steps / --serial / --workers / --years / --fetch

  catalog/                 schemas.json, value_labels.json, footnotes.json, presence.csv
  raw_zipped/              the 50 archives exactly as INE ships them        [gitignored]
  raw/<BASE>/<PERIOD>/     normalized ECV<PERIOD><LETTER>.csv               [gitignored]
  disenos/  temp/  output/                                                  [gitignored]

Quick start

pip install -r requirements.txt

python download.py                       # 50 archives from INE -> raw_zipped/  (~1 GB, idempotent)
python normalize_filenames.py            # dry run: print the rename plan, write nothing
python normalize_filenames.py --execute  # unzip + rename -> raw/   (205 data files)
python parse_disenos.py                  # design workbooks -> catalog/
python run.py                            # build -> temp/ -> output/   (~25 s on a laptop)

Or in one go: python run.py --fetch.

If you already have the archives, skip download.py and drop them in raw_zipped/ named <BASE>__<PERIOD>.zip (e.g. T2013__2020.zip, L2013__2022-2025.zip).

Every stage is idempotent, so re-running is cheap. python run.py --steps 03 05 rebuilds just the indicators and re-assembles.

Output

file	rows	cols	contents
ecv_person_year.parquet	780,780	149	person-year cross-sections, base 2013, 2008–2025
ecv_household_year.parquet	306,240	481	household-year cross-sections, base 2013
ecv_panel_person.parquet	1,476,952	34	longitudinal panel, keyed (PERIOD, RB030, YEAR)
ecv_indicators.parquet	18	15	national indicator series
ecv_indicators_ca.parquet	342	6	indicator series by comunidad autónoma

The household-year table is wide because INE rotates a different ad-hoc module through the H and P files every year; the union of eighteen years' columns is 481 wide and mostly null off-diagonal. See VARIABLES.md.

Validation

The pipeline recomputes each indicator from the raw items and compares it with the derived column INE ships in the R file. Both checks are exact in every year where INE ships the comparison column (2014–2025), across 12 years:

check	result
severe material and social deprivation, recomputed from the 13 items vs vrMATSOCDEP	100.000% agreement, 0 disagreements
AROPE recomputed as AROP ∪ SMSD ∪ LWI vs vrEU2030_nuevo	100.000% agreement, 0 disagreements

And against INE's published national figures:

indicator	pipeline	INE published
at-risk-of-poverty rate, 2025	19.47%	19.5%
at-risk-of-poverty rate, 2024	19.67%	19.7%
AROPE (EU2030), 2025	25.67%	25.7%
AROPE (EU2030), 2024	25.81%	25.8%
AROPE (EU2020), 2008	23.78%	23.8%
S80/S20, 2025	5.24	5.2

The regional series reproduces INE's published tasa de riesgo de pobreza por comunidad autónoma for 2025 to the published precision, across the whole range — País Vasco 9.28% (INE 9.3), Navarra 12.34% (12.3), Extremadura 26.23% (26.2), Andalucía 27.74% (27.7), Melilla 39.3% (39.3).

temp/validation.parquet carries the per-year agreement rates; step05 prints the minimum and labels it exact or CHECK.

Data notes — the traps

These are the places where the obvious implementation is silently wrong. Each was found by checking the data, not by reading the documentation.

Consecutive cross-sections cannot be linked into a panel. In the transversal files DB030 is a sequential counter that INE re-assigns every year (the workbook calls it the identificación transversal), and RB030 = DB030*100 + person_number inherits that. Joining the 2020 and 2021 cross-sections on RB030 produces 20,735 "matches" of which 51.2% agree on sex and 1.9% on year of birth — a coin flip. Use the longitudinal files, which have their own stable ids.

The longitudinal releases must not be stacked on RB030 either. Ids are reused across releases that do not overlap in time. Stack all fifteen and you get "persons" observed in 5–11 waves, which a four-year rotating panel cannot produce; the long spans disagree on sex ~50% of the time. The panel key is (PERIOD, RB030), exposed as PERSON_UID. Since nobody stays more than four waves, a trajectory always fits inside one release and there is nothing to gain from chaining.

A person-year can have two rows, and exit rows are not obviously exit rows. Someone who moves house mid-year appears in the origin household with RB110 = 5 (moved out) and in the destination with RB110 = 2 (moved in). Filter on IS_MEMBER (RB110 ∈ {1,2,3,4}) before counting people — and in particular before counting household members. It is tempting to drop exit rows by dropping null ages, because in the three newest releases they do carry a null RB080. In the other twelve they carry a birth year. Counting them inflates household size, hence the equivalence scale, hence understates equivalised income: on the 2016–2019 release it overstates the scale for 4.1% of households by 41% on average, and across the panel it falsely flagged 5,235 person-years as poor.

Renaming variables across the 2021 break corrupts them. Regulation (EU) 2019/1700 changed names and code lists, and not together. PL015 (1 = Sí, 2 = No) became PL016 (1 = No, 2 = Sí ocasional, 3 = Sí regular) — a rename inverts every answer. PL140 (1 = indefinido, 2 = temporal) became PL141 (11, 12 = temporal; 21, 22 = indefinido) — disjoint codes, opposite order. HH020 has four tenure categories, HH021 five, and in the longitudinal files both are populated on the same row. harmonise.py therefore never renames: it recodes each generation into a canonical variable with a documented code list, then coalesces.

INE's design workbooks contain errors. PL040A is labelled "último empleo principal" and PL040B "empleo principal actual"; the data say the opposite — in every year 2021–2025, PL040A is filled for ~99% of respondents reporting PL032 = 1 (currently working). PL145 is labelled "Tipo de contrato (duración)" but its codes are tiempo completo / tiempo parcial. Trust the data and Eurostat's codebook.

Base 2004 and base 2013 are not comparable in levels. From base 2013 INE builds the income variables from administrative registers rather than the interview. For survey year 2010 the mean equivalised income is 14,320 € under base 2004 and 16,686 € under base 2013 — a 16% level shift — while the Gini moves only from 0.345 to 0.332. Never mix the two bases in one series. Base 2013 is the long consistent series (2008–2025); base 2004 is kept for 2004–2007 back-history.

vrLOWJOB = 9 means "not applicable", not "no". It marks persons above the age ceiling (60+ for the old definition, 65+ for the new). Averaging the raw column silently mixes 9s into a rate.

The 2014–2015 EU2030 indicators live in a separate file. esudb14r_NuevoArope.csv and its 2015 twin carry vrMATSOCDEP, vrLOWJOB_nuevo and vrEU2020_nuevo, one row per RB030. read_r() joins them, which is what extends the AROPE-EU2030 series back to 2014.

esudb21rk.csv is not an R file. 632 rows, RK* columns, no RB030: a 2021-only module on non-resident children, with its own population. It must never be joined onto the R file.

Delimiters lie. ECV_T*_<year>.csv is tab-separated despite the .csv extension (2016–2022), and the two newest longitudinal releases ship only .tab. readers.py sniffs the header rather than trusting the name, and strips padding: 2025's vrMATSOCDEP holds 958 single-space values that would otherwise survive as a third, silent category.

Raw file naming quirks (why normalize_filenames.py exists)

Two decades of INE archiving. The script walks every archive recursively rather than special-casing years, and maps whatever it finds onto ECV<PERIOD><LETTER>.csv. This is what it absorbs:

archive	original naming	problem
T2013/2008	datos_ecv2008FA.7z	data ships as a loose .7z inside the outer zip
L2013/2008-2011	es11L.7z	same
T2013/2019	disreg_ecv19.7z	design ships as a loose .7z
T2013/2018	disreg_ecv18zip.zip	typo: 18zip
T2013/2024	disreg_ecv_2024.zip	stray underscore; CSVs duplicated under CSV/ and R/
T2013/2024, /2025	dr_ECV_CM_Th_2025.xlsx	design workbooks gain a CM_ prefix
L2013/2019-2022	ECV_Lh_19a22.zip.zip	double extension
T2013/2016..2020	esudb18r_AROPE.csv	R file renamed; there is no plain r file
T2013/2014, /2015	esudb14r_NuevoArope.csv	EU2030 indicators ship outside the R file
T2013/2021	esudb21rk.csv	extra file, different population (non-resident children)
T2004/2005	inghog05.csv, ingper05.csv	extra income detail, base-2004 only
T2013/2016..2022	ECV_Td_2020.csv	tab-delimited despite the .csv extension
T2013/2022	ECV_Th_2022.tab	H is .tab while D/P/R are .csv — all four tab-delimited
L2013/2021-2024, /2022-2025	ECV_Lp_22a25.tab	no comma CSV at all; tab files are the only source

Every archive but the last two ships the same data twice — as comma-delimited EU-SILC files (esudb20d.csv, es23p.csv) and as INE's tab-delimited twins. Normalization takes the comma files and drops the twins. For the two tab-only releases the twin is the source, so it keeps a .tab extension: afterwards the delimiter is a property of the filename, never sniffed. The script verifies each file's delimiter against its target extension before accepting it.

python normalize_filenames.py is a dry run that prints the mapping; --execute writes.

Configuration

Machine-specific paths are centralised in config.py and overridable by environment variable:

env var	meaning	default
ECV_PROJECT	repo root	this file's directory
ECV_RAW_DIR	scratch dir for extracted .csv	<root>/raw
ECV_YEARS	restrict the year range (also set by run.py --years)	all

Correspondence with the sibling pipelines

mvazcar/MCVL	mvazcar/panel-de-hogares	this pipeline
raw_zipped/ → raw/<year>/	raw_zipped/ → raw/	raw_zipped/ → raw/<BASE>/<PERIOD>/
normalize_filenames.py	unpack.py	download.py + normalize_filenames.py
raw_path(year, filetype, part)	hogares_path(year, kind)	raw_path(key, letter)
config.py / readers.py	config.py / readers.py	config.py / readers.py (+ harmonise.py, indicators.py)
step01…step07	step01…step06	step01…step05
pipeline.py / run.py	pipeline.py / run.py	pipeline.py / run.py
—	parse_disenos.py / build_catalog.py	parse_disenos.py / build_catalog.py

Like MCVL, the data you supply is raw_zipped/; unlike MCVL, download.py can fetch it for you.

Requirements

Python 3.10+, and pip install -r requirements.txt (polars, pyarrow, openpyxl, py7zr). No external system dependency.

License

The code in this repository is in the public domain — Unlicense. This licence covers the code and nothing else; see Data below for the terms of the underlying microdata.

Data

ECV microdata is published openly by INE and is free to download and use under INE's terms of use. This repository ships only code and the parsed schema metadata — .gitignore excludes raw_zipped/, raw/, temp/, output/ and the design binaries. Run download.py to fetch the microdata yourself; it takes about a minute.

Source: INE — Encuesta de Condiciones de Vida.

Related pipelines

Same design, other Spanish microdata sources:

• mvazcar/MCVL — Muestra Continua de Vidas Laborales (Social Security)
• mvazcar/panel-de-hogares — AEAT/IEF Panel de Declarantes de IRPF
• mvazcar/EFF — Banco de España, Encuesta Financiera de las Familias
• mvazcar/ECV — this repository

Built with Claude Code.