-
Notifications
You must be signed in to change notification settings - Fork 0
Home
A guided tour through the engineering and financial model behind Personal Finance ETL.
This Wiki is the exploration layer for the project.
If the root README is the trailer and /docs is the technical specification, the Wiki is the tour through the system:
Source code
↓
implementation truth
/docs
↓
canonical technical knowledge
Wiki
↓
guided exploration
The Wiki does not replace /docs.
It connects the pieces. When a Wiki explanation and the frozen canonical documentation ever differ, /docs wins, and source code remains the implementation ground truth.
Repository README → project showcase
Canonical /docs → technical specification
Wiki → you are here, the guided exploration layer
Personal Finance ETL is a local-first financial data engineering and decision-support platform that turns fragmented financial evidence into one reconciled financial state.
flowchart LR
SRC["Financial Evidence<br/>Bank · Broker · SQLite · CSV · Excel · Market Data"]
CP["SQLite Control Plane<br/>Evidence · Runs · Failures · Provenance"]
BR["DuckDB Bronze<br/>Persistent Source State"]
CAN["Canonical Finance"]
INV["Investment Engine<br/>FIFO · Tax · Benchmark · XIRR"]
WEALTH["Wealth Engine<br/>Ledger · Cash Flow · FIRE"]
SG["Silver + Gold<br/>Analytical Contracts"]
APP["Power BI · CLI · Desktop"]
SRC --> CP
CP --> BR
BR --> CAN
CAN --> INV
CAN --> WEALTH
INV --> WEALTH
INV --> SG
WEALTH --> SG
SG --> APP
The defining architectural boundary is:
SQLite owns operational truth and raw evidence. DuckDB owns analytical state.
As of 24 September 2026, my production environment contains:
1,608 source artifacts
and the broker-history population grows by approximately:
1 stock snapshot/day
+
1 mutual-fund snapshot/day
=
~2 additional artifacts/day
Those files sit alongside transaction history, market/reference data, mappings, masters, opening-state inputs, and my personal-finance SQLite data.
That growing source population is why the application has mechanisms such as:
content hashing
change-aware synchronization
persistent raw evidence
persistent Bronze
file-aware replacement
run tracking
deterministic downstream rebuilds
The architecture grew around a real workload.
Start here:
Architecture Tour → Follow the Data → Inside the Control Plane → Warehouse & Power BI
You will see how the project handles source identity, hashing, Raw/Bronze synchronization, canonical transformation, analytical contracts, and serving.
Follow:
Financial Model → Investment Engine → Tax-Lot Accounting → Cash Flow & Wealth
The focus is financial meaning:
activity
→ reconstructed state
→ reconciliation
→ market value
→ tax state
→ decision support
Follow:
Investment Engine → Tax-Lot Accounting → FIRE & Monte Carlo
That route covers:
cash-flow-aware returns
shadow benchmark portfolios
tax-aware terminal state
drawdown
market regimes
fat tails
jump events
human-capital shocks
Monte Carlo
Follow:
Architecture Tour → Inside the Control Plane → Reliability & Recovery → Extending the Platform → Engineering Decisions
The interesting boundaries include:
repositories
facades
canonical contracts
asset pipelines
DataContract registry
multiprocessing
failure propagation
configuration vs strategies
Follow:
Financial Model → Warehouse & Power BI → Follow the Data
The core themes are:
semantic modelling
grain
Silver contracts
Gold marts
non-additive metrics
Power BI serving
flowchart TB
WHY["01 · Why This Exists"] --> ARCH["02 · Architecture Tour"]
ARCH --> DATA["03 · Follow the Data"]
DATA --> CP["04 · Inside the Control Plane"]
CP --> FIN["05 · Financial Model"]
FIN --> INV["06 · Investment Engine"]
INV --> TAX["07 · Tax-Lot Accounting"]
TAX --> WEALTH["08 · Cash Flow & Wealth"]
WEALTH --> FIRE["09 · FIRE & Monte Carlo"]
FIRE --> WH["10 · Warehouse & Power BI"]
WH --> REL["11 · Reliability & Recovery"]
REL --> EXT["12 · Extending the Platform"]
EXT --> DEC["13 · Engineering Decisions"]
DEC --> ROAD["14 · Project Journey & Roadmap"]
You can read it linearly or jump directly to the part of the system that interests you.
| Discipline | Where it appears |
|---|---|
| Data Engineering | Control Plane, hashing, Bronze synchronization, deterministic rebuild |
| Python Engineering | Pydantic, repositories, facades, stateful algorithms, multiprocessing |
| Software Architecture | ownership boundaries, contracts, strategies, failure paths |
| BI Engineering | semantic grain, Silver/Gold contracts, Power BI serving |
| Finance | ledger semantics, reconciliation, wealth, tax |
| Investment Analytics | FIFO, broker reconciliation, benchmark state, XIRR |
| Quantitative Engineering | regimes, fat tails, jumps, Numba Monte Carlo |
| Reliability | run lifecycle, rollback, recovery, provenance |
| Product Engineering | CLI, desktop, Power BI, packaged documentation |
The Wiki will not keep announcing those skills.
The tour shows where they live.
Derived state can be rebuilt.
Original financial evidence deserves a stronger durability boundary.
Portfolio XIRR
≠ average(ISIN XIRR)
A correct formula at the wrong analytical grain can still produce the wrong financial answer.
reconstructed quantity
vs
broker quantity
and:
calculated closing cash
vs
actual closing cash
give the model something capable of proving it wrong.
Financial assumptions belong in validated policy rather than hidden literals.
Polars
→ dataframe transformation
stateful Python
→ FIFO
multiprocessing
→ instrument isolation
NumPy / Numba
→ stochastic simulation
Next → Why This Exists
For the detailed technical specification:
Personal Finance ETL · Source · Canonical Docs · PyPI