A portfolio-grade, end-to-end data analytics project simulating a 6-week Data Analytics internship engagement for a nonprofit ("InAmigos Foundation" — a synthetic demo instance).
The platform analyzes volunteer participation and program delivery data to identify engagement and retention drivers, classify regional performance tiers, and produce data-driven resource-allocation recommendations for NGO leadership.
The project is built as a fully reproducible, six-stage pipeline with a fixed random seed. Every number in this README, in the generated reports, and in the dashboard can be regenerated byte-for-byte from a clean checkout.
The Streamlit dashboard (dashboard/app.py) is deployed live on Streamlit Community Cloud.
- Filter by region, state, program category, and date range.
- Explore state performance tiers on the interactive bubble map.
- Review hypothesis-test and exploratory-model results dynamically.
All data in this repository is 100% synthetically generated. No real NGO records, volunteer PII, or beneficiary data are used anywhere in this project. The synthetic dataset is engineered to mirror the structure and realistic messiness of genuine field-collected NGO data—including missing values, inconsistent formatting, duplicates, outliers, seasonality, and regional heterogeneity. This ensures the cleaning, analysis, and modeling work is genuinely meaningful, rather than a toy exercise on pre-cleaned data.
For full disclosure and generation methodology, see: docs/DATA_PROVENANCE.md.
# 1. Create and activate your virtual environment (recommended)
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# 2. Install dependencies
pip install -r requirements.txt
# 3. Run the entire analytics pipeline end-to-end
python src/data_pipeline.py
# (Alternatively, run: make pipeline)
# 4. Launch the dashboard locally
streamlit run dashboard/app.py
# 5. Run the automated test suite
pytest tests/ -v
# (Alternatively, run: make test)Individual stages can be run on their own — see
python src/data_pipeline.py --help or make help.
| Phase | Focus | Key Artifacts |
|---|---|---|
| 1 | Setup & Synthetic Data Generation | src/data_generation.py, data/raw/*.csv |
| 2 | Data Cleaning & Feature Engineering | src/cleaning.py, src/feature_engineering.py, data/processed/*.csv |
| 3 | EDA & Statistical Hypothesis Testing | src/stats_analysis.py, notebooks/01_EDA.ipynb, reports/STATS_REPORT.txt |
| 4 | Clustering & Exploratory Logistic Regression | src/modeling.py, reports/MODELING_REPORT.txt |
| 5 | Streamlit Dashboard & Static Visual Reports | dashboard/app.py, reports/figures/ |
| 6 | Automated Testing & Final Documentation | tests/, docs/ |
All six phases are complete. See docs/methodology.md
for the full reasoning behind every methodological choice (including two
places where an initial approach was deliberately corrected after review —
the t-test's target metric, and the K-Means k selection logic).
ngo-program-impact-analytics/
├── README.md <- this file
├── Makefile <- convenience shortcuts for every pipeline stage
├── pytest.ini <- pytest configuration
├── requirements.txt <- pip dependencies
├── config/
│ └── config.yaml <- single source of truth for every tunable parameter
├── src/
│ ├── data_generation.py <- Phase 1: synthetic data generation engine
│ ├── cleaning.py <- Phase 2: missing values, dedup, standardization
│ ├── feature_engineering.py <- Phase 2: derived analysis-ready feature tables
│ ├── data_pipeline.py <- orchestrates all 5 processing stages
│ ├── stats_analysis.py <- Phase 3: EDA, correlations, chi-square, t-test
│ └── modeling.py <- Phase 4: PCA/K-Means clustering + exploratory logistic regression
├── data/
│ ├── raw/ <- generated raw (intentionally messy) CSVs + generation log
│ └── processed/ <- cleaned tables, feature tables, cluster/risk outputs, audit logs
├── notebooks/
│ ├── 01_EDA.ipynb <- thin notebook wrapper around stats_analysis.py
│ └── _build_eda_notebook.py <- utility that (re)generates the notebook JSON
├── reports/
│ ├── STATS_REPORT.txt <- full plain-language statistical report (Phase 3)
│ ├── MODELING_REPORT.txt <- full plain-language clustering/modeling report (Phase 4)
│ └── figures/ <- 10 exported static PNG charts + index README
├── dashboard/
│ ├── app.py <- interactive Streamlit dashboard (Phase 5) — live at the link above
│ └── state_geo.py <- static lat/lon centroids for the 28-state bubble map
├── tests/
│ ├── conftest.py <- shared pytest configuration
│ ├── test_pipeline.py <- generation/cleaning/feature-engineering tests
│ └── test_stats.py <- stats_analysis.py & modeling.py tests
└── docs/
├── methodology.md <- full methodology write-up & design-decision rationale
├── data_dictionary.md <- every column, every table, fully documented
└── DATA_PROVENANCE.md <- the complete synthetic-data disclosure statement
src/data_generation.py produces four interlinked tables (6 programs, 28
Indian states, 240+ volunteers, ~3,500+ activity logs spanning Jan
2020–Dec 2025) with deliberately injected, config-controlled messiness:
missing values, near-duplicate records, inconsistent state-name/date
formatting, implausible outliers, seasonal activity patterns, and
per-state behavioral heterogeneity. Every injection is logged to
data/raw/GENERATION_LOG.txt. Identity fields (names, emails, phones) are
generated by a small, dependency-free SyntheticIdentityGenerator — no
third-party PII library required.
src/cleaning.py standardizes state names (55–137 raw string variants
resolved down to the canonical 28) and dates (5 different raw formats),
resolves duplicate volunteer registrations and activity logs, imputes
missing values using defensible group-wise strategies (never a blind
global mean, never a fabricated contact detail), and non-destructively
winsorizes outliers with a full audit trail (_raw columns +
_flag columns). src/feature_engineering.py then derives
volunteer-level, state-monthly, and state-summary feature tables — most
notably engagement_consistency, the behavioral metric that turns out to
matter more than raw activity volume throughout the rest of the project.
src/stats_analysis.py (narrated in notebooks/01_EDA.ipynb) covers
summary statistics, seasonality/time-series trends, correlation analysis,
a chi-square test of independence, and a two-sample t-test — every result
paired with a plain-language business interpretation, reported honestly
whether or not it's statistically significant.
⚠️ Every model in this phase is a lightweight, exploratory proof-of-concept on a small, synthetic dataset — never a validated predictive system, causal model, or production decision engine. See the disclosure at the top ofsrc/modeling.py.
src/modeling.py segments the 28 states into 3 empirically-justified
performance tiers via PCA + K-Means, and fits an exploratory logistic
regression (at the volunteer level, for a defensible sample size) to
surface candidate factors associated with non-retention, aggregating
predicted risk back up to the state level.
dashboard/app.py is a Streamlit app — live here
— that reads only from data/processed/ (it never regenerates or
re-fits anything itself). Sidebar filters (region, state, program
category, date range) drive a live KPI row and four tabs: Overview, State
Explorer (approximate bubble map), Hypothesis Tests (recomputed live on
the filtered subset), and Exploratory Model (with the same strict framing
carried into the UI).
tests/test_pipeline.py and tests/test_stats.py provide 39+ automated
pytest checks covering the full pipeline, including regression guards
against two specific bugs caught and fixed during development (see
docs/methodology.md §5–6). docs/ contains the full methodology
write-up, data dictionary, and data-provenance disclosure.
| Analysis | Result | Business Implication |
|---|---|---|
| Chi-square: retention × program category | Not significant (p = 0.368) | Retention interventions should be organization-wide, not program-specific |
| Two-sample t-test: engagement consistency, Tier1 vs Tier3 | Significant (p = 0.0017, Cohen's d ≈ 0.51) | Lagging states need coordination-capacity investment, not just recruitment |
| K-Means clustering (k=3, empirically justified) | 14 High-Performing / 4 Emerging / 10 Needs-Support states | Gives leadership a concrete shortlist for coordination-capacity investment |
| Exploratory logistic regression | Engagement consistency ↓ risk (OR≈0.45); tenure ↑ risk (OR≈2.06) | Consistency-building engagement tactics likely matter more than tenure alone |
| Cross-model disagreement | Telangana & Sikkim: "High-Performing" cluster, but low retention + high individual risk | Some states may look healthy in aggregate while quietly losing volunteers |
The live dashboard is deployed on Streamlit Community Cloud, pointed
at dashboard/app.py with requirements.txt as the dependency manifest.
A conda environment.yml was originally included as an alternative local
setup path but was removed — it isn't used by Streamlit Cloud's build
process and was causing deployment issues, so requirements.txt /
pip install is now the single supported install path for both local
development and deployment.
This is an educational/portfolio project built to demonstrate data analytics engineering skills. It is not affiliated with, and does not represent real data from, any actual NGO. "InAmigos Foundation" is used here as a fictional placeholder name for a synthetic demo instance.