🏪 Milan Data Care — Data Quality Analysis of Milan Neighbourhood Shops

A reproducible data-quality workflow for the City of Milan's Esercizi di vicinato in sede fissa (neighbourhood shops) open dataset.

🌟 Highlights

• End-to-end data-quality pipeline: profiling, cleaning, repair and duplicate detection in one workflow.
• Parses the raw Ubicazione address field into structured components (street type, street name, civic number).
• Repairs inconsistent surface-area totals and standardises street and business-sector names.
• Imputes missing Isolato values with a trained Random Forest classifier instead of dropping records.
• Detects both exact and fuzzy duplicates using record-linkage and string-similarity techniques.
• Ships with the original dataset, the cleaned output, and a validation script — fully reproducible without re-downloading data.

ℹ️ Overview

This project takes the City of Milan's open dataset of neighbourhood shops — a real-world CSV export with missing values, inconsistent formatting and duplicate entries — and turns it into a clean, well-structured dataset ready for analysis. It is meant for anyone who needs to work with open government data and wants a concrete, worked example of how to assess and improve its quality rather than a theoretical checklist.

The workflow profiles the raw data along the completeness, validity, consistency and uniqueness dimensions, then applies targeted fixes: parsing addresses, standardising text fields, repairing numeric inconsistencies, imputing missing categorical values, and flagging duplicate records. The result is a cleaned CSV plus a lightweight test that checks its shape stays correct over time.

🧩 How It Works

1. Load the raw CSV (data/raw/...csv) into a pandas DataFrame.
2. Profile the data (completeness, validity, consistency, uniqueness) using descriptive statistics and ydata-profiling.
3. Parse and standardise the Ubicazione field and other text columns (street type, street name, business name, business sector).
4. Repair missing address/surface-area values and inconsistent surface totals.
5. Impute missing Isolato values with a Random Forest classifier.
6. Detect duplicates — both exact matches and fuzzy matches via recordlinkage, Levenshtein and Jaro-Winkler similarity.
7. Export the cleaned dataset to data/processed/cleaned_dataset.csv.

🚀 Usage

Run the full workflow as a script:

python src/data_quality_milan_shops.py

Or explore it interactively as a notebook:

jupyter notebook notebooks/data_quality_milan_shops.ipynb

Once the cleaned dataset is generated, validate its structure:

python tests/check_dataset.py

Expected output:

OK: data/processed/cleaned_dataset.csv has 22733 rows and 17 columns

Some steps use ydata-profiling and Nominatim geocoding, so a full run can take a while and may require an internet connection.

⬇️ Installation

Requirements

• Python 3.10 or later
• Internet connection (for the profiling and geocoding steps)

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

No further configuration is needed — the raw and cleaned datasets are already versioned in the repository.

📁 Project Structure

milan-data-care/
├── data/
│   ├── raw/            # Original Comune di Milano dataset
│   └── processed/      # Cleaned dataset produced by the workflow
├── notebooks/          # Main analysis notebook
├── src/                # Python export of the notebook
├── tests/              # Structural validation of the cleaned dataset
├── docs/                # Project report (PDF)
├── requirements.txt    # Python dependencies
└── README.md

🛠️ Technologies

• Language: Python 3.10+
• Data processing: pandas, NumPy
• Machine learning: scikit-learn (Random Forest imputation)
• Data profiling: ydata-profiling
• Duplicate detection: recordlinkage, python-Levenshtein, jaro-winkler
• Geocoding: geopy (Nominatim)
• Visualization: matplotlib, seaborn
• Environment: Jupyter / IPython

📊 Results

Dataset	Records	Columns
Original dataset	24,136	13
Cleaned dataset	22,733	17

✍️ Authors

• Alessandro Assini
• Angelo Baturi

This project was developed as part of a university course assignment on data quality.

🤝 Contributing

Contributions are welcome. If you'd like to help:

• open an issue to report a bug or suggest an improvement;
• submit a pull request with your proposed change;
• start a discussion if you have questions about the workflow or the data.

📄 License

This project is licensed under the MIT License. Before redistributing the data, verify and comply with the source dataset's terms of use.

📚 Documentation

• docs/data_quality_report.pdf — full project report.