Dataspool

Lightweight versioning for data files with automatic change detection.

Dataspool is a production-ready library for tracking changes in pandas DataFrames with minimal configuration. It automatically detects changes via content hashing and maintains a clean version history.

Features

• Smart change detection - only saves when content actually changes
• Automatic versioning with timestamps and comprehensive metadata
• SHA256 content hashing for reliable change detection
• Complete version history and audit trail
• Zero configuration with sensible defaults
• Comprehensive test coverage

Installation

pip install dataspool

Quick Start

import pandas as pd
from dataspool import save_with_version_control

df = pd.DataFrame({
    "Product": ["Apple", "Banana", "Cherry"],
    "Price": [1.50, 0.75, 2.25]
})

result = save_with_version_control(df, "prices")

if result["saved"]:
    print(f"New version saved: {result['version']}")
else:
    print("No changes detected")

Usage

Basic Versioning

from pathlib import Path
from dataspool import save_with_version_control

result = save_with_version_control(
    df,
    base_filename="my_data",
    data_dir=Path("./data"),
    save_latest=True
)

Version History

from dataspool import get_version_history, get_latest_version

history = get_version_history(Path("./data"))
for version in history:
    print(f"{version['timestamp']}: {version['row_count']} rows")

latest = get_latest_version(Path("./data"))
print(f"Latest: {latest['timestamp']}")

Change Detection

df1 = pd.DataFrame({"A": [1, 2, 3]})
result1 = save_with_version_control(df1, "test")  # saved=True

# Same content, different order - detected as identical
df2 = pd.DataFrame({"A": [3, 1, 2]})
result2 = save_with_version_control(df2, "test")  # saved=False

# Modified content - creates new version
df3 = pd.DataFrame({"A": [1, 2, 4]})
result3 = save_with_version_control(df3, "test")  # saved=True

Development

Setup with uv

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone and install
git clone https://github.com/jmiloser/dataspool.git
cd dataspool
uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"

# Install pre-commit hooks
pre-commit install

Running Tests

pytest
pytest --cov=dataspool --cov-report=html

Code Quality

ruff format .
ruff check .
mypy src/

Roadmap

• Support for Polars DataFrames
• Parquet format support
• Diff functionality between versions
• Version rollback and restore
• Cloud storage backends (S3, GCS, Azure)
• CLI tool for version inspection
• Automatic version pruning

License

MIT License - see LICENSE file for details.

Author

Jim Miloser - GitHub