tidycsv

Clean and validate messy real-world CSV files. One CLI, zero dependencies.

Real-world CSV files are messy: trailing whitespace, mixed encodings, CRLF/LF/CR line endings, accidental duplicates, blank rows dumped at the end, columns that look like integers until they aren't. tidycsv is a small, dependency-free Python CLI that fixes the common cases and gives you a JSON schema validator for the rest.

$ tidycsv inspect examples/messy_users.csv
{
  "file": "examples/messy_users.csv",
  "size_bytes": 523,
  "encoding": "utf-8",
  "delimiter": "comma",
  "rows": 10,
  "columns": 7,
  "per_column": { ... }
}

Features

• Trim whitespace from every cell
• Drop empty rows (configurable)
• Remove duplicate rows (optional, case-insensitive mode)
• Detect column-count mismatches and report them
• Auto-detect delimiter (, \t ; |) and encoding (utf-8-sig, utf-8, cp1252, latin-1)
• Convert to/from JSON and NDJSON, or swap delimiters
• Validate against a JSON schema with type checks (integer, float, boolean, email, URL, ISO date, UUID, string with regex / length / allowed values)
• Inspect a file: column summary, value frequencies, encoding and delimiter detection
• One Python file per concern, no third-party deps — works offline, in CI, on a server

Install

pip install .
# or, for development:
pip install -e .

Or just run it without installing:

python -m tidycsv <command> [args]

Quick start

Inspect a file

$ tidycsv inspect data.csv
{
  "file": "data.csv",
  "size_bytes": 12453,
  "encoding": "utf-8",
  "delimiter": "comma",
  "rows": 320,
  "columns": 8,
  "per_column": { "id": { "non_empty": 320, ... }, ... }
}

Clean a file

# Print the cleaned file to stdout, write a report to stderr
$ tidycsv clean data.csv > clean.csv

# Replace the file in place
$ tidycsv clean data.csv --in-place

# Keep the duplicates but still trim
$ tidycsv clean data.csv --keep-duplicates -o clean.csv

Example output (stderr):

# encoding: utf-8
# delimiter: ','
# rows in:    10
# rows out:   7
# duplicates removed:  2
# empty rows removed:   1
# whitespace fixes:     8
# column mismatches:    0

Convert formats

# CSV -> JSON
$ tidycsv convert data.csv --to json -o data.json

# JSON -> CSV
$ tidycsv convert data.json --to json | tidycsv convert - --to ndjson

# Comma -> tab (or any delimiter to any)
$ tidycsv convert data.csv --to-delim $'\t' -o data.tsv

# CSV -> NDJSON
$ tidycsv convert data.csv --to ndjson -o data.ndjson

Validate against a schema

$ tidycsv validate data.csv --schema schema.json
# validated 320 rows against 8 columns
# 0 rows had at least one error

Schema format (a JSON array of column specs):

[
  {"name": "id",    "type": "integer", "min_value": 1},
  {"name": "email", "type": "email",   "required": true},
  {"name": "age",   "type": "integer", "min_value": 0, "max_value": 150},
  {"name": "signup_date", "type": "iso_date"},
  {"name": "active", "type": "boolean"}
]

Supported types: string, integer, float, boolean, email, url, iso_date, uuid, any. Aliases: int, num/number/decimal, bool, date/datetime, text, str.

Detect delimiter only

$ tidycsv detect-delim data.csv
comma

Use as a library

from tidycsv import clean, CleanReport
from tidycsv.schema import Schema, ColumnSpec, ColumnType

with open("data.csv", encoding="utf-8") as f:
    text = f.read()

report: CleanReport = clean(text, trim=True, drop_duplicates=True, drop_empty=True)
print(report.summary())

schema = Schema(columns=[
    ColumnSpec("id",    type=ColumnType.INTEGER, min_value=1),
    ColumnSpec("email", type=ColumnType.EMAIL, required=True),
    ColumnSpec("age",   type=ColumnType.INTEGER, min_value=0, max_value=150),
])

for i, row in enumerate(report.rows, start=2):
    errs = schema.validate_row(report.headers, row)
    if errs:
        print(f"line {i}:", *errs, sep="\n  ")

Commands

tidycsv clean       [path] [flags]   trim, dedup, drop empty, fix mismatches
tidycsv convert     [path] [flags]   CSV <-> JSON / NDJSON, delimiter swap
tidycsv validate    [path] --schema  validate rows against a JSON schema
tidycsv inspect     [path]           encoding, delimiter, per-column stats
tidycsv detect-delim [path]          print likely delimiter
tidycsv --version                    show version

Run tidycsv <command> --help for the full flag list.

Why a new tool?

• csvkit is heavy and not focused on cleanup
• pandas.read_csv is overkill for a quick fix, and you usually need to write a 5-line script around it
• miller is powerful but has its own DSL — overkill when you just want a flag
• The browser-based "CSV cleaners" require uploading data

tidycsv is one file you can pip install on any box, run in a cron job, and trust to do the boring 80% of CSV cleanup without surprising you.

Running the tests

python -m unittest discover -s tests -v

License

MIT — see LICENSE.