Progress bar for sqlite-utils insert

[simonw]

It would be nice if sqlite-utils insert had a progress bar, for when it's churning through huge CSV files.

[simonw]

This can only work when it's reading from a file, not when it's reading from standard input.

[simonw]

I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.

Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

[simonw]

I'm using a click.File() at the moment:

sqlite-utils/sqlite_utils/cli.py

Line 496 in 5a63b9e

click.argument("json_file", type=click.File(), required=True),

I'll need to change that to be something that I can easily measure progress through. Also I should change its name - json_file is a bad name when it sometimes handles csv or tsv instead.

It looks like the argument provided by click.File doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path

[simonw]

Relevant code:

sqlite-utils/sqlite_utils/cli.py

Lines 550 to 560 in 5a63b9e

	if csv or tsv:
	dialect = "excel-tab" if tsv else "excel"
	reader = csv_std.reader(json_file, dialect=dialect)
	headers = next(reader)
	docs = (dict(zip(headers, row)) for row in reader)
	elif nl:
	docs = (json.loads(line) for line in json_file)
	else:
	docs = json.load(json_file)
	if isinstance(docs, dict):
	docs = [docs]

Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.

[simonw]

I could use ijson to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since sqlite-utils is a library dependency for many other projects and it would be using ijson purely for the CLI component.

Here's how to iterate through a list of objects being read from a file:

import json
parser = ijson.items(open(
    "/tmp/list.json"
), "item")
for object in parser:
   # ...

[Florents-Tselai]

I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.

Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

I was thinking about this, while inserting a stream of ~40M line-delimited json docs. Wouldn't a --total-expected flag work ?

That's how tqdm does it