New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Progress bar for sqlite-utils insert #173
Comments
This can only work when it's reading from a file, not when it's reading from standard input. |
I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen. Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read. |
I'm using a sqlite-utils/sqlite_utils/cli.py Line 496 in 5a63b9e
I'll need to change that to be something that I can easily measure progress through. Also I should change its name - It looks like the argument provided by |
Relevant code: sqlite-utils/sqlite_utils/cli.py Lines 550 to 560 in 5a63b9e
Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard. |
I could use Here's how to iterate through a list of objects being read from a file: import json
parser = ijson.items(open(
"/tmp/list.json"
), "item")
for object in parser:
# ... |
I was thinking about this, while inserting a stream of ~40M line-delimited json docs. Wouldn't a That's how tqdm does it |
It would be nice if
sqlite-utils insert
had a progress bar, for when it's churning through huge CSV files.The text was updated successfully, but these errors were encountered: