Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Progressbar not shown when inserting/upserting jsonlines file #485

Closed
MischaU8 opened this issue Sep 8, 2022 · 1 comment
Closed

Progressbar not shown when inserting/upserting jsonlines file #485

MischaU8 opened this issue Sep 8, 2022 · 1 comment

Comments

@MischaU8
Copy link
Contributor

MischaU8 commented Sep 8, 2022

When inserting or upserting a jsonlines file, no progressbar is shown. Expected behavior is that, just like with .csv/.tsv files, also for a jsonlines file (--nl), unless --silent is provided, a progressbar is shown.

sql-utils upsert mydb.db posts posts.jl --nl --pk post_id
(silence)

Currently file_progress is only called within the tsv/csv logic, however I think it can be safely wrapped around all the all the input formats that use decoded: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/cli.py#L963

MischaU8 added a commit to MischaU8/sqlite-utils that referenced this issue Sep 8, 2022
@simonw simonw closed this as completed in 0b315d3 Sep 15, 2022
@simonw
Copy link
Owner

simonw commented Sep 15, 2022

A note from PR #486: #486 (comment)_

sqlite-utils insert /tmp/t3.db t /tmp/big.json 
  [####################################]  100%

This is actually not doing the right thing. The problem is that sqlite-utils doesn't include a streaming JSON parser, so it instead reads that entire JSON file into memory first (exhausting the progress bar to 100% instantly) and then does the rest of the work in-memory while the bar sticks at 100%.

I decided to land this anyway. If a streaming JSON parser is added later it will start to work.

simonw added a commit that referenced this issue Oct 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants