Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

delete JsonNode as far as possible for speed up #271

Open
itsumura-h opened this issue Dec 20, 2023 · 1 comment
Open

delete JsonNode as far as possible for speed up #271

itsumura-h opened this issue Dec 20, 2023 · 1 comment
Assignees

Comments

@itsumura-h
Copy link
Owner

No description provided.

@itsumura-h itsumura-h self-assigned this Dec 20, 2023
@sgmihai
Copy link
Contributor

sgmihai commented Sep 29, 2024

I wonder if this is related to my performance problem.
Trying to insert 30.000 rows in an empty database, it takes about 5-6 minutes, doing one row insert at a time.
If I insert all the data at once (seq[JsonNode]), in sqlite, I get a "too many sql variables" error. Somehow the library doesn't batch them correctly.
This is how the query looks, truncated, from the output from a seq[JsonNode] insert, taken from the allographer log file, 1.1MB in size: https://pastebin.com/raw/tB6WBz27
Seems that you can only have max 999 such values in sqlite insert, so the lib might be buggy ? Also the docs are outdated, they mention an "inserts" method, which is not available anymore, got merged in the normal insert one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants