-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize performance #20
Comments
While
|
I modified the sqlite test to provide a benchmark for appending. Code:
Switching off the debug output temporarily:
Since b.N has a value of 10, I decided to create a .sql file with 10 INSERTs. Benchmarking the execution of a .sql file with INSERTs:
The .sql file:
Looks like it's a problem on my side. I'll do more research on this issue. I'm still not sure if it's due to disk access issues. Will attempt placing the DB in RAM. EDIT: Placing the database in RAM had a major effect:
The results were extremely fast, as shown above. EDIT 2: Since the benchmarks above were rather unreliable, doing only 10 inserts, I decided to use the previous benchmark tests. I threw the whole sqlite test suite in RAM and built it, then benchmarked it.
Appears to be a hard drive issue. I'll run a couple of scans on my drive and update you on the issue. Thanks! EDIT 3: Did a drive benchmark. Looks fine to me. |
What a fine and detailed contribution! I added your benchmark function to the Then I ran it on a SSD disk:
And then on a normal disk:
Got the same awful performance... Then I tried with a
Same results, so I asked strace:
Huge output but I quickly noticed some odd file creations and deletions related to SQLite3's journal feature, so I tried to disable the journal feature with a line at the beginning of the
Noticed an improvement!
One second is still slow for just 20 rows but at least we know this problem seems to be on sqlite3 configuration. File creation and deletion could be slow, even if write operations are fast. This does not seem the be the case with SSD. If you want to turn off journalizing on your go program you could use the |
I noticed the constant reupdating of the journal file and thought it might be the cause of the problem. Thanks for helping to debug this problem :) BTW, more benchmarks coming soon from my fork. I'll try disabling the journal file and let you know of the results, probably this afternoon. EDIT 1: http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html Might be helpful. I've looked at these and the options which seem most promising to me are:
As for file journaling:
In my use case, it seems fine to disable journaling, so I'll go ahead. EDIT 2: Applying the BEGIN and END transaction optimizations helped a lot. Code is attached in a pull request at #22
A rather significant improvement in my opinion. |
I've done some tests and it seems that applying BEGIN and END transactions are very helpful to optimization. Note that this will only be useful for multi-INSERTs. |
So we got to the root of the problem! I'm closing this issue as is related to the database itself, I'll add the link in the docs. Thanks, |
I will write a pull request applying the optimization. |
That's actually a good idea, I think we could add Database methods named Begin and End. See #23, what do you think? |
gosexy/db tends to work slowly. I haven't benchmarked any results yet, but will do so as soon as possible. Right now, it appears to be one insert per second.
Using sqlite wrapper.
The text was updated successfully, but these errors were encountered: