-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nwis_client
"sqlite3.OperationalError: database is locked"
#215
Comments
Hmmm, thanks for reporting this @jarq6c! This is a little surprising given that we are not using multiprocessing. This smells like an upstream bug, but i'll need to do a little digging to verify that. For a little context This error is surprising to me because we are using an async approach to retrieving data instead of threads or multiprocessing. It is surprising because I would expect that writing to the cache is a blocking operation. I suspect what is happening is a lot of async writes to the cache (db) (that might be large in size and take more time) and no logic to enforce Ill do some digging later this morning and try to find a way around this and report back. Thanks again, @jarq6c! |
Starting to dig now. It looks like others have run into similar problems when using |
In omnilib/aiosqlite#234 (the issue I previously linked) a provided solution was to limit the number of concurrent
|
@aaraney Increasing Of note: I haven't been able to get this error to appear on all systems. I expect this is related to the performance of |
Great!
I think it's less about sqlite's performance per say, but instead io speed. So for example, at the hardware level an SSD vs a HDD and at the software level OS userspace / kernel space switching that's done during reading and writing.
I will have to look if I have the numbers somewhere, but it was a back-of-the-napkin calculation. I requested data at something like 10, 20, 40, 60, 80, 100, 500 sites and looked at request-response full travel times and 20 was on average the fastest. |
Barring submission of another issue it's probably safe to close this one. The default of 20 sites per request likely covers most users, so changing it might do more harm than good. Note the machine on which I got this error was a virtual machine, running on an ancient OS, stored on an HDD, and frequently subject to scans that eat up a significant portion of system resources. So, quite an extraordinary case remedied by a simple solution. Thanks for the help! |
When running the following code, I get an error. I've noticed this error seems to occur around the ~2500 site mark. That is I'm having trouble retrieving data from more than ~2500 sites at a time. The script raises the error and then freezes until I cancel it with ctrl-c.
Advice?
The text was updated successfully, but these errors were encountered: