MAPI SQLite DB gets corrupted when accessed by multiple processes at once

[Rambomst]

Basically I had 20 or so mnamer instances processing a bunch of different files and that seems to have corrupted the mapi sqlite db. I could no longer use the app until until I completely removed the db.

[jkwill87]

Just to clarify, you mean that you had 20 instances of mnamer running concurrently?

[Rambomst]

Yes.

edit

I should have given more information.
It doesn't always seem to happen. I has happened a total of like 5 times of me doing the same thing about 100. I am just assuming it was due them being all run at once. That could be completely untrue, I haven't investigated it. Its only happened when I have had multiple instances running at once.

Also sorry about the open and closing... misclick.

[Rambomst]

The error:

Query Results
database disk image is malformed
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mapi/endpoints.py", line 114, in _request_json
timeout=1
File "/usr/local/lib/python3.5/dist-packages/requests_cache/core.py", line 126, in request
**kwargs
File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.5/dist-packages/requests_cache/core.py", line 97, in send
response, timestamp = self.cache.get_response_and_time(cache_key)
File "/usr/local/lib/python3.5/dist-packages/requests_cache/backends/base.py", line 71, in get_response_and_time
key = self.keys_map[key]
File "/usr/local/lib/python3.5/dist-packages/requests_cache/backends/storage/dbdict.py", line 117, in getitem
self.table_name, (key,)).fetchone()
sqlite3.DatabaseError: database disk image is malformed
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/mapi/endpoints.py", line 114, in _request_json
timeout=1
File "/usr/local/lib/python3.5/dist-packages/requests_cache/core.py", line 126, in request
**kwargs
File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.5/dist-packages/requests_cache/core.py", line 97, in send
response, timestamp = self.cache.get_response_and_time(cache_key)
File "/usr/local/lib/python3.5/dist-packages/requests_cache/backends/base.py", line 71, in get_response_and_time
key = self.keys_map[key]
File "/usr/local/lib/python3.5/dist-packages/requests_cache/backends/storage/dbdict.py", line 117, in getitem
self.table_name, (key,)).fetchone()
sqlite3.DatabaseError: database disk image is malformed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/mnamer", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.5/dist-packages/mnamer/main.py", line 386, in main
process_files(targets, user_media, test_run, id_key, **config)
File "/usr/local/lib/python3.5/dist-packages/mnamer/main.py", line 226, in process_files
hit = next(results)
File "/usr/local/lib/python3.5/dist-packages/mnamer/utils.py", line 232, in provider_search
for result in provider_search.providers[media].search(id_key, **metadata):
File "/usr/local/lib/python3.5/dist-packages/mapi/providers.py", line 222, in search
for result in self._search_series(series, season, episode):
File "/usr/local/lib/python3.5/dist-packages/mapi/providers.py", line 274, in _search_series
self.token, series, cache=self.cache
File "/usr/local/lib/python3.5/dist-packages/mapi/endpoints.py", line 413, in tvdb_search_series
url, parameters, headers=headers, cache=cache
File "/usr/local/lib/python3.5/dist-packages/mapi/endpoints.py", line 127, in _request_json
SESSION.cache.clear()
File "/usr/local/lib/python3.5/dist-packages/requests_cache/backends/base.py", line 102, in clear
self.keys_map.clear()
File "/usr/local/lib/python3.5/dist-packages/requests_cache/backends/storage/dbdict.py", line 147, in clear
con.execute("drop table %s" % self.table_name)
sqlite3.DatabaseError: database disk image is malformed

[jkwill87]

This is outside what I would expect to be a normal use case for this application. I would generally not advise running multiple instances of mnamer concurrently as the file detection phase occurs prior to the rename one and this will likely lead to failure when one instance moves onto processing a file that has already been renamed or relocated. If you are not dissuaded, one way I could see his working would be to separate jobs using something like GNU Parallel and passing the nocache flag to mnamer in order to disable caching.