Broken identity cache (shelve / bdb) when multiple parallel logins happen: _dbm.error: cannot add item to database #946
Description
We have a web service running with multiprocessing and therefore cannot have a in-memory identity cache but must share them between the instances.
The documentation of the used shelve module for the identity cache says:
The shelve module does not support concurrent read/write access to shelved objects. (Multiple simultaneous read accesses are safe.) When a program has a shelf open for writing, no other program should have it open for reading or writing. Unix file locking can be used to solve this, but this differs across Unix versions and requires knowledge about the database implementation used.
But pysaml2 doesn't seem to have some locking around write operations.
The exception we receive from customer systems then are:
response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries)
File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response
self.users.add_information_about_person(resp.session_info())
File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person
session_info["not_on_or_after"])
File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set
self._db[cni] = data
File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__
self.dict[key.encode(self.keyencoding)] = f.getvalue()
_dbm.error: cannot add item to database
The database is then in corrupt state:
# db_verify saml-18202.bdb.db
db_verify: BDB0540 Page 1: invalid next_pgno 8780
db_verify: BDB0540 Page 2: invalid next_pgno 6952
db_verify: saml-18202.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed
BDB5105 Verification of saml-18202.bdb.db failed.