Add global locks to databases

[bouthilx]

Some procedures might require atomic operations on many rows. For that we would need a lock on the database.

Regarding MongoDB, we can read in the doc that it supports many level of locks.

MongoDB uses multi-granularity locking [1] that allows operations to lock at the global, database or collection level, and allows for individual storage engines to implement their own concurrency control below the collection level

I don't know if global would be enough.

Here is a code snippet for mongodb from @dendisuhubdy

def lock(self):
    """Lock a database"""
    try:
        lock = self.conn.fsync(lock=True)
    except pymongo.errros.ConnectionFailure as e:
        self.logger.debug("Could not lock a database")
        raise_from(DatabaseError("Could not determine lock due to connection."), e)
    return lock

def unlock(self):
    """ Unlocking a database """
    try:
        unlock = self.conn.unlock()
    except pymongo.errors.ConnectionFailure as e:
        self.logger.debug("Could not determine unlock due to connection")
        raise_from(DatabaseError("Could not determine unlock due to connection."), e)
    return unlock

[tsirif]

I think fsync should better be in the database. We read somewhere that it is intended for reserving time for maintenance/backup, could it be a slow operation?

[dendisuhubdy]

@tsirif is referring to https://stackoverflow.com/a/11531818/3492631

[bouthilx]

Alright so that should not be how we lock the database. I haven't read the doc yet. We should read it when we try to address this issue.

[dendisuhubdy]

@bouthilx there's no other way to manually lock the database, although according to the MongoDB documentation, these operations cause locking https://docs.mongodb.com/manual/faq/concurrency/#which-operations-lock-the-database

[bouthilx]

I see. There is no way of locking over many updates, unless we use fsync or a retry loop like explained here. It seems it is just better to forget about atomic operations on many documents at a time.

[dendisuhubdy]

@bouthilx in that case, would it mean that the if block not be needed and also operations updateMany too? Or this interface would be just fine? Mind double checking again for code errors/logic so that both of you can approve it for merging by tomorrow? :)