AssertionError: Increments must match decrements, usage counter negative

[valeriocos]

Hi,

I'm trying to import the information contained in a Git repository to a database. In order to speed up the database population process I run GitPython in a parallel way. Basically I have defined two thread pools (one for branches and tags and the other one for commits). However, it can happen that the process raise this error:

File "C:\Python27\lib\site-packages\smmap\mman.py", line 130, in use_region self._region.increment_client_count() File "C:\Python27\lib\site-packages\smmap\util.py", line 234, in increment_client_count assert self._uc > -1, "Increments must match decrements, usage counter negative: %i" % self._uc AssertionError: Increments must match decrements, usage counter negative

Do you know what is this error message about? Could you give some pointers?

thanks in advance

[Byron]

Hi,

The first thing that comes to my mind is that GitPython is not thread-save at all. Therefore you must not share anything related to and including the Repo object, unless all access is behind a mutex. Thus, each thread needs at least its own Repo instance. Once that is done, you might actually be hindered by the GIL, which will only be released if a thread is waiting for IO. Doing Multithreading in Python is usually prevented by it, and MultiProcessing should be used instead.

The easiest way to (possibly) gain speed in a single thread is to initialize the Repo instance with a GitCmdObjectDB.

I don't think the issue you experience is in any way related to smmap.

Good Luck - please close the issue if you see my point.

[valeriocos]

thanks for your answer