Timeout waiting for gpu lock

[klutchell]

When the event server machine learning hooks are enabled with tinyyolov4 and gpu the process locks seem to be getting stuck resulting in this error.

12/28/20 18:28:35 zmesdetect_m1[1010] ERR yolo.py:51 [Timeout waiting for gpu lock for 300 seconds]
12/28/20 18:28:35 zmesdetect_m1[1010] FAT zm_detect.py:860 [Unrecoverable error:Timeout waiting for gpu lock for 300 seconds Traceback:Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pyzm/ml/yolo.py", line 46, in acquire_lock
    self.lock.acquire()
  File "/usr/local/lib/python3.6/dist-packages/portalocker/utils.py", line 343, in acquire
    raise exceptions.AlreadyLocked()
portalocker.exceptions.AlreadyLocked

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/zmeventnotification/bin/zm_detect.py", line 857, in <module>
    main_handler()
  File "/var/lib/zmeventnotification/bin/zm_detect.py", line 432, in main_handler
    b, l, c = m.detect(original_image)
  File "/usr/local/lib/python3.6/dist-packages/pyzm/ml/object.py", line 54, in detect
    b,l,c = self.model.detect(image)
  File "/usr/local/lib/python3.6/dist-packages/pyzm/ml/yolo.py", line 128, in detect
    self.acquire_lock()
  File "/usr/local/lib/python3.6/dist-packages/pyzm/ml/yolo.py", line 52, in acquire_lock
    raise ValueError ('Timeout waiting for {} lock for {} seconds'.format(self.processor, self.lock_timeout))
ValueError: Timeout waiting for gpu lock for 300 seconds
]

I'm watching these included projects for changes while I continue to investigate.

https://github.com/pliablepixels/zmeventnotification
https://github.com/pliablepixels/pyzm