Worker Timeout After Restart

[Phaeton]

Not a show stopper, but a consistent issue I am seeing. The first time you try and review a detection after a system restart, I get a timeout of the initial worker. This happens if you play audio or navigate to the Detection Details page. Once the new worker is booted, everything runs as expected.

[2023-06-07 15:58:36 -0400] [7] [CRITICAL] WORKER TIMEOUT (pid:18)
Exception ignored on calling ctypes callback function: <function ExecutionEngine._raw_object_cache_notify at 0x7f48250c3880>
Traceback (most recent call last):
 File "/usr/local/lib/python3.10/site-packages/llvmlite/binding/executionengine.py", line 171, in _raw_object_cache_notify
   def _raw_object_cache_notify(self, data):
 File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/base.py", line 203, in handle_abort
   sys.exit(1)
SystemExit: 1
[2023-06-07 15:58:37 -0400] [7] [WARNING] Worker with pid 18 was terminated due to signal 9
[2023-06-07 15:58:37 -0400] [524] [INFO] Booting worker with pid: 524

[mmcc-xx]

Well that's weird.

I haven't seen that error, but I have noticed that the first time a spectrogram is created takes considerably longer than subsequent ones. The generation is done with a library called librosa and some code that I copied and pasted as I know little about audio signal analysis.

The next time you restart, could you try calling the API directly to see if the error happens?

it would be http://apiserver:apiserverport/api/spectrogram/path:filename.png

where path:filename is the name of one of the mp3s in your detection directory, complete with the ".mp3"

The spectrograms are created on the fly because my assumption is that most of them would never be seen. I should probably at least save them off when they are generated so they don't have to be generated twice.

[Phaeton]

Yes, get the same error if calling the spectrogram via API for the first time. After it boots a new worker, hit refresh and it pulls back.

[mmcc-xx]

That’s interesting… I’ll have to see if I can preload the library or something.

[mmcc-xx]

(I'm writing this mostly for my own reference)

This looks like gunicorn is timing out, probably because the initial librosa call takes a long time:

https://stackoverflow.com/questions/10855197/frequent-worker-timeout

Fixing this may just be a matter of increasing the timeout value.