Whiskerbook GPU out of memory

[goddesswarship]

Sometimes MiewID match results don't display in Whiskerbook and show an "attempting to fetch results" message instead. According to @holmbergius:

We have tracked this down to a GPU out-of-memory issue related to MiewID. Restarting the MiewID deep learning server solves the issue, restarts the jobs, and the links above are now completed with results.

We are working with our machine learning engineers to understand whether this is a memory leak or whether we simply need a more powerful GPU.

Opening this ticket to investigate whether this is a memory leak or GPU issue.

Community link
https://community.wildme.org/t/match-not-found-gpu-issue-with-miewid/2633/2

[goddesswarship]

The Whiskerbook server was allowing up to three simultaneous MiewID/detector jobs (1 fast lane, two slow lane). We have limited this to max 2 (1 slow, 1 fast).