OpenVino: facial detection is broken with 1.99.0 #8226
Comments
LeoAdL
changed the title
|
Can you check if setting the concurrency to 1 and restarting helps? And if that doesn't work, deleting the model cache and restarting? |
|
Setting the concurrency to 1, I almost get immediately a similar error: immich_redis | 1:M 24 Mar 2024 00:46:41.343 * Background saving terminated with success
immich_machine_learning | [03/24/24 00:46:43] INFO Setting 'buffalo_l' execution providers to
immich_machine_learning | ['OpenVINOExecutionProvider',
immich_machine_learning | 'CPUExecutionProvider'], in descending order of
immich_machine_learning | preference
immich_machine_learning | [03/24/24 00:46:43] INFO Loading facial recognition model 'buffalo_l' to
immich_machine_learning | memory
immich_machine_learning | [03/24/24 00:46:45] ERROR Exception in ASGI application
immich_machine_learning |
immich_machine_learning | ╭─────── Traceback (most recent call last) ───────╮
immich_machine_learning | │ /usr/src/app/main.py:118 in predict │
immich_machine_learning | │ │
immich_machine_learning | │ 115 │ │
immich_machine_learning | │ 116 │ model = await load(await model_cache. │
immich_machine_learning | │ ttl=settings.model_ttl, **kwargs)) │
immich_machine_learning | │ 117 │ model.configure(**kwargs) │
immich_machine_learning | │ ❱ 118 │ outputs = await run(model.predict, in │
immich_machine_learning | │ 119 │ return ORJSONResponse(outputs) │
immich_machine_learning | │ 120 │
immich_machine_learning | │ 121 │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/src/app/main.py:125 in run │
immich_machine_learning | │ │
immich_machine_learning | │ 122 async def run(func: Callable[..., Any], i │
immich_machine_learning | │ 123 │ if thread_pool is None: │
immich_machine_learning | │ 124 │ │ return func(inputs) │
immich_machine_learning | │ ❱ 125 │ return await asyncio.get_running_loop │
immich_machine_learning | │ 126 │
immich_machine_learning | │ 127 │
immich_machine_learning | │ 128 async def load(model: InferenceModel) -> │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/lib/python3.10/concurrent/futures/thread.p │
immich_machine_learning | │ y:58 in run │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/src/app/models/base.py:59 in predict │
immich_machine_learning | │ │
immich_machine_learning | │ 56 │ │ self.load() │
immich_machine_learning | │ 57 │ │ if model_kwargs: │
immich_machine_learning | │ 58 │ │ │ self.configure(**model_kwargs │
immich_machine_learning | │ ❱ 59 │ │ return self._predict(inputs) │
immich_machine_learning | │ 60 │ │
immich_machine_learning | │ 61 │ @abstractmethod │
immich_machine_learning | │ 62 │ def _predict(self, inputs: Any) -> An │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/src/app/models/facial_recognition.py:49 in │
immich_machine_learning | │ _predict │
immich_machine_learning | │ │
immich_machine_learning | │ 46 │ │ else: │
immich_machine_learning | │ 47 │ │ │ decoded_image = image │
immich_machine_learning | │ 48 │ │ assert is_ndarray(decoded_image, n │
immich_machine_learning | │ ❱ 49 │ │ bboxes, kpss = self.det_model.dete │
immich_machine_learning | │ 50 │ │ if bboxes.size == 0: │
immich_machine_learning | │ 51 │ │ │ return [] │
immich_machine_learning | │ 52 │ │ assert is_ndarray(kpss, np.float32 │
immich_machine_learning | │ │
immich_machine_learning | │ /opt/venv/lib/python3.10/site-packages/insightf │
immich_machine_learning | │ ace/model_zoo/retinaface.py:224 in detect │
immich_machine_learning | │ │
immich_machine_learning | │ 221 │ │ det_img = np.zeros( (input_size[1 │
immich_machine_learning | │ 222 │ │ det_img[:new_height, :new_width, │
immich_machine_learning | │ 223 │ │ │
immich_machine_learning | │ ❱ 224 │ │ scores_list, bboxes_list, kpss_li │
immich_machine_learning | │ 225 │ │ │
immich_machine_learning | │ 226 │ │ scores = np.vstack(scores_list) │
immich_machine_learning | │ 227 │ │ scores_ravel = scores.ravel() │
immich_machine_learning | │ │
immich_machine_learning | │ /opt/venv/lib/python3.10/site-packages/insightf │
immich_machine_learning | │ ace/model_zoo/retinaface.py:152 in forward │
immich_machine_learning | │ │
immich_machine_learning | │ 149 │ │ kpss_list = [] │
immich_machine_learning | │ 150 │ │ input_size = tuple(img.shape[0:2] │
immich_machine_learning | │ 151 │ │ blob = cv2.dnn.blobFromImage(img, │
immich_machine_learning | │ (self.input_mean, self.input_mean, self.i │
immich_machine_learning | │ ❱ 152 │ │ net_outs = self.session.run(self. │
immich_machine_learning | │ 153 │ │ │
immich_machine_learning | │ 154 │ │ input_height = blob.shape[2] │
immich_machine_learning | │ 155 │ │ input_width = blob.shape[3] │
immich_machine_learning | │ │
immich_machine_learning | │ /opt/venv/lib/python3.10/site-packages/onnxrunt │
immich_machine_learning | │ ime/capi/onnxruntime_inference_collection.py:22 │
immich_machine_learning | │ 0 in run │
immich_machine_learning | │ │
immich_machine_learning | │ 217 │ │ if not output_names: │
immich_machine_learning | │ 218 │ │ │ output_names = [output.name │
immich_machine_learning | │ 219 │ │ try: │
immich_machine_learning | │ ❱ 220 │ │ │ return self._sess.run(output │
immich_machine_learning | │ 221 │ │ except C.EPFail as err: │
immich_machine_learning | │ 222 │ │ │ if self._enable_fallback: │
immich_machine_learning | │ 223 │ │ │ │ print(f"EP Error: {err!s │
immich_machine_learning | ╰─────────────────────────────────────────────────╯
immich_machine_learning | RuntimeException: [ONNXRuntimeError] : 6 :
immich_machine_learning | RUNTIME_EXCEPTION : Encountered unknown exception
immich_machine_learning | in Run()
immich_microservices | [Nest] 7 - 03/24/2024, 12:46:45 AM ERROR [JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request for facial recognition failed with status 500: Internal Server Error
immich_microservices | [Nest] 7 - 03/24/2024, 12:46:45 AM ERROR [JobService] Error: Machine learning request for facial recognition failed with status 500: Internal Server Error
immich_microservices | at MachineLearningRepository.predict (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:23:19)
immich_microservices | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
immich_microservices | at async PersonService.handleDetectFaces (/usr/src/app/dist/domain/person/person.service.js:248:23)
immich_microservices | at async /usr/src/app/dist/domain/job/job.service.js:137:36
immich_microservices | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_microservices | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_microservices | [Nest] 7 - 03/24/2024, 12:46:45 AM ERROR [JobService] Object:
immich_microservices | {
immich_microservices | "id": "9e1d4bbf-84c2-40dd-9aec-c913e5a1a662"
immich_microservices | }
immich_microservices | Deleting the model cache did not help neither. |
|
Can you try again with these env variables set? ORT_OPENVINO_ENABLE_CI_LOG=1
ORT_OPENVINO_ENABLE_DEBUG=1
OPENVINO_LOG_LEVEL=5
LOG_LEVEL=debugThis will help get more info on what's causing the error. |
|
I have the same issue with the N100 too. I tried changing the model to see if that was a workaround but I'm still getting the issue. Complete logsHope this helps! |
|
Here are my logs, I think very similar: immich_machine_learning | [03/24/24 16:17:42] DEBUG Available ORT providers:
immich_machine_learning | {'OpenVINOExecutionProvider',
immich_machine_learning | 'CPUExecutionProvider'}
immich_machine_learning | [03/24/24 16:17:42] DEBUG Available OpenVINO devices: ['CPU', 'GPU']
immich_machine_learning | [03/24/24 16:17:42] INFO Setting 'buffalo_l' execution providers to
immich_machine_learning | ['OpenVINOExecutionProvider',
immich_machine_learning | 'CPUExecutionProvider'], in descending order of
immich_machine_learning | preference
immich_machine_learning | [03/24/24 16:17:42] DEBUG Setting execution provider options to
immich_machine_learning | [{'device_type': 'GPU_FP32', 'cache_dir':
immich_machine_learning | '/cache/facial-recognition/buffalo_l/openvino'},
immich_machine_learning | {'arena_extend_strategy': 'kSameAsRequested'}]
immich_machine_learning | [03/24/24 16:17:42] DEBUG Setting execution_mode to ORT_SEQUENTIAL
immich_machine_learning | [03/24/24 16:17:42] DEBUG Setting inter_op_num_threads to 0
immich_machine_learning | [03/24/24 16:17:42] DEBUG Setting intra_op_num_threads to 0
immich_machine_learning | [03/24/24 16:17:42] DEBUG Setting preferred runtime to onnx
immich_machine_learning | [03/24/24 16:17:42] INFO Loading facial recognition model 'buffalo_l' to
immich_machine_learning | memory
immich_machine_learning | In the OpenVINO EP
immich_machine_learning | Model is fully supported on OpenVINO
immich_machine_learning | CreateNgraphFunc
immich_machine_learning | In the OpenVINO EP
immich_machine_learning | Model is fully supported on OpenVINO
immich_machine_learning | CreateNgraphFunc
immich_machine_learning | [03/24/24 16:17:44] ERROR Exception in ASGI application
immich_machine_learning |
immich_machine_learning | ╭─────── Traceback (most recent call last) ───────╮
immich_machine_learning | │ /usr/src/app/main.py:118 in predict │
immich_machine_learning | │ │
immich_machine_learning | │ 115 │ │
immich_machine_learning | │ 116 │ model = await load(await model_cache. │
immich_machine_learning | │ ttl=settings.model_ttl, **kwargs)) │
immich_machine_learning | │ 117 │ model.configure(**kwargs) │
immich_machine_learning | │ ❱ 118 │ outputs = await run(model.predict, in │
immich_machine_learning | │ 119 │ return ORJSONResponse(outputs) │
immich_machine_learning | │ 120 │
immich_machine_learning | │ 121 │
immich_machine_learning | │ │
immich_machine_learning | │ ╭────────────────── locals ───────────────────╮ │
immich_machine_learning | │ │ image = UploadFile(filename='blob', │ │
immich_machine_learning | │ │ size=591599, │ │
immich_machine_learning | │ │ headers=Headers({'content-dis… │ │
immich_machine_learning | │ │ 'form-data; name="image"; │ │
immich_machine_learning | │ │ filename="blob"', │ │
immich_machine_learning | │ │ 'content-type': │ │
immich_machine_learning | │ │ 'application/octet-stream'})) │ │
immich_machine_learning | │ │ inputs = b'\xff\xd8\xff\xe2\x01\xf0ICC… │ │
immich_machine_learning | │ │ \x00\x00mntrRGB XYZ │ │
immich_machine_learning | │ │ \x07\xe2\x00\x03\x00\x14\x00\… │ │
immich_machine_learning | │ │ kwargs = { │ │
immich_machine_learning | │ │ │ 'minScore': 0.7, │ │
immich_machine_learning | │ │ │ 'maxDistance': 0.5, │ │
immich_machine_learning | │ │ │ 'minFaces': 3 │ │
immich_machine_learning | │ │ } │ │
immich_machine_learning | │ │ model = <app.models.facial_recognitio… │ │
immich_machine_learning | │ │ object at 0x7ab23c362fe0> │ │
immich_machine_learning | │ │ model_name = 'buffalo_l' │ │
immich_machine_learning | │ │ model_type = <ModelType.FACIAL_RECOGNITION: │ │
immich_machine_learning | │ │ 'facial-recognition'> │ │
immich_machine_learning | │ │ options = '{"minScore":0.7,"maxDistance… │ │
immich_machine_learning | │ │ text = None │ │
immich_machine_learning | │ ╰─────────────────────────────────────────────╯ │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/src/app/main.py:125 in run │
immich_machine_learning | │ │
immich_machine_learning | │ 122 async def run(func: Callable[..., Any], i │
immich_machine_learning | │ 123 │ if thread_pool is None: │
immich_machine_learning | │ 124 │ │ return func(inputs) │
immich_machine_learning | │ ❱ 125 │ return await asyncio.get_running_loop │
immich_machine_learning | │ 126 │
immich_machine_learning | │ 127 │
immich_machine_learning | │ 128 async def load(model: InferenceModel) -> │
immich_machine_learning | │ │
immich_machine_learning | │ ╭────────────────── locals ───────────────────╮ │
immich_machine_learning | │ │ func = <bound method │ │
immich_machine_learning | │ │ InferenceModel.predict of │ │
immich_machine_learning | │ │ <app.models.facial_recognition.Fa… │ │
immich_machine_learning | │ │ object at 0x7ab23c362fe0>> │ │
immich_machine_learning | │ │ inputs = b'\xff\xd8\xff\xe2\x01\xf0ICC_PRO… │ │
immich_machine_learning | │ │ \x00\x00mntrRGB XYZ │ │
immich_machine_learning | │ │ \x07\xe2\x00\x03\x00\x14\x00\t\x0… │ │
immich_machine_learning | │ ╰─────────────────────────────────────────────╯ │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/lib/python3.10/concurrent/futures/thread.p │
immich_machine_learning | │ y:58 in run │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/src/app/models/base.py:59 in predict │
immich_machine_learning | │ │
immich_machine_learning | │ 56 │ │ self.load() │
immich_machine_learning | │ 57 │ │ if model_kwargs: │
immich_machine_learning | │ 58 │ │ │ self.configure(**model_kwargs │
immich_machine_learning | │ ❱ 59 │ │ return self._predict(inputs) │
immich_machine_learning | │ 60 │ │
immich_machine_learning | │ 61 │ @abstractmethod │
immich_machine_learning | │ 62 │ def _predict(self, inputs: Any) -> An │
immich_machine_learning | │ │
immich_machine_learning | │ ╭────────────────── locals ───────────────────╮ │
immich_machine_learning | │ │ inputs = b'\xff\xd8\xff\xe2\x01\xf0I… │ │
immich_machine_learning | │ │ \x00\x00mntrRGB XYZ │ │
immich_machine_learning | │ │ \x07\xe2\x00\x03\x00\x14\x0… │ │
immich_machine_learning | │ │ model_kwargs = {} │ │
immich_machine_learning | │ │ self = <app.models.facial_recognit… │ │
immich_machine_learning | │ │ object at 0x7ab23c362fe0> │ │
immich_machine_learning | │ ╰─────────────────────────────────────────────╯ │
immich_machine_learning | │ │
immich_machine_learning | │ /usr/src/app/models/facial_recognition.py:49 in │
immich_machine_learning | │ _predict │
immich_machine_learning | │ │
immich_machine_learning | │ 46 │ │ else: │
immich_machine_learning | │ 47 │ │ │ decoded_image = image │
immich_machine_learning | │ 48 │ │ assert is_ndarray(decoded_image, n │
immich_machine_learning | │ ❱ 49 │ │ bboxes, kpss = self.det_model.dete │
immich_machine_learning | │ 50 │ │ if bboxes.size == 0: │
immich_machine_learning | │ 51 │ │ │ return [] │
immich_machine_learning | │ 52 │ │ assert is_ndarray(kpss, np.float32 │
immich_machine_learning | │ │
immich_machine_learning | │ ╭────────────────── locals ───────────────────╮ │
immich_machine_learning | │ │ decoded_image = array([[[ 0, 17, 34], │ │
immich_machine_learning | │ │ │ │ [ 0, 17, 34], │ │
immich_machine_learning | │ │ │ │ [ 0, 18, 35], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 8, 7, 9], │ │
immich_machine_learning | │ │ │ │ [ 14, 13, 15], │ │
immich_machine_learning | │ │ │ │ [ 19, 18, 20]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 0, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 2, 20, 37], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 7, 6, 8], │ │
immich_machine_learning | │ │ │ │ [ 14, 13, 15], │ │
immich_machine_learning | │ │ │ │ [ 20, 19, 21]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 2, 22, 39], │ │
immich_machine_learning | │ │ │ │ [ 4, 22, 39], │ │
immich_machine_learning | │ │ │ │ [ 4, 22, 39], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 5, 4, 6], │ │
immich_machine_learning | │ │ │ │ [ 14, 13, 15], │ │
immich_machine_learning | │ │ │ │ [ 22, 21, 23]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 32, 56, 86], │ │
immich_machine_learning | │ │ │ │ [ 36, 60, 90], │ │
immich_machine_learning | │ │ │ │ [ 42, 66, 96], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 28, 54, 71], │ │
immich_machine_learning | │ │ │ │ [ 31, 57, 74], │ │
immich_machine_learning | │ │ │ │ [ 36, 62, 79]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 37, 59, 87], │ │
immich_machine_learning | │ │ │ │ [ 43, 65, 93], │ │
immich_machine_learning | │ │ │ │ [ 53, 75, 103], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 27, 53, 70], │ │
immich_machine_learning | │ │ │ │ [ 28, 54, 71], │ │
immich_machine_learning | │ │ │ │ [ 32, 58, 75]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 44, 66, 94], │ │
immich_machine_learning | │ │ │ │ [ 50, 72, 100], │ │
immich_machine_learning | │ │ │ │ [ 61, 83, 111], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 27, 53, 70], │ │
immich_machine_learning | │ │ │ │ [ 25, 51, 68], │ │
immich_machine_learning | │ │ │ │ [ 27, 53, 70]]], │ │
immich_machine_learning | │ │ dtype=uint8) │ │
immich_machine_learning | │ │ image = b'\xff\xd8\xff\xe2\x01\xf0… │ │
immich_machine_learning | │ │ \x00\x00mntrRGB XYZ │ │
immich_machine_learning | │ │ \x07\xe2\x00\x03\x00\x14\x… │ │
immich_machine_learning | │ │ self = <app.models.facial_recogni… │ │
immich_machine_learning | │ │ object at 0x7ab23c362fe0> │ │
immich_machine_learning | │ ╰─────────────────────────────────────────────╯ │
immich_machine_learning | │ │
immich_machine_learning | │ /opt/venv/lib/python3.10/site-packages/insightf │
immich_machine_learning | │ ace/model_zoo/retinaface.py:224 in detect │
immich_machine_learning | │ │
immich_machine_learning | │ 221 │ │ det_img = np.zeros( (input_size[1 │
immich_machine_learning | │ 222 │ │ det_img[:new_height, :new_width, │
immich_machine_learning | │ 223 │ │ │
immich_machine_learning | │ ❱ 224 │ │ scores_list, bboxes_list, kpss_li │
immich_machine_learning | │ 225 │ │ │
immich_machine_learning | │ 226 │ │ scores = np.vstack(scores_list) │
immich_machine_learning | │ 227 │ │ scores_ravel = scores.ravel() │
immich_machine_learning | │ │
immich_machine_learning | │ ╭────────────────── locals ───────────────────╮ │
immich_machine_learning | │ │ det_img = array([[[ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 3, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 0, 13, 31], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 6, 24, 41], │ │
immich_machine_learning | │ │ │ │ [ 6, 22, 39], │ │
immich_machine_learning | │ │ │ │ [ 1, 15, 33], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 3, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 0, 14, 32], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 40, 67, 101], │ │
immich_machine_learning | │ │ │ │ [ 28, 55, 89], │ │
immich_machine_learning | │ │ │ │ [ 33, 60, 94], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 35, 60, 92], │ │
immich_machine_learning | │ │ │ │ [ 33, 58, 90], │ │
immich_machine_learning | │ │ │ │ [ 43, 68, 100], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 43, 65, 93], │ │
immich_machine_learning | │ │ │ │ [ 69, 91, 119], │ │
immich_machine_learning | │ │ │ │ [ 64, 86, 114], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]]], │ │
immich_machine_learning | │ │ dtype=uint8) │ │
immich_machine_learning | │ │ det_scale = 0.3333333333333333 │ │
immich_machine_learning | │ │ im_ratio = 1.3333333333333333 │ │
immich_machine_learning | │ │ img = array([[[ 0, 17, 34], │ │
immich_machine_learning | │ │ │ │ [ 0, 17, 34], │ │
immich_machine_learning | │ │ │ │ [ 0, 18, 35], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 8, 7, 9], │ │
immich_machine_learning | │ │ │ │ [ 14, 13, 15], │ │
immich_machine_learning | │ │ │ │ [ 19, 18, 20]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 0, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 2, 20, 37], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 7, 6, 8], │ │
immich_machine_learning | │ │ │ │ [ 14, 13, 15], │ │
immich_machine_learning | │ │ │ │ [ 20, 19, 21]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 2, 22, 39], │ │
immich_machine_learning | │ │ │ │ [ 4, 22, 39], │ │
immich_machine_learning | │ │ │ │ [ 4, 22, 39], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 5, 4, 6], │ │
immich_machine_learning | │ │ │ │ [ 14, 13, 15], │ │
immich_machine_learning | │ │ │ │ [ 22, 21, 23]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 32, 56, 86], │ │
immich_machine_learning | │ │ │ │ [ 36, 60, 90], │ │
immich_machine_learning | │ │ │ │ [ 42, 66, 96], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 28, 54, 71], │ │
immich_machine_learning | │ │ │ │ [ 31, 57, 74], │ │
immich_machine_learning | │ │ │ │ [ 36, 62, 79]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 37, 59, 87], │ │
immich_machine_learning | │ │ │ │ [ 43, 65, 93], │ │
immich_microservices | [Nest] 7 - 03/24/2024, 4:17:44 PM ERROR [JobService] Unable to run job handler (faceDetection/face-detection): Error: Machine learning request for facial recognition failed with status 500: Internal Server Error
immich_microservices | [Nest] 7 - 03/24/2024, 4:17:44 PM ERROR [JobService] Error: Machine learning request for facial recognition failed with status 500: Internal Server Error
immich_microservices | at MachineLearningRepository.predict (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:23:19)
immich_microservices | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
immich_microservices | at async PersonService.handleDetectFaces (/usr/src/app/dist/domain/person/person.service.js:248:23)
immich_microservices | at async /usr/src/app/dist/domain/job/job.service.js:137:36
immich_microservices | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_microservices | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_microservices | [Nest] 7 - 03/24/2024, 4:17:44 PM ERROR [JobService] Object:
immich_microservices | {
immich_microservices | "id": "ea7fe00a-068b-46d3-8ae1-61dd6146286f"
immich_microservices | }
immich_microservices |
immich_machine_learning | │ │ │ │ [ 53, 75, 103], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 27, 53, 70], │ │
immich_machine_learning | │ │ │ │ [ 28, 54, 71], │ │
immich_machine_learning | │ │ │ │ [ 32, 58, 75]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 44, 66, 94], │ │
immich_machine_learning | │ │ │ │ [ 50, 72, 100], │ │
immich_machine_learning | │ │ │ │ [ 61, 83, 111], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 27, 53, 70], │ │
immich_machine_learning | │ │ │ │ [ 25, 51, 68], │ │
immich_machine_learning | │ │ │ │ [ 27, 53, 70]]], │ │
immich_machine_learning | │ │ dtype=uint8) │ │
immich_machine_learning | │ │ input_size = (640, 640) │ │
immich_machine_learning | │ │ max_num = 0 │ │
immich_machine_learning | │ │ metric = 'default' │ │
immich_machine_learning | │ │ model_ratio = 1.0 │ │
immich_machine_learning | │ │ new_height = 640 │ │
immich_machine_learning | │ │ new_width = 480 │ │
immich_machine_learning | │ │ resized_img = array([[[ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 3, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 0, 13, 31], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 2, 1, 3], │ │
immich_machine_learning | │ │ │ │ [ 2, 1, 3], │ │
immich_machine_learning | │ │ │ │ [ 14, 13, 15]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 6, 24, 41], │ │
immich_machine_learning | │ │ │ │ [ 6, 22, 39], │ │
immich_machine_learning | │ │ │ │ [ 1, 15, 33], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 1], │ │
immich_machine_learning | │ │ │ │ [ 4, 3, 5], │ │
immich_machine_learning | │ │ │ │ [ 16, 15, 17]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 3, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 0, 14, 32], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 2, 1, 3], │ │
immich_machine_learning | │ │ │ │ [ 2, 1, 3], │ │
immich_machine_learning | │ │ │ │ [ 18, 17, 19]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 40, 67, 101], │ │
immich_machine_learning | │ │ │ │ [ 28, 55, 89], │ │
immich_machine_learning | │ │ │ │ [ 33, 60, 94], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 49, 75, 92], │ │
immich_machine_learning | │ │ │ │ [ 37, 63, 80], │ │
immich_machine_learning | │ │ │ │ [ 32, 58, 75]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 35, 60, 92], │ │
immich_machine_learning | │ │ │ │ [ 33, 58, 90], │ │
immich_machine_learning | │ │ │ │ [ 43, 68, 100], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 29, 55, 72], │ │
immich_machine_learning | │ │ │ │ [ 33, 59, 76], │ │
immich_machine_learning | │ │ │ │ [ 33, 59, 76]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 43, 65, 93], │ │
immich_machine_learning | │ │ │ │ [ 69, 91, 119], │ │
immich_machine_learning | │ │ │ │ [ 64, 86, 114], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 27, 53, 70], │ │
immich_machine_learning | │ │ │ │ [ 30, 56, 73], │ │
immich_machine_learning | │ │ │ │ [ 28, 54, 71]]], │ │
immich_machine_learning | │ │ dtype=uint8) │ │
immich_machine_learning | │ │ self = <insightface.model_zoo.retin… │ │
immich_machine_learning | │ │ object at 0x7ab23c362dd0> │ │
immich_machine_learning | │ ╰─────────────────────────────────────────────╯ │
immich_machine_learning | │ │
immich_machine_learning | │ /opt/venv/lib/python3.10/site-packages/insightf │
immich_machine_learning | │ ace/model_zoo/retinaface.py:152 in forward │
immich_machine_learning | │ │
immich_machine_learning | │ 149 │ │ kpss_list = [] │
immich_machine_learning | │ 150 │ │ input_size = tuple(img.shape[0:2] │
immich_machine_learning | │ 151 │ │ blob = cv2.dnn.blobFromImage(img, │
immich_machine_learning | │ (self.input_mean, self.input_mean, self.i │
immich_machine_learning | │ ❱ 152 │ │ net_outs = self.session.run(self. │
immich_machine_learning | │ 153 │ │ │
immich_machine_learning | │ 154 │ │ input_height = blob.shape[2] │
immich_machine_learning | │ 155 │ │ input_width = blob.shape[3] │
immich_machine_learning | │ │
immich_machine_learning | │ ╭────────────────── locals ───────────────────╮ │
immich_machine_learning | │ │ bboxes_list = [] │ │
immich_machine_learning | │ │ blob = array([[[[-0.71484375, │ │
immich_machine_learning | │ │ -0.71484375, -0.75390625, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.67578125, │ │
immich_machine_learning | │ │ -0.69140625, -0.73828125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.71484375, │ │
immich_machine_learning | │ │ -0.71484375, -0.74609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [-0.20703125, │ │
immich_machine_learning | │ │ -0.30078125, -0.26171875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.27734375, │ │
immich_machine_learning | │ │ -0.29296875, -0.21484375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.26953125, │ │
immich_machine_learning | │ │ -0.06640625, -0.10546875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375]], │ │
immich_machine_learning | │ │ │ │ │ │
immich_machine_learning | │ │ │ │ [[-0.84765625, │ │
immich_machine_learning | │ │ -0.84765625, -0.89453125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.80859375, │ │
immich_machine_learning | │ │ -0.82421875, -0.87890625, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.84765625, │ │
immich_machine_learning | │ │ -0.84765625, -0.88671875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [-0.47265625, │ │
immich_machine_learning | │ │ -0.56640625, -0.52734375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.52734375, │ │
immich_machine_learning | │ │ -0.54296875, -0.46484375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.48828125, │ │
immich_machine_learning | │ │ -0.28515625, -0.32421875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375]], │ │
immich_machine_learning | │ │ │ │ │ │
immich_machine_learning | │ │ │ │ [[-0.98828125, │ │
immich_machine_learning | │ │ -0.97265625, -0.99609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.94921875, │ │
immich_machine_learning | │ │ -0.94921875, -0.98828125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.98828125, │ │
immich_machine_learning | │ │ -0.97265625, -0.99609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [-0.68359375, │ │
immich_machine_learning | │ │ -0.77734375, -0.73828125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.72265625, │ │
immich_machine_learning | │ │ -0.73828125, -0.66015625, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.66015625, │ │
immich_machine_learning | │ │ -0.45703125, -0.49609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375]]]], │ │
immich_machine_learning | │ │ dtype=float32) │ │
immich_machine_learning | │ │ img = array([[[ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 3, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 0, 13, 31], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 6, 24, 41], │ │
immich_machine_learning | │ │ │ │ [ 6, 22, 39], │ │
immich_machine_learning | │ │ │ │ [ 1, 15, 33], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 1, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 3, 19, 36], │ │
immich_machine_learning | │ │ │ │ [ 0, 14, 32], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 40, 67, 101], │ │
immich_machine_learning | │ │ │ │ [ 28, 55, 89], │ │
immich_machine_learning | │ │ │ │ [ 33, 60, 94], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 35, 60, 92], │ │
immich_machine_learning | │ │ │ │ [ 33, 58, 90], │ │
immich_machine_learning | │ │ │ │ [ 43, 68, 100], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]], │ │
immich_machine_learning | │ │ │ │ │
immich_machine_learning | │ │ │ [[ 43, 65, 93], │ │
immich_machine_learning | │ │ │ │ [ 69, 91, 119], │ │
immich_machine_learning | │ │ │ │ [ 64, 86, 114], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0], │ │
immich_machine_learning | │ │ │ │ [ 0, 0, 0]]], │ │
immich_machine_learning | │ │ dtype=uint8) │ │
immich_machine_learning | │ │ input_size = (640, 640) │ │
immich_machine_learning | │ │ kpss_list = [] │ │
immich_machine_learning | │ │ scores_list = [] │ │
immich_machine_learning | │ │ self = <insightface.model_zoo.retin… │ │
immich_machine_learning | │ │ object at 0x7ab23c362dd0> │ │
immich_machine_learning | │ │ threshold = 0.7 │ │
immich_machine_learning | │ ╰─────────────────────────────────────────────╯ │
immich_machine_learning | │ │
immich_machine_learning | │ /opt/venv/lib/python3.10/site-packages/onnxrunt │
immich_machine_learning | │ ime/capi/onnxruntime_inference_collection.py:22 │
immich_machine_learning | │ 0 in run │
immich_machine_learning | │ │
immich_machine_learning | │ 217 │ │ if not output_names: │
immich_machine_learning | │ 218 │ │ │ output_names = [output.name │
immich_machine_learning | │ 219 │ │ try: │
immich_machine_learning | │ ❱ 220 │ │ │ return self._sess.run(output │
immich_machine_learning | │ 221 │ │ except C.EPFail as err: │
immich_machine_learning | │ 222 │ │ │ if self._enable_fallback: │
immich_machine_learning | │ 223 │ │ │ │ print(f"EP Error: {err!s │
immich_machine_learning | │ │
immich_machine_learning | │ ╭────────────────── locals ───────────────────╮ │
immich_machine_learning | │ │ input_feed = { │ │
immich_machine_learning | │ │ │ 'input.1': │ │
immich_machine_learning | │ │ array([[[[-0.71484375, │ │
immich_machine_learning | │ │ -0.71484375, -0.75390625, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.67578125, │ │
immich_machine_learning | │ │ -0.69140625, -0.73828125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.71484375, │ │
immich_machine_learning | │ │ -0.71484375, -0.74609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [-0.20703125, │ │
immich_machine_learning | │ │ -0.30078125, -0.26171875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.27734375, │ │
immich_machine_learning | │ │ -0.29296875, -0.21484375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.26953125, │ │
immich_machine_learning | │ │ -0.06640625, -0.10546875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375]], │ │
immich_machine_learning | │ │ │ │ │ │
immich_machine_learning | │ │ │ │ [[-0.84765625, │ │
immich_machine_learning | │ │ -0.84765625, -0.89453125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.80859375, │ │
immich_machine_learning | │ │ -0.82421875, -0.87890625, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.84765625, │ │
immich_machine_learning | │ │ -0.84765625, -0.88671875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [-0.47265625, │ │
immich_machine_learning | │ │ -0.56640625, -0.52734375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.52734375, │ │
immich_machine_learning | │ │ -0.54296875, -0.46484375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.48828125, │ │
immich_machine_learning | │ │ -0.28515625, -0.32421875, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375]], │ │
immich_machine_learning | │ │ │ │ │ │
immich_machine_learning | │ │ │ │ [[-0.98828125, │ │
immich_machine_learning | │ │ -0.97265625, -0.99609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.94921875, │ │
immich_machine_learning | │ │ -0.94921875, -0.98828125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.98828125, │ │
immich_machine_learning | │ │ -0.97265625, -0.99609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ ..., │ │
immich_machine_learning | │ │ │ │ [-0.68359375, │ │
immich_machine_learning | │ │ -0.77734375, -0.73828125, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.72265625, │ │
immich_machine_learning | │ │ -0.73828125, -0.66015625, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375], │ │
immich_machine_learning | │ │ │ │ [-0.66015625, │ │
immich_machine_learning | │ │ -0.45703125, -0.49609375, │ │
immich_machine_learning | │ │ ..., -0.99609375, │ │
immich_machine_learning | │ │ │ │ -0.99609375, │ │
immich_machine_learning | │ │ -0.99609375]]]], │ │
immich_machine_learning | │ │ dtype=float32) │ │
immich_machine_learning | │ │ } │ │
immich_machine_learning | │ │ output_names = [ │ │
immich_machine_learning | │ │ │ '448', │ │
immich_machine_learning | │ │ │ '471', │ │
immich_machine_learning | │ │ │ '494', │ │
immich_machine_learning | │ │ │ '451', │ │
immich_machine_learning | │ │ │ '474', │ │
immich_machine_learning | │ │ │ '497', │ │
immich_machine_learning | │ │ │ '454', │ │
immich_machine_learning | │ │ │ '477', │ │
immich_machine_learning | │ │ │ '500' │ │
immich_machine_learning | │ │ ] │ │
immich_machine_learning | │ │ run_options = None │ │
immich_machine_learning | │ │ self = <onnxruntime.capi.onnxrunti… │ │
immich_machine_learning | │ │ object at 0x7ab23c362b00> │ │
immich_machine_learning | │ ╰─────────────────────────────────────────────╯ │
immich_machine_learning | ╰─────────────────────────────────────────────────╯
immich_machine_learning | RuntimeException: [ONNXRuntimeError] : 6 :
immich_machine_learning | RUNTIME_EXCEPTION : Encountered unknown exception
immich_machine_learning | in Run()
immich_machine_learning | [03/24/24 16:17:48] DEBUG Checking for inactivity... |
|
I made an upstream issue for this based on the info here. Let's wait for their response. |
|
I am having this problem too. |
|
I'm having the same issue in 1.100.0 too. |
|
For future reference, it's better to leave a thumbs up for issues if they affect you. Leaving a comment notifies every participant in the issue, so it should be reserved for comments that drive the issue forward. |
|
same issue in 1.100.0 too. |
|
I am also facing this issue. It runs fine on my laptop which has Can we provide an option to disable |
|
The i5-1145G7 has Iris graphics, same with the CPU I tested with. It seems to be that UHD graphics doesn't work, but Iris does. |
This comment has been minimized.
This comment has been minimized.
|
similar here with smart search. 1.102.3 on N5095. good news, when change from immich-machine-learning:${IMMICH_VERSION}-openvino to immich-machine-learning:${IMMICH_VERSION}, it works! (related #8918) |
|
same error on 1.102.3, was working until 1.099
|
|
me too |
|
Same problem in new version 1.103.1 `[05/02/24 07:05:03] INFO Setting 'antelopev2' execution providers to |
|
Thanks for the hint and "workaround". |
|
still not working in 1.104.0 |
|
I was going to try downgrading Edit: nope, not that simple if you try to edit the Dockerfile and change just the onnxruntime version unfortunately. logs |
|
I got it working, give me a bit to upload the image and do a writeup :) Thanks @mertalev for the help! |
|
Want to use my image? In your
https://github.com/Snuupy/immich/pkgs/container/immich-machine-learning Want to build your own image? Here's how I did it:
In the
to then do a |
|
tested this morning seems to work, its scrapping ! |
|
Nice ! Thank you @Snuupy @alextran1502 can you merge it into the next version, please? ;) |
|
Yes. for me: 105: working let me look at what commits were between 105.1 and 106.4... I'll try again with env vars removed, maybe new onnxruntime -> no need for env vars Edit: I removed the env vars and it stopped compiling forever, I get these in logs and smart search is broken now. Edit 2: I re-ran the facial recognition (a 2nd? 3rd? time) and now the face clusters show up under Explore but smart search like searching for a "car" still is broken, I get nonsense results (I used to get pictures/videos of a car), let me try to clear model cache and see if that fixes anything... Edit 3: nope, smart search still broken. Facial detection works. Any other ideas to try? |
|
Interesting that the env vars were causing that issue for you. Two ideas:
|
|
now it says: but this is not true, the smart search results are non-sensical it returns results, but they are bogus (and the same) results each time |
It's true, it's just that the compiled OpenVINO model probably has a mistake that makes it produce different outputs. Was smart search working well in 105.1? Does face detection behave any differently with mimalloc removed? |
Ah. Is there a way for us to fix it?
Yes. When I switch back to cpu I get correct results. What is the fix for this?
I only did 1) (setting ARG DEVICE), haven't really tested it actually |
There are a few things we can try, but it's ultimately up to OpenVINO. I have some modified models with shape inference and static dimensions lying around that could make it easier for it OpenVINO to know how the model should work. Let me see if I can link one for you to try. |
|
Oh, that's really interesting! Could you share:
|
sorry, was testing a bunch of changes so the code and docker image are out of sync (I went as far as trying to revert back to a specific commit and applying all my changes on top again, but I didn't realize I also had to build the server image and ran out of time on my end). can you please test smart search? that was still broken for me. try to search for a car or something. |
Update: Without OpenVINO, significantly more faces are found. Also Smart Search seems to work. Smart search doesn't really seem to work: Queries like 'beach' or 'grass' return plausible result, but more specific queries (e.g. 'desk') do not.
I'm using the
Intel Pentium G4600
I'm not sure about that, I tried different versions of Immich with and without OpenVINO acceleration every now and then.
Also, I noticed that pressing the And a last remark: I also got errors as reported here, not sure if this is relevant. |
yeah smart search is broken, it's quite obvious when it's working searching for car gets you a car, receipt gets you receipts, a sign gets you a sign, etc. |
|
Just FYI, face detection works works for me with |
|
I upgraded main to use the latest Search produces very wrong model outputs as mentioned by others. After trying a few things, changing the models to have static dimensions gives the expected results. I'll upload revised models to address this for now and let y'all know when they're up. |
|
As someone who is an experienced kubernetes engineer but who knows absolutely nothing about AI/ML models, I just wanted to say thanks all for your continued knowledge and effort on this issue. I appreciate it |
|
New search models are up! You can delete your model cache volume to make it download the updated models. This only fixes smart search being inaccurate, though - face detection still needs an image with the above changes (or main) until the next release. Let me know if you still have issues! |
Awesome news!! Much appreciated and a long time coming. Is there a PR image that I can use to test this out? |
|
This image is pinned so it won't update with main: |
|
After I updated the latest "CLIP" model (I used "XLM-Roberta-Large-Vit-B-16Plus"), the recognition was twice as slow，Using a "UHD730", Is this an inevitable problem? |
+1 |
The performance will depend heavily on how good the processor's iGPU is relative to the CPU. Can you confirm that the results are correct / same as CPU? |
|
Can you clarify if you mean searching is slower than on CPU, or if processing jobs is slower? I have an idea if it's the latter. |
|
I'm sorry. Maybe I wasn't clear enough. Using "postman", using the same image, test "visual" and "textual" many times, averaging the response time
Here are the test results in my current environment, and 607ms is "obtrusive" |
I only tested "car" and "车", and both models returned the same results using CPU and iGPU (at least from the retrieved photos). |
|
Thanks, that chart is very helpful. It does seem like the XLM visual model is worse than before. I'm curious how many models this affects; the default is fine, at least. It might be specific to M-CLIP models like this one, or it could possibly affect larger visual models in general. ViT-B-32__openai
XLM-Roberta-Large-Vit-B-16Plus
|
|
:-( Hi, seem ghcr.io/immich-app/immich-machine-learning:main-openvino@sha256:fb55668b598823f3101174ae3f7e6a1911a894523cc93f69fbafe86a175ebea4 is not available. So I tried using the latest main-openvino image from today... Seems I can't load clip model... I haven't been using the OpenVino image for quite some time since it was broken from 1.99.0. |
I made an issue for this here. My guess is that it's doing an optimization that ends up slowing down the model. I don't think this is really an Immich issue, though. |
Could you share more about the server environment, like distro, kernel version, etc.? If it's Ubuntu, you can try installing |
It worked today using this: === Works now with the main-openvino image! |
|
Awesome! With that, I think we can close this issue at this point. |
and others
The bug
When I try launching the face detection, whatever model I use, I get the following error:
Regular Smart Search proceeds without issue.
The OS that Immich Server is running on
Proxmox 8.1 (6.5 Linux Kernel)
Version of Immich Server
1.99.0
Version of Immich Mobile App
1.99.0
Platform with the issue
Your docker-compose.yml content
Your .env content
Reproduction steps
Additional information
My processor is an Intel N100.
Previously to 1.99.0, the face detection was working, but I had issues with the smart search, so I guess it's hard to get all of it with OpenVino haha.
Thank you for the gigantic work up to now!
Leo
The text was updated successfully, but these errors were encountered: