Coral TPU: "Unable to run inference" unhandled error.

[mac22321]

Area of Concern

• Server

Server Info

Server version:   2.9.5
System:           Windows
Operating System: Windows (Windows 11 24H2)
CPUs:             Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Intel)
                  1 CPU x 6 cores. 12 logical processors (x64)
GPU (Primary):    Quadro P1000 (4 GiB) (NVIDIA) 
                  Driver: 552.74, CUDA: 12.4 (up to: 12.4), Compute: 6.1, cuDNN: 
System RAM:       32 GiB
Platform:         Windows
BuildConfig:      Release
Execution Env:    Native
Runtime Env:      Production
Runtimes installed:
  .NET runtime:     9.0.0
  .NET SDK:         7.0.410
  Default Python:   Not found
  Go:               Not found
  NodeJS:           18.17.0
  Rust:             Not found
Video adapter info:
  NVIDIA Quadro P1000:
    Driver Version     31.0.15.5274
    Video Processor    Quadro P1000
System GPU info:
  GPU 3D Usage       2%
  GPU RAM Usage      3.8 GiB
Global Environment variables:
  CPAI_APPROOTPATH = <root>
  CPAI_PORT        = 32168

Describe the bug
Using CPAI within the context of Blue Iris NVR software. When Coral TPU object detection module is enabled, an unhandled exception frequently occurs with error message "Unable to run inference: There is at least 1 reference to internal data in the interpreter in the form of a numpy array or slice. Be sure to only hold the function returned from tensor() if you are using raw data access.".

Actual error:

T-394 msec  [475 msec]   

[
	{
		"api":"objects",
		"found":{
			"success":true,
			"inferenceMs":7,
			"processMs":85,
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"moduleId":"ObjectDetectionCoral",
			"moduleName":"Object Detection (Coral)",
			"code":200,
			"command":"detect",
			"requestId":"d1c26a48-68ae-4714-8725-807e8d1aa788",
			"inferenceDevice":"TPU",
			"analysisRoundTripMs":161,
			"processedBy":"localhost",
			"timestampUTC":"Thu,
			19 Dec 2024 21:07:38 GMT"}
		}
	,
	{
		"api":"ipcam-dark",
		"found":{
			"success":false,
			"error":"Unable to run inference: There is at least 1 reference to internal data\n      in the interpreter in the form of a numpy array or slice. Be sure to\n      only hold the function returned from tensor() if you are using raw\n      data access.",
			"inferenceMs":0,
			"processMs":80,
			"predictions":[
				]
			,
			"message":"",
			"count":0,
			"moduleId":"ObjectDetectionCoral",
			"moduleName":"Object Detection (Coral)",
			"code":500,
			"command":"custom",
			"requestId":"3634a008-e41c-4179-bd56-697a75ae66f4",
			"inferenceDevice":"TPU",
			"analysisRoundTripMs":198,
			"processedBy":"localhost",
			"timestampUTC":"Thu,
			19 Dec 2024 21:07:38 GMT"}
		}
	]

Expected behavior
CPAI should work similarly to how it does when using CPU/GPU hardware and process the image handed to it, returning either a "No objects found" payload

T-28 msec  [380 msec]  *

[
	{
		"api":"objects",
		"found":{
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":147,
			"processMs":148,
			"analysisRoundTripMs":154,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"detect",
			"requestId":"11dd1b12-31be-461d-9858-6e8b971ac252",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 07:13:35 GMT"}
		}
	,
	{
		"api":"ipcam-combined",
		"found":{
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":150,
			"processMs":150,
			"analysisRoundTripMs":154,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"custom",
			"requestId":"dc6a3e7d-d679-4832-8023-24aee843fe1b",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 07:13:35 GMT"}
		}
	]

or an "object found" payload.

T+206 msec  [576 msec]  *

[
	{
		"api":"objects",
		"found":{
			"message":"Found person,
			chair",
			"count":2,
			"predictions":[
				{
					"label":"person",
					"confidence":0.8436014,
					"y_min":1410,
					"x_min":1347,
					"y_max":1724,
					"x_max":1937}
				,
				{
					"label":"chair",
					"confidence":0.47627652,
					"y_min":810,
					"x_min":0,
					"y_max":1366,
					"x_max":653}
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":237,
			"processMs":237,
			"analysisRoundTripMs":245,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"detect",
			"requestId":"807846c8-be05-44c4-96b4-a0377700a893",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 19:47:03 GMT"}
		}
	,
	{
		"api":"ipcam-combined",
		"found":{
			"message":"No objects found",
			"count":0,
			"predictions":[
				]
			,
			"inferenceDevice":"GPU",
			"inferenceMs":196,
			"processMs":196,
			"analysisRoundTripMs":211,
			"success":true,
			"moduleName":"Object Detection (YOLOv5 .NET)",
			"moduleId":"ObjectDetectionYOLOv5Net",
			"command":"custom",
			"requestId":"7fc2b07d-d5f5-49ef-bedc-446bd14a6e5e",
			"processedBy":"localhost",
			"timestampUTC":"Wed,
			11 Dec 2024 19:47:04 GMT"}
		}
	]

Screenshots

Additional context
https://coral.ai/products/m2-accelerator-ae

[Hardware Resources]

Resource	Device	Status	

<snip>
IRQ 4294967272	Coral PCIe Accelerator	OK	
IRQ 4294967271	Coral PCIe Accelerator	OK	
IRQ 4294967270	Coral PCIe Accelerator	OK	
IRQ 4294967269	Coral PCIe Accelerator	OK	
<snip>

Resource	Device	Status	
<snip>
0xFC000-0xFFFFF	Coral PCIe Accelerator	OK	
0x100000-0x1FFFFF	Coral PCIe Accelerator	OK	
<snip>

[System Drivers]

Name	Description	File	Type	Started	Start Mode	State	Status	Error Control	Accept Pause	Accept Stop	
<snip>
coral	coral	c:\windows\system32\drivers\coral.sys	Kernel Driver	Yes	Manual	Running	OK	Normal	No	Yes	
<snip>

[hpmkruit]

Having similair issues on Agent DVR. Using coral tpu on windows sometimes gives the same error.

[mac22321]

Another oddity, maybe coincidentally, is that this error condition only occurs on one of my many cameras, which is a Reolink Duo 2.
That camera produces high resolution, wide images (two images stitched together horizontally by the hardware: 4608x1728 resulting image).

[mac22321]

Furthermore, CPAI behaves as expected with images from this camera when objects are detected closer to the fence line (in attached photo).

[ChrisMaunder]

The issue occurs in the Coral.AI API when the interpreter is called while it's still processing a previous operation. Large images sent frequently could cause this.

I've added a lock around the inference call that should help with this. It'll be in the next release

[mac22321]

The issue occurs in the Coral.AI API when the interpreter is called while it's still processing a previous operation. Large images sent frequently could cause this.

I've added a lock around the inference call that should help with this. It'll be in the next release

TYVM. 👍

[melyux]

Will the lock queue the inferences?

[ChrisMaunder]

Locks in Python will block execution and wait until the lock is released. Anything blocked will then continue. There is no queue at this point.

CodeProject.AI server does, however, contain a queue for requests, and this queue will hold pending requests. However, the queue is limited in size (1,000 I think) and since the requests are all HTTP calls with timeouts, there is a risk that a long queue could result in the API call itself timing out and being discarded.

My advice: work out how to have smaller images sent to the server. Maybe you can point a lower resolution stream from the camera to CodeProject.AI server, or maybe you can send just a small section of the main image area to the server.