Coral USB is not working

[Martin2kid]

• Raspberry Pi Deep PanTilt version: 4 with 4GB & 64SD
• Python version: 3.7
• Operating System: Debian Buster

Description

Ran "rpi-deep-pantilt detect --edge-tpu --loglevel=INFO" after installing Coral USB per instruction & Googl's (Note it is now 2.1 version Runtime)"

Error: "RuntimeError: Internal: Unsupported data type in custom op handler: 0Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare."

It works without "--edge-tpu"

What I Did: multiple re-install & reboot & same result

Paste the command(s) you ran and the output.: "rpi-deep-pantilt detect --edge-tpu" and "rpi-deep-pantilt detect --edge-tpu --loglevel=INFO" ----generate same error.

Whole output of error from CMD
(.venv) pi@raspberrypi:~/rpi-deep-pantilt $ rpi-deep-pantilt detect --edge-tpu
INFO: Initialized TensorFlow Lite runtime.
Traceback (most recent call last):
  File "/home/pi/rpi-deep-pantilt/.venv/bin/rpi-deep-pantilt", line 8, in <module>
    sys.exit(main())
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/rpi_deep_pantilt/cli.py", line 107, in main
    cli()
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/rpi_deep_pantilt/cli.py", line 52, in detect
    model = SSDMobileNet_V3_Coco_EdgeTPU_Quant()
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/rpi_deep_pantilt/detect/ssd_mobilenet_v3_coco.py", line 56, in __init__
    self.tflite_interpreter.allocate_tensors()
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/tensorflow_core/lite/python/interpreter.py", line 244, in allocate_tensors
    return self._interpreter.AllocateTensors()
  File "/home/pi/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/tensorflow_core/lite/python/interpreter_wrapper/tensorflow_wrap_interpreter_wrapper.py", line 106, in AllocateTensors
    return _tensorflow_wrap_interpreter_wrapper.InterpreterWrapper_AllocateTensors(self)
RuntimeError: Internal: Unsupported data type in custom op handler: 0Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare.

If there was a crash, please include the traceback here.

[leigh-johnson]

Thank you for reporting this bug!

What version of TensorFlow are you using?

pip show tensorflow

[Martin2kid]

Leigh,

You are very welcome & thank you very much for responding!

It's
Name: tensorflow
Version: 2.0.0
(downloaded from your link $ pip install https://github.com/leigh-johnson/Tensorflow-bin/blob/master/tensorflow-2.0.0-cp37-cp37m-linux_armv7l.whl?raw=true

with
Name: tflite-runtime
Version: 2.1.0
(I was looking for 2.00 Version & couldn't find it anywhere)

It was clean install over buster (only rpi-deep-pantilt + update)

[Martin2kid]

Leigh,

Please see following link referencing that same issue & Namburger and Feranick stated problem solved & case closed, but obviously it doesn't appear that way in case of Pi4.

google-coral/edgetpu#44

I left a comment referring to this issue at the site.

[leigh-johnson]

Thank you for the additional info!

Could you try the following?

1. Uninstall tflite-runtime

pip remove tflite_runtime

I'm pretty sure this is a standalone TensorFlow lite runtime, which we don't need here. We're installing the full TensorFlow wheel, and then initializing a TF lite interpreter (ref: https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/rpi_deep_pantilt/detect/ssd_mobilenet_v3_coco.py#L49)

I'll add better docs around getting the Coral setup!

2. sudo apt-get install libedgetpu1-std=12-1

My reference implementation might be using a different version of this library, which might be the culprit.

apt-cache policy libedgetpu1-std
libedgetpu1-std:
  Installed: 12-1
  Candidate: 13.0
  Version table:
     13.0 500
        500 https://packages.cloud.google.com/apt coral-edgetpu-stable/main armhf Packages
 *** 12-1 100
        100 /var/lib/dpkg/status

[Martin2kid]

You work long hours too!

I appreciate your attention & certainly will.

I'm in the middle of testing your rpi-deep-pantilt detect on my direct driven brushless motor Pan & Tilt control HW-device that I firmly believe that it will provide minimal latency unlike any other conventional mechanical gear driven reduction type, similar to that of standard servo motors or super expensive Flir's and also direct driven stepper motor type's that relate to annoying constant noise plus awkward positional step motions, thus provide promissing real time object or face tracking that matches speed of CPU processing in real time & minimize PID processing resource .

So far, it's been working great & I'm getting fluid like positional movement (rather slow but you deserve all the credit--(I didn't think Pi's GPIO can provide such smooth PWM output unless I'm using Arduino via Serial output of Pi attached & use of VAR servo) but I'm having to adjust inner PID within the software side & PID at HW side together. (driving me crazy because of slow SW input speed).

I would like to prepare back up of what I have so far tonight or tomorrow morning --(I'm exhausted tonight) & follow your response tomorrow and I'll report back to you tomorrow. (I'm in Virginia USA & it is 1:04 AM now)

I'll also appreciate if you can also look into issue of person's face to being in centroid of frame when you have chance please. (My first priority)

I'll be happy to share my HW brushless Pan & Tilt device implementation with you in private conversation & providing you with video or perhaps even with actual prototype.

I'm newbie in Software, but I'm a professional carpenter worked on Washington DC Kennedy Performing Art Center Remodeling project & Martin Luther King Library specialized flooring project including Gym, Dance Studio, Sports Flooring, audotorium construction and ongoing.

[Martin2kid]

I'm getting this output

(.venv) pi@raspberrypi:~/rpi-deep-pantilt $ sudo apt-get install libedgetpu1-std=12-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '12-1' for 'libedgetpu1-std' was not found

[Martin2kid]

Run following;
pip uninstall tflite_runtime
verified installed libedgetpu1-std is 13.0 (since I couldn't get 12-1)
Run;
rpi-deep-pantilt detect --edge-tpu

same error output; "RuntimeError: Internal: Unsupported data type in custom op handler: 0Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare."

It still run without --edge-tpu

I also tried new install of Buster & only rpi-deep-pantilt related install package and I2c & picam enabled only, same error.

[leigh-johnson]

Hey @Martin2kid! Sorry about the dead air on this issue. I am going to release Docker images in v1.2.0 and script the platform-level installation with pinned versions of system-level dependencies. Right now, the manual installation instructions feel like they're already out of date.

I haven't upgraded to libedgetpu1-std@13.0 to try and reproduce this issue yet, will let you know if that ends up being the cause.

[leigh-johnson]

I am extremely interested in your brushless motor setup, by the way. Is this the module you're working with? https://www.flir.com/oem/pan-tilt-systems/

My email is hi@leighjohnson.me if you'd like to share more about your prototype and use case.

I'll be happy to share my HW brushless Pan & Tilt device implementation with you in private conversation & providing you with video or perhaps even with actual prototype.

[Martin2kid]

Leigh,

Thank you very much following up & updating me.

I saw couple of Pi4 & TensorFlow 2.1 related issue here too & may have something to do with this issue for your reference; https://www.raspberrypi.org/forums/viewtopic.php?t=262595

https://raspberrypi.stackexchange.com/questions/107483/error-installing-tensorflow-cannot-find-libhdfs-so

It's not a Flir or Raytheon's (same product, they used precisely machiened worm-geared mechanism with stepper setup) which still produce mechanical latency) of I'll send you info to your email

[Namburger]

@Martin2kid The problem here is that this repo is using tf.lite.experimental.load_delegate which is a tensorflow api:
https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/rpi_deep_pantilt/detect/facessd_mobilenet_v2.py#L53

I suggest using the tflite_runtime api instead which is the reason why it was upgraded recently in the first place. You can see here as an example to do this:
https://github.com/google-coral/tflite/blob/master/python/examples/detection/detect_image.py#L57

[Martin2kid]

@Martin2kid The problem here is that this repo is using tf.lite.experimental.load_delegate which is a tensorflow api:
https://github.com/leigh-johnson/rpi-deep-pantilt/blob/master/rpi_deep_pantilt/detect/facessd_mobilenet_v2.py#L53

I suggest using the tflite_runtime api instead which is the reason why it was upgraded recently in the first place. You can see here as an example to do this:
https://github.com/google-coral/tflite/blob/master/python/examples/detection/detect_image.py#L57

Nam Vu,
Great & appreciate your comment and link!!!

[leigh-johnson]

Thank you @Namburger! 🙏 Appreciate the example code. I'll fix this in my next release.

[IanShow15]

for anyone that needs a temporary fix, you can do the following:

1. in your venv install this pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0-cp37-cp37m-linux_aarch64.whl
2. go into your python environment's lib/site-packages and find rpi-deep-pantilt, edit both facessd_mobilenet_v2.py and ssd_mobilenet_v3_coco.py with the following:

• add import tflite_runtime.interpreter as tflite at the top

• replace line 50-55
  self.tflite_interpreter = tf.lite.Interpreter( model_path=self.model_path, experimental_delegates=[ tf.lite.experimental.load_delegate(self.EDGETPU_SHARED_LIB) ] )

• with the following
  self.tflite_interpreter = tflite.Interpreter( model_path=model_file, experimental_delegates=[ tflite.load_delegate(EDGETPU_SHARED_LIB) ])

[Yuji09]

Complementing to IanShow15's contribution, the following worked for my RPI4:

1. in your venv, install this pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp37-cp37m-linux_armv7l.whl

2. the same procedure as lanShow15, but at the last step replace line 50-55 with the following instead:
   self.tflite_interpreter = tflite.Interpreter( model_path=self.model_path, experimental_delegates=[ tflite.load_delegate(self.EDGETPU_SHARED_LIB) ])

[TCIII]

Hi Yuji09,

Like you, I am running a Rpi 4B+ also and I experienced the same problem described in this issue when I tried to run detect with the Coral TPU.

Modifying the Python code per your instructions did the trick as I am now detecting objects at 27-28 frames/second without any errors at startup.

Nice work, much appreciated.

[leigh-johnson]

Hey y'all, this should be fixed in versions >= 1.2.0 - let me know experience any issues!

I'm also looking into why edgetpulib needs to package its own Interpreter and Delegate base classes, this is non-standard.

My understanding is that these interfaces are provided by TensorFlow's tensorflow.lite lib, and that tf.lite.experimental.load_delegate can be used to load a library that implements a custom delegate class. The custom delegate is responsible for registering a kernel node, which parses a TensorFlow graph and "claims" operations it knows how to execute.

The Unsupported data type in custom op handler error raised by using libedgetpu's shared object with tensorflow.lite smells like a SWIG typemap issue, but I'd need more context from Coral folks to get to the bottom of that issue.

[Namburger]

@leigh-johnson

I'm also looking into why edgetpulib needs to package its own Interpreter and Delegate base classes, this is non-standard.

This is indeed non standard and we apologize. The issue is that libedgetpu started depending on tensorflow as a dependency, so you'd need the exact tensorflow commit that we used to build libedgetpu in order to be compatible. That's why we packaged tflite_runtime package that is built from the same commit as libedgetpu.so. The deeper reason why this is so is because our library isn't open source which means users cannot build their own .so to fit with the tensorflow package. We are working diligently on this issue. Thanks for this repo

[leigh-johnson]

@Namburger

No worries / no apology necessary! Thank you for jumping in and helping everyone in this issue. ❤️

Let me know if there's anything I can do to assist you in the meantime!

The Google GDE program operates behind Google NDAs (we don't work for or otherwise represent Google though). GDEs often organize/participate in early access programs if you're looking for extra feedback, testing, support before open sourcing. I have a couple USB accelerators, the Dev Board, and I'm more than happy to do bonkers things like try the SOM on an RPI running an aarch64 distribution like Mendel, Fedora, Arch, etc.

Shoot me an email hi@leighjohnson.me if you want to connect and chat more about working with the GDE program.

[Namburger]

@leigh-johnson I mentioned you to our FAE team and our Developer Advocates, they'll contact you if we need your helps!