[Runtime] EdgeTPU runtime for Coral Boards #4698

tmoreau89 · 2020-01-13T20:33:26Z

This PR extends the TFLite runtime to support edgeTPU-equipped Coral boards in order to measure inference time of models on edgeTPU with TVM RPC.

Instructions to run the EdgeTPU runtime experiments

Coral Board setup

You'll need to follow these instructions: https://coral.ai/docs/dev-board/get-started/

# Clone TensorFlow, and prepare the library dir
# Note the older version of TF that we'll need to use
git clone https://github.com/tensorflow/tensorflow --recursive --branch=1.8.0
cd tensorflow
mkdir tensorflow/lite/tools/make/gen
mkdir tensorflow/lite/tools/make/gen/generic-aarch64_armv8-a
mkdir tensorflow/lite/tools/make/gen/generic-aarch64_armv8-a/lib

# TF dependence
cd ~ && git clone https://github.com/google/flatbuffers.git
cd flatbuffers && cmake -G "Unix Makefiles" && make && sudo make install

# EdgeTPU lib
cd ~ && git clone https://github.com/google-coral/edgetpu.git

Cross compile tflite static library on x86 machine

# Prerequisites 
sudo apt-get update
sudo apt-get install crossbuild-essential-arm64

# cross-compile tflite library (note you need to use older version)
git clone https://github.com/tensorflow/tensorflow.git --recursive --branch=1.8.0
cd tensorflow
./tensorflow/lite/tools/make/download_dependencies.sh
./tensorflow/lite/tools/make/build_aarch64_lib.sh
# Copy the tensorflow lib over to your coral board
scp tensorflow/lite/tools/make/gen/generic-aarch64_armv8-a/lib/libtensorflow-lite.a  mendel@coral:/home/mendel/tensorflow/tensorflow/lite/tools/make/gen/generic-aarch64_armv8-a/lib/

Build TVM runtime on Coral Board

cd ~ && git clone --recursive --branch=master https://github.com/apache/incubator-tvm.git tvm
cd tvm && mkdir build && cp cmake/config.cmake build
echo 'set(USE_GRAPH_RUNTIME_DEBUG ON)' >> build/config.cmake
echo 'set(USE_TFLITE ON)' >> build/config.cmake
echo 'set(USE_TENSORFLOW_PATH /home/mendel/tensorflow)' >> build/config.cmake
echo 'set(USE_EDGETPU /home/mendel/edgetpu)' >> build/config.cmake
cd build && cmake ..
make runtime -j4

Execute the RPC server on Coral

First, follow this guide to set up a tracker for your remote devices: https://docs.tvm.ai/tutorials/autotvm/tune_relay_arm.html#start-rpc-tracker.
On the coral, once TVM runtime has been built, execute:

PYTHONPATH=/home/mendel/tvm/python:$PYTHONPATH python3 -m tvm.exec.rpc_server --tracker $TVM_TRACKER_HOST:$TVM_TRACKER_NODE --key coral

Evaluate MobileNet on Coral board

Execute the following python script:

import numpy as np

import tvm
from tvm import autotvm, relay
from tvm.contrib import tflite_runtime

target = "cpu"

# Note: replace "tracker" and 9191 with your tracker host and port name
remote = autotvm.measure.request_remote("coral", "tracker", 9191, timeout=60)
ctx = remote.cpu(0)

if target == "edge_tpu":
    tflite_fp = "mobilenet_v2_1.0_224_quant_edgetpu.tflite"
else:
    tflite_fp = "mobilenet_v2_1.0_224_quant.tflite"
input_data = np.random.rand(1,224,224,3).astype("uint8")
with open(tflite_fp, 'rb') as f:
    runtime = tflite_runtime.create(f.read(), ctx, runtime_target=target)
    runtime.set_input(0, tvm.nd.array(input_data, ctx))
    ftimer = runtime.module.time_evaluator("invoke", ctx,
            number=10,
            repeat=3)
    times = np.array(ftimer().results) * 1000
    print("It took {0:.2f}ms to run mobilenet".format(np.mean(times)))

Upon running it, you'll get:
It took 143.74ms to run mobilenet

Now, set target = "edge_tpu" and you'll get:
It took 3.22ms to run mobilenet

Notable interface changes

The TFLite runtime API does not expose the allocate() method anymore, and tensor allocation is done as part of the initialization process.

ZihengJiang · 2020-01-13T20:39:31Z

python/tvm/contrib/tflite_runtime.py

@@ -18,7 +18,7 @@
 from .._ffi.function import get_global_func
 from ..rpc import base as rpc_base

-def create(tflite_model_bytes, ctx):
+def create(tflite_model_bytes, ctx, target_edgetpu=False):


instead of a boolean argument, try to use a target string for future expansion: target='edge_tpu'/'cpu'

thanks for the suggestion, I've made the changes

ZihengJiang · 2020-01-13T20:41:02Z

python/tvm/contrib/tflite_runtime.py

        return TFLiteModule(fcreate(bytearray(tflite_model_bytes), ctx))
-    fcreate = get_global_func("tvm.tflite_runtime.create")
+    if target_edgetpu:
+        fcreate = get_global_func("tvm.edgetpu_runtime.create")


if these two create function share the same arguments, we can unify them as one create function with different returned runtime

Unification here is less desired due to the fact that we won't always want to build the edgeTPU runtime when building the TFLite runtime. The limitation is that we need to build TVM with the edgeTPU library which comes in a separate repo; it's an extra software dependence that is not always wanted for users of vanilla TFLite.

ZihengJiang · 2020-01-13T20:42:36Z

src/runtime/contrib/tflite/tflite_runtime.cc

-  ctx_ = ctx;
-}
+  // Build interpreter
+  if (tflite::InterpreterBuilder(*model, resolver)(&interpreter_) != kTfLiteOk) {


we can define macro for TFLite error checking: CHECK_STATUS(cond, msg)

thanks for the suggestion, this should be fixed by now

ZihengJiang · 2020-01-13T20:44:42Z

for allocate, does tflite runtime removed the AllocateTensors API or just EdgeTPU does not need it?

tmoreau89 · 2020-01-16T00:35:49Z

@ZihengJiang thanks for the feedback! The TFLite Interpreter still has AllocateTensors; however I wan't sure if we'd ever need to call it separately from interpreter initialization. If you believe that we need to decouple them, I can revert the interface change.

tmoreau89 · 2020-01-16T01:58:30Z

@ZihengJiang I should have addressed all of your comments by now; let me know if you're happy with the changes

ZihengJiang · 2020-01-16T19:43:18Z

Looks good! Thanks! @tmoreau89

Msabih · 2020-08-05T13:10:36Z

@tmoreau89
I have tried the setup with the same versions of tvm/tensorflow on the host and the board and the "cpu" part of the inference works fine. But when I set the target to edge_tpu, I get this error on the rpc server

ERROR: Internal: Unsupported data type: 0
ERROR: Node number 0 (edgetpu-custom-op) failed to prepare

And on the host machine, it says

 File "tvm_inference.py", line 21, in <module>
    runtime = tflite_runtime.create(f.read(), ctx, runtime_target=target)

  File "/home/sabih/Documents/phd_work/MAP_WORK/tvm_env/tvm/python/tvm/contrib/tflite_runtime.py", line 49, in create
    return TFLiteModule(fcreate(bytearray(tflite_model_bytes), ctx))

  File "/home/sabih/Documents/phd_work/MAP_WORK/tvm_env/tvm/python/tvm/_ffi/_ctypes/function.py", line 207, in __call__
    raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (3) /tvm_env/tvm/build/libtvm.so(TVMFuncCall+0x69) [0x7f2fb63f8489]
  [bt] (2) /tvm_env/tvm/build/libtvm.so(std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::RPCModuleNode::WrapRemote(void*)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)+0x46) [0x7f2fb644ad36]
  [bt] (1) /tvm_env/tvm/build/libtvm.so(tvm::runtime::RPCSession::CallFunc(void*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*, void* (*)(int, tvm::runtime::TVMArgValue const&), tvm::runtime::PackedFunc const*)+0x2c8) [0x7f2fb6454168]
  [bt] (0) /tvm_env/tvm/build/libtvm.so(+0xc21d6b) [0x7f2fb6450d6b]
  File "/tvm_env/tvm/src/runtime/rpc/rpc_session.cc", line 993
TVMError: Check failed: code == RPCCode: :kReturn: code=4

The inference directly on the edge TPU works fine.

amai-gsu · 2023-11-08T00:18:22Z

@tmoreau89 I have tried the setup with the same versions of tvm/tensorflow on the host and the board and the "cpu" part of the inference works fine. But when I set the target to edge_tpu, I get this error on the rpc server

ERROR: Internal: Unsupported data type: 0
ERROR: Node number 0 (edgetpu-custom-op) failed to prepare

And on the host machine, it says

 File "tvm_inference.py", line 21, in <module>
    runtime = tflite_runtime.create(f.read(), ctx, runtime_target=target)

  File "/home/sabih/Documents/phd_work/MAP_WORK/tvm_env/tvm/python/tvm/contrib/tflite_runtime.py", line 49, in create
    return TFLiteModule(fcreate(bytearray(tflite_model_bytes), ctx))

  File "/home/sabih/Documents/phd_work/MAP_WORK/tvm_env/tvm/python/tvm/_ffi/_ctypes/function.py", line 207, in __call__
    raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (3) /tvm_env/tvm/build/libtvm.so(TVMFuncCall+0x69) [0x7f2fb63f8489]
  [bt] (2) /tvm_env/tvm/build/libtvm.so(std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::RPCModuleNode::WrapRemote(void*)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)+0x46) [0x7f2fb644ad36]
  [bt] (1) /tvm_env/tvm/build/libtvm.so(tvm::runtime::RPCSession::CallFunc(void*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*, void* (*)(int, tvm::runtime::TVMArgValue const&), tvm::runtime::PackedFunc const*)+0x2c8) [0x7f2fb6454168]
  [bt] (0) /tvm_env/tvm/build/libtvm.so(+0xc21d6b) [0x7f2fb6450d6b]
  File "/tvm_env/tvm/src/runtime/rpc/rpc_session.cc", line 993
TVMError: Check failed: code == RPCCode: :kReturn: code=4

The inference directly on the edge TPU works fine.
have you solved this issue? i got a same one.

tmoreau89 requested review from tqchen and ZihengJiang January 13, 2020 20:33

ZihengJiang reviewed Jan 13, 2020

View reviewed changes

ZihengJiang requested changes Jan 13, 2020

View reviewed changes

tqchen assigned ZihengJiang Jan 14, 2020

tmoreau89 added 12 commits January 15, 2020 16:21

edgetpu runtime

bb336b7

edgetpu unit testing

31e589e

fix typo

c52315a

rebase fixes for refactor

9bfe3d8

unit test update

efdcf96

adding default option for EDGETPU compilation

b3aec77

cmake fix

a79b8b6

update api; remove prints

c88dee3

comment out tests that are problematic

9413c45

source refactor to make EdgeTPURuntime a subclass of TFLiteRuntime

3cbd6ef

unit test fix

ef41c59

unit test edits

92596c0

tmoreau89 added 3 commits January 15, 2020 17:25

integrating feedback from ziheng

4d216ce

confirm that unit test works with correct tf/tflite versioning

2b39910

adding includes

bbcf752

tmoreau89 force-pushed the tflite_runtime branch from 29bb50b to bbcf752 Compare January 16, 2020 01:36

improvements on error reporting

7949234

ZihengJiang approved these changes Jan 16, 2020

View reviewed changes

tqchen merged commit 31021d2 into apache:master Jan 16, 2020

tqchen added the status: accepted label Jan 16, 2020

tmoreau89 deleted the tflite_runtime branch February 13, 2020 21:26

alexwong pushed a commit to alexwong/tvm that referenced this pull request Feb 26, 2020

[Runtime] EdgeTPU runtime for Coral Boards (apache#4698)

404586c

alexwong pushed a commit to alexwong/tvm that referenced this pull request Feb 28, 2020

[Runtime] EdgeTPU runtime for Coral Boards (apache#4698)

0cef36d

zhiics pushed a commit to neo-ai/tvm that referenced this pull request Mar 2, 2020

[Runtime] EdgeTPU runtime for Coral Boards (apache#4698)

795cf73

ZihengJiang mentioned this pull request Sep 17, 2020

TVM v0.7 Release Note Candidate #6486

Closed

tk1012 mentioned this pull request May 13, 2021

[Fix][Runtime] Use flatBuffersBuffer_ in EdgeTPURuntime::Init() #8034

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Runtime] EdgeTPU runtime for Coral Boards #4698

[Runtime] EdgeTPU runtime for Coral Boards #4698

tmoreau89 commented Jan 13, 2020 •

edited

ZihengJiang Jan 13, 2020

tmoreau89 Jan 16, 2020

ZihengJiang Jan 13, 2020

tmoreau89 Jan 16, 2020

ZihengJiang Jan 13, 2020

tmoreau89 Jan 16, 2020

ZihengJiang commented Jan 13, 2020

tmoreau89 commented Jan 16, 2020

tmoreau89 commented Jan 16, 2020

ZihengJiang commented Jan 16, 2020

Msabih commented Aug 5, 2020 •

edited

amai-gsu commented Nov 8, 2023

[Runtime] EdgeTPU runtime for Coral Boards #4698

[Runtime] EdgeTPU runtime for Coral Boards #4698

Conversation

tmoreau89 commented Jan 13, 2020 • edited

Instructions to run the EdgeTPU runtime experiments

Coral Board setup

Cross compile tflite static library on x86 machine

Build TVM runtime on Coral Board

Execute the RPC server on Coral

Evaluate MobileNet on Coral board

Notable interface changes

ZihengJiang Jan 13, 2020

Choose a reason for hiding this comment

tmoreau89 Jan 16, 2020

Choose a reason for hiding this comment

ZihengJiang Jan 13, 2020

Choose a reason for hiding this comment

tmoreau89 Jan 16, 2020

Choose a reason for hiding this comment

ZihengJiang Jan 13, 2020

Choose a reason for hiding this comment

tmoreau89 Jan 16, 2020

Choose a reason for hiding this comment

ZihengJiang commented Jan 13, 2020

tmoreau89 commented Jan 16, 2020

tmoreau89 commented Jan 16, 2020

ZihengJiang commented Jan 16, 2020

Msabih commented Aug 5, 2020 • edited

amai-gsu commented Nov 8, 2023

tmoreau89 commented Jan 13, 2020 •

edited

Msabih commented Aug 5, 2020 •

edited