tensor_filter TFLite subplugin - XNNPACK delegate @open sesame 11/29 12:40 #3577
Conversation
TFLite libray may be built to use default delegates when no delegate is explicitly bound to the interpreter. In case default delegates are not supported by TFLite filter, system may crash. Make sure that no default delegate is applied by using relevant interpreter builder API available from TFLite 2.4 onward. Signed-off-by: Julien Vuillaumier <julien.vuillaumier@nxp.com>
Signed-off-by: Julien Vuillaumier <julien.vuillaumier@nxp.com>
jvuillaumier
requested review from
again4you,
anyj0527,
dongju-chae,
gichan-jang,
helloahn,
jaeyun-jung,
jijoongmoon,
kimjh12,
kparichay,
leemgs,
lhs8928,
myungjoo,
wooksong,
zhoonit and
a team
as code owners
|
📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #3577. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://nnstreamer.mooo.com/. |
|
|
cibot:
|
|
XNNPACK delegate requires fixed addresses for input and output tensors, for consecutive invoke(). Therefore input and output tensors data coming from GStreamer buffers has to copied into tensors buffer allocated by TFLite runtime. Signed-off-by: Julien Vuillaumier <julien.vuillaumier@nxp.com>
jvuillaumier
force-pushed
the
nnstreamer-tflite-xnnpack
branch
from
41063e9
to
65bc020
Compare
|
Patchset updated to fallback using (TfLiteTensor*)->data.raw (deprecated since TFLite 2.2) instead of more recent (TfLiteTensor*)->data.data as it is breaking some builds. |
|
|
anyj0527
changed the title
There was a problem hiding this comment.
@jvuillaumier, 💯 All CI checkers are successfully verified. Thanks.
There was a problem hiding this comment.
Great thanks for this contribution. I've packaged tflite v2.6.0+ for android and tizen without default delegate feature to address this default delegate issue.
But using tflite::ops::builtin::BuiltinOpResolverWithoutDefaultDelegates resolver seems more elegant considering those who manually build nnstreamer with their own tflite support.
LGTM✔
Changes proposed in this PR:
2 items related to XNNPACK delegate usage with TfLite framework (tested on TfLite 2.6):
Peculiarity of TfLite XNNPACK delegate usage in TfLite is that both input and output tensors buffer addresses are cached within XNNPACK runtime during first invoke(). NNStreamer tensor_filter input/output buffer memory addresses change from an inference to an other, therefore i/o buffer data has to be copied between Gst memory and TfLite allocated memory for tensors. Failure to do so typically leads to system crash as TfLite interpreter tries to access during subsequent invoke() to the GStreamer buffers populated during 1st invoke that have chances to have been deallocated and unmapped.
Currently only output tensors data copy is provisioned for XNNPACK case.
When TfLite library is built with XNNPACK delegate support, XNNPACK delegate is used as a default delegate for the graph, even though not explicitly applied via ModifyGraphWithDelegate().
Issue is that XNNPACK usage requires tensor data copies so for NNStreamer case, XNNPACK should only be used when explicitly selected by the user (custom=Delegate:XNNPACK) so that TfLite subplugin is aware about XNNPACK usage and handle input/output tensors accordingly.
API for OpResolver has been introduced in TfLite 2.4 for this:
tflite::ops::builtin::BuiltinOpResolverWithoutDefaultDelegates resolver;
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/register.h#L36
Self evaluation:
SSAT test suite passed
How to evaluate: