tfjs-tflite gives the error RuntimeError: Aborted(). Build with -s ASSERTIONS=1 for more info.

[RaphaelRoyerRivard]

I'm trying to use the tfjs-tflite package to run a segmentation model but when I load the model, I get the following warning from tflite_web_api_cc_simd.js:9

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

And when I try to run the model, I get the following error from the same line

RuntimeError: Aborted(). Build with -s ASSERTIONS=1 for more info.
    at abort (tflite_web_api_cc_simd.js:9:9277)
    at _abort (tflite_web_api_cc_simd.js:9:59963)
    at tflite_web_api_cc_simd.wasm:0x1e418
    at tflite_web_api_cc_simd.wasm:0x2c4ac6
    at tflite_web_api_cc_simd.wasm:0x39623
    at tflite_web_api_cc_simd.wasm:0x49174
    at tflite_web_api_cc_simd.wasm:0x3084f7
    at tflite_web_api_cc_simd.wasm:0x17b2a
    at TFLiteWebModelRunner$Infer [as Infer] (eval at new_ (tflite_web_api_cc_simd.js:9:37941), <anonymous>:8:10)
    at module$exports$google3$third_party$tensorflow_lite_support$web$tflite_web_api_client.TFLiteWebModelRunner.infer (tflite_web_api_client.js?965d:1745:134)

I don't understand why it's being aborted and what I should build with -s ASSERTIONS=1 to get more info...
Can somebody explain to me what's the procedure?

[RaphaelRoyerRivard]

Maybe @jinjingforever or @mattsoulanille could have an idea of what I should do?

[RaphaelRoyerRivard]

When I try to build tfjs-tflite with those parameters, I get that error

rroyerrivard@rroyerrivard-MBP tfjs-tflite % yarn build -s ASSERTIONS=1
yarn run v1.22.11
$ bazel build :tfjs-tflite_pkg -s ASSERTIONS=1
ERROR: Skipping 'ASSERTIONS=1': no such target '//tfjs-tflite:ASSERTIONS=1': target 'ASSERTIONS=1' not declared in package 'tfjs-tflite' defined by /Users/rroyerrivard/Projects/tfjs/tfjs-tflite/BUILD.bazel
WARNING: Target pattern parsing failed.
ERROR: no such target '//tfjs-tflite:ASSERTIONS=1': target 'ASSERTIONS=1' not declared in package 'tfjs-tflite' defined by /Users/rroyerrivard/Projects/tfjs/tfjs-tflite/BUILD.bazel
INFO: Elapsed time: 0.473s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[mattsoulanille]

I'm pretty sure it's asking you to build the TFLite wasm bundle with -s ASSERTIONS=1. Unfortunately, with our current setup, we compile the TFLite wasm bundle in google3 (Google's internal monorepo) and host the results externally, meaning there's no easy way for you to do this.

@jinjingforever, do we host a debug bundle (built with assertions) that @RaphaelRoyerRivard can use?

[jinjingforever]

Hi @RaphaelRoyerRivard

@mattsoulanille is right. The core code is internal to google and we don't have a debug bundle available now. If it is ok, could you please share your model? I think it is high likely due to some unsupported ops.

Thanks!

[RaphaelRoyerRivard]

Thanks for your answers, I cannot share the model unfortunately, but I will try to extract the operators that are used. Do you have a list of supported operators?

[jinjingforever]

Hi @RaphaelRoyerRivard

I think the best way is probably to run your model using tflite benchmark tool. See more info here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark. tfjs-tflite uses some common c++ libraries behind the scene. Try setting xnnpack delegate to true.

By default, I believe the benchmark tool uses built-in ops, and it will throw errors if it sees an unsupported op. Give it a try!

[RaphaelRoyerRivard]

Hi @jinjingforever I just tested it and succeeded to run the benchmark without xnnpack, but when I try to use it, I get that vague error.

rroyerrivard@rroyerrivard-MBP tensorflow % bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model --graph=/Users/rroyerrivard/Downloads/test_model.tflite --use_xnnpack=true
STARTING!
Log parameter values verbosely: [0]
Graph: [/Users/rroyerrivard/Downloads/test_model.tflite]
Use xnnpack: [1]
Loaded model /Users/rroyerrivard/Downloads/test_model.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
XNNPACK delegate created.
Failed to apply XNNPACK delegate.
Benchmarking failed.

Using verbose=true doesn't give any hint on what is going wrong and I didn't find any solution by googling the error.
The closest thing I found was someone that failed to use the XNNPACK delegate when running TFLite on Android because his model's Tensors have dynamic shapes.
https://stackoverflow.com/questions/66789210/converting-tf-keras-model-to-tflite-model-is-slow-and-doesnt-work-with-xnn-pac
I don't think that applies to my model since its required input size is static.

Do you have any idea of what I could do to know why using the XNNPACK delegate fails?
Thanks

Edit: BTW, the list of ops used by the model, according to the benchmark tool when using --enable_op_profiling=true (without XNNPACK) is the following:

CONV_2D
DEPTHWISE_CONV_2D
RESIZE_BILINEAR
PRELU
ADD	
CONCATENATION
MUL
TRANSPOSE
SPLIT_V
LOGISTIC	
MEAN
RESHAPE
AVERAGE_POOL_2D
REDUCE_MAX
SUB
REDUCE_PROD
FULLY_CONNECTED
PACK
SHAPE
GATHER

According to https://github.com/google/XNNPACK#operator-coverage, there are several of these operators that are not supported...

[jinjingforever]

Hi @RaphaelRoyerRivard

Sorry, I am not an expert on this topic, but I guess one way you could try is to instrument the code here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/core/subgraph.cc#L1657. When this method returns a non-ok state, the delegate will be failed to be applied.

[RaphaelRoyerRivard]

Thanks @jinjingforever. I was able to add some logs to that function and find out that our model was converted without specifying the input size as a parameter, so it flagged its input layer as dynamically sized. Specifying the input size when using the converted actually did the trick. I can now see that several operations are delegated to XNNPACK, but still half of them are ran by tflite.

	             [Node type]	  [count]	  [avg ms]	    [avg %]	    [cdf %]	  [mem KB]	[times called]
	   TfLiteXNNPackDelegate	       44	     4.620	    86.178%	    86.178%	     0.000	       44
	           CONCATENATION	       22	     0.336	     6.267%	    92.445%	     0.000	       22
	               TRANSPOSE	       10	     0.228	     4.253%	    96.698%	     0.000	       10
	                 SPLIT_V	       10	     0.153	     2.854%	    99.552%	     0.000	       10
	              REDUCE_MAX	        1	     0.024	     0.448%	   100.000%	     0.000	        1

I don't know if this has a big impact on the inference speed or not, but from my tests, even when using the simd_threaded version of XNNPACK, I do not get a performance improvement compared when running with tfjs in the browser (I compare with quantized models for both engines)... Would you have any tips on what I could try to improve further the performance? Knowing that around half of the inference time is taken by the predict() method while the other half is taken by the data() method to retrieve the values from the tensors.

[jinjingforever]

Hi @RaphaelRoyerRivard

Glad you found the problem. As you observed, the data() method is indeed a bottleneck that probably cannot be worked around with our tflite package for now.

Recently, my colleague @lina128 added a great new feature to our webgl backend where you can keep data (such as segmentation results) on GPU and use it downstream without reading it back to CPU. She saw ~20% performance improvement for segmentation models. Maybe you can give it a try. See #5953. It will be released soon in tfjs v3.13.0

Thanks!

[RaphaelRoyerRivard]

Wow, that sounds really promising! I can't wait to try it out!
Thanks a lot for your help, it was extremely appreciated :)