TFLite converter aborts (dumps core) in BroadcastAdd4DSlow

[jaltmayerpizzorno]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Pop! OS 20.04 LTS
• TensorFlow installed from (source or binary): binary
• TensorFlow version (or github SHA if from source): 2.3.0

Command used to run the converter or code if you’re using the Python API
If possible, please share a link to Colab/Jupyter/any notebook.
https://colab.research.google.com/drive/1rfdr8bu_UfTzpGCXH-tQFVHMIKDreXad?usp=sharing

When running on my Pop! OS system under gdb, I get the backtrace:

gdb python
[...]
(gdb) run tflite.py
[...]
(gdb) where
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7ddb859 in __GI_abort () at abort.c:79
#2  0x00007ffbf82ccb1f in tflite::reference_ops::BroadcastAdd4DSlow(tflite::ArithmeticParams const&, tflite::RuntimeShape const&, float const*, tflite::RuntimeShape const&, float const*, tflite::RuntimeShape const&, float*) ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#3  0x00007ffbf82d54ec in void tflite::ops::builtin::add::EvalAdd<(tflite::ops::builtin::add::KernelType)2>(TfLiteContext*, TfLiteNode*, TfLiteAddParams*, tflite::ops::builtin::add::OpData const*, TfLiteTensor const*, TfLiteTensor const*, TfLiteTensor*) ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#4  0x00007ffbf82d846d in TfLiteStatus tflite::ops::builtin::add::Eval<(tflite::ops::builtin::add::KernelType)2>(TfLiteContext*, TfLiteNode*) () from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#5  0x00007ffbf82ab1bf in tflite::optimize::calibration::(anonymous namespace)::LoggingEval(TfLiteContext*, TfLiteNode*) ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#6  0x00007ffbf850f20b in tflite::impl::Subgraph::Invoke() ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#7  0x00007ffbf85121c0 in tflite::impl::Interpreter::Invoke() ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#8  0x00007ffbf827a581 in tflite::calibration_wrapper::CalibrationWrapper::FeedTensor(_object*) ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#9  0x00007ffbf8274285 in pybind11::cpp_function::initialize<pybind11_init__pywrap_tensorflow_lite_calibration_wrapper(pybind11::module&)::{lambda(tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&)#4}, pybind11::object, tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&, pybind11::name, pybind11::is_method, pybind11::sibling>(pybind11_init__pywrap_tensorflow_lite_calibration_wrapper(pybind11::module&)::{lambda(tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&)#4}&&, pybind11::object (*)(tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#10 0x00007ffbf8271fe7 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) ()
   from /home/juan/.local/lib/python3.8/site-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#11 0x00000000005f17e5 in PyCFunction_Call ()
#12 0x00000000005f2406 in _PyObject_MakeTpCall ()
#13 0x000000000050795f in ?? ()
#14 0x000000000056c475 in _PyEval_EvalFrameDefault ()
#15 0x0000000000565972 in _PyEval_EvalCodeWithName ()
#16 0x00000000005f1d85 in _PyFunction_Vectorcall ()
#17 0x00000000005677c7 in _PyEval_EvalFrameDefault ()
#18 0x0000000000565972 in _PyEval_EvalCodeWithName ()
#19 0x00000000005f1d85 in _PyFunction_Vectorcall ()
#20 0x0000000000507729 in ?? ()
#21 0x00000000005f1107 in PyObject_Call ()
#22 0x0000000000568e1f in _PyEval_EvalFrameDefault ()
#23 0x000000000050712e in ?? ()
#24 0x000000000056c475 in _PyEval_EvalFrameDefault ()
#25 0x0000000000565972 in _PyEval_EvalCodeWithName ()
#26 0x00000000005f1d85 in _PyFunction_Vectorcall ()
--Type <RET> for more, q to quit, c to continue without paging--
#27 0x00000000005677c7 in _PyEval_EvalFrameDefault ()
#28 0x0000000000565972 in _PyEval_EvalCodeWithName ()
#29 0x0000000000686053 in PyEval_EvalCode ()
#30 0x00000000006753d1 in ?? ()
#31 0x000000000067544f in ?? ()
#32 0x0000000000675507 in PyRun_FileExFlags ()
#33 0x000000000067758a in PyRun_SimpleFileExFlags ()
#34 0x00000000006ae99e in Py_RunMain ()
#35 0x00000000006aed29 in Py_BytesMain ()
#36 0x00007ffff7ddd0b3 in __libc_start_main (main=0x4ebd20 <main>, argc=2, argv=0x7fffffffe2c8, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe2b8) at ../csu/libc-start.c:308
#37 0x00000000005f62ee in _start ()

[abattery]

@jaltmayerpizzorno Could you try the tf-nightly version instead?

[jaltmayerpizzorno]

@jaltmayerpizzorno Could you try the tf-nightly version instead?

Hi @abattery, I did earlier, with the same result, but will try again and report back.

[jaltmayerpizzorno]

@abattery It crashed on colab again with 2.4.0-dev20200916. It also does so using the tensorflow/tensorflow:nightly-gpu (also 2.4.0-dev20200916) docker image. On the latter, gdb has the backtrace at:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fe09246f8b1 in __GI_abort () at abort.c:79
#2  0x00007fdf60088eff in tflite::reference_ops::BroadcastAdd4DSlow(tflite::ArithmeticParams const&, tflite::RuntimeShape const&, float const*, tflite::RuntimeShape const&, float const*, tflite::RuntimeShape const&, float*) ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#3  0x00007fdf6009136c in void tflite::ops::builtin::add::EvalAdd<(tflite::ops::builtin::add::KernelType)2>(TfLiteContext*, TfLiteNode*, TfLiteAddParams*, tflite::ops::builtin::add::OpData const*, TfLiteTensor const*, TfLiteTensor const*, TfLiteTensor*) ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#4  0x00007fdf600946d9 in TfLiteStatus tflite::ops::builtin::add::Eval<(tflite::ops::builtin::add::KernelType)2>(TfLiteContext*, TfLiteNode*) () from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#5  0x00007fdf600676bf in tflite::optimize::calibration::(anonymous namespace)::LoggingEval(TfLiteContext*, TfLiteNode*) ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#6  0x00007fdf60301e6b in tflite::impl::Subgraph::Invoke() ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#7  0x00007fdf603049b0 in tflite::impl::Interpreter::Invoke() ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#8  0x00007fdf600365c1 in tflite::calibration_wrapper::CalibrationWrapper::FeedTensor(_object*) ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#9  0x00007fdf6002d5a5 in void pybind11::cpp_function::initialize<pybind11_init__pywrap_tensorflow_lite_calibration_wrapper(pybind11::module&)::{lambda(tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&)#4}, pybind11::object, tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&, pybind11::name, pybind11::is_method, pybind11::sibling>(pybind11_init__pywrap_tensorflow_lite_calibration_wrapper(pybind11::module&)::{lambda(tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&)#4}&&, pybind11::object (*)(tflite::calibration_wrapper::CalibrationWrapper&, pybind11::handle&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so
#10 0x00007fdf6002e279 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) ()
   from /usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/optimize/_pywrap_tensorflow_lite_calibration_wrapper.so

[jaltmayerpizzorno]

Before crashing, it also logs the following... Just passing on in case it matters.

see current operation: %174 = "tfl.concatenation"(%173, %128) {axis = 3 : i32, fused_activation_function = "NONE"} : (tensor<?x?x?x256xf32>, tensor<?x38x38x512xf32>) -> tensor<?x38x38x768xf32>
2020-09-16 23:31:50.934601: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %174 = "tfl.concatenation"(%173, %128) {axis = 3 : i32, fused_activation_function = "NONE"} : (tensor<?x?x?x256xf32>, tensor<?x38x38x512xf32>) -> tensor<?x38x38x768xf32>

see current operation: %193 = "tfl.concatenation"(%192, %77) {axis = 3 : i32, fused_activation_function = "NONE"} : (tensor<?x?x?x128xf32>, tensor<?x76x76x256xf32>) -> tensor<?x76x76x384xf32>
2020-09-16 23:31:50.934633: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %193 = "tfl.concatenation"(%192, %77) {axis = 3 : i32, fused_activation_function = "NONE"} : (tensor<?x?x?x128xf32>, tensor<?x76x76x256xf32>) -> tensor<?x76x76x384xf32>

see current operation: %216 = "tfl.concatenation"(%212, %213, %215) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x76x76x3x2xf32>, tensor<?x76x76x3x2xf32>, tensor<?x76x76x3x81xf32>) -> tensor<?x76x76x3x85xf32>
2020-09-16 23:31:50.934644: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %216 = "tfl.concatenation"(%212, %213, %215) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x76x76x3x2xf32>, tensor<?x76x76x3x2xf32>, tensor<?x76x76x3x81xf32>) -> tensor<?x76x76x3x85xf32>

see current operation: %231 = "tfl.concatenation"(%219, %222, %226, %230, %215) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x81xf32>) -> tensor<?x76x76x3x85xf32>
2020-09-16 23:31:50.934655: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %231 = "tfl.concatenation"(%219, %222, %226, %230, %215) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x1xf32>, tensor<?x76x76x3x81xf32>) -> tensor<?x76x76x3x85xf32>

see current operation: %242 = "tfl.concatenation"(%238, %239, %241) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x38x38x3x2xf32>, tensor<?x38x38x3x2xf32>, tensor<?x38x38x3x81xf32>) -> tensor<?x38x38x3x85xf32>
2020-09-16 23:31:50.934666: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %242 = "tfl.concatenation"(%238, %239, %241) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x38x38x3x2xf32>, tensor<?x38x38x3x2xf32>, tensor<?x38x38x3x81xf32>) -> tensor<?x38x38x3x85xf32>

see current operation: %257 = "tfl.concatenation"(%245, %248, %252, %256, %241) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x81xf32>) -> tensor<?x38x38x3x85xf32>
2020-09-16 23:31:50.934677: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %257 = "tfl.concatenation"(%245, %248, %252, %256, %241) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x1xf32>, tensor<?x38x38x3x81xf32>) -> tensor<?x38x38x3x85xf32>

see current operation: %268 = "tfl.concatenation"(%264, %265, %267) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x19x19x3x2xf32>, tensor<?x19x19x3x2xf32>, tensor<?x19x19x3x81xf32>) -> tensor<?x19x19x3x85xf32>
2020-09-16 23:31:50.934705: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %268 = "tfl.concatenation"(%264, %265, %267) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x19x19x3x2xf32>, tensor<?x19x19x3x2xf32>, tensor<?x19x19x3x81xf32>) -> tensor<?x19x19x3x85xf32>

see current operation: %283 = "tfl.concatenation"(%271, %274, %278, %282, %267) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x81xf32>) -> tensor<?x19x19x3x85xf32>
2020-09-16 23:31:50.934717: W tensorflow/compiler/mlir/lite/tf_to_tfl_flatbuffer.cc:138] see current operation: %283 = "tfl.concatenation"(%271, %274, %278, %282, %267) {axis = 4 : i32, fused_activation_function = "NONE"} : (tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x1xf32>, tensor<?x19x19x3x81xf32>) -> tensor<?x19x19x3x85xf32>

[jaltmayerpizzorno]

I just confirmed that the issue is still there with TF 2.3.1.