`tflite-runtime-nightly` on ARM64 has minimum GLIBC requirement 2.33, which prevents installation in official `python` container images

[johnthagen]

System information

• Have I written custom code (as opposed to using a stock example script
  provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): python:3.10-slim-bullseye container image on ARM64
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue
  happens on a mobile device: N/A
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): tflite-runtime-nightly 2.13.0.dev20230207
• Python version: 3.10.10
• Bazel version (if compiling from source): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: N/A
• GPU model and memory: N/A
• Exact command to reproduce:

Describe the problem

1. Create a new python:3.10-slim-bullseye container
2. Install tflite-runtime-nightly 2.13.0.dev20230207
3. Build and run the container on ARM64 (such as from an ARM Macbook using Docker Desktop)
4. Run

>>> import tflite_runtime.interpreter as tflite
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/venv/lib/python3.10/site-packages/tflite_runtime/interpreter.py", line 33, in <module>
    from tflite_runtime import _pywrap_tensorflow_interpreter_wrapper as _interpreter_wrapper
ImportError: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /opt/venv/lib/python3.10/site-packages/tflite_runtime/_pywrap_tensorflow_interpreter_wrapper.so)

There are not yet any Debian bookworm-based python official images yet, so it's not possible to bump up to a newer Debian version with a newer GLIBC.

Other background

This worked fine in a python:3.10-slim-bullseye x86 container, so perhaps the GLIBC version requirements are different due to how the ARM64 wheels were built?

Here are the GLIBC versions of various Linux platforms:

Debian Buster:   2.28
       Bullseye: 2.31
       Bookworm: 2.36

Ubuntu 20.04:    2.31
       22.04:    2.35

RHEL   8:        2.28
       9:        2.34

Related to:

• tflite-runtime wheels for Python 3.10 #59423

CC @samypr100 @terryheo

[samypr100]

Interesting that we'd get 2.33 when Ubuntu 20.04 build images should be 2.31

[samypr100]

ldd -v output from the arm64 build:

> ldd -v _pywrap_tensorflow_interpreter_wrapper.so
./_pywrap_tensorflow_interpreter_wrapper.so: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000005501c4a000)
libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000005501cf5000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000005501ecd000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000005501ef1000)
/lib/ld-linux-aarch64.so.1 (0x0000005500000000)

This does not happen in the x86_64 build:

> ldd -v _pywrap_tensorflow_interpreter_wrapper.so
linux-vdso.so.1 (0x00007ffc9cb4a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f31e875c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f31e8618000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f31e8612000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f31e8445000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f31e842b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f31e8254000)
/lib64/ld-linux-x86-64.so.2 (0x00007f31e8c6b000)

[samypr100]

objdump -T output of failing symbols on arm64

...
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.33  fstat
...
0000000000000000      DO *UND*	0000000000000000  GLIBC_2.32  __libc_single_threaded
...
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  dlopen
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  pthread_create
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  pthread_join
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  pthread_once
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  dlsym
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  dlerror
...
0000000000000000      DF *UND*	0000000000000000  GLIBCXX_3.4.29 _ZSt28__throw_bad_array_new_lengthv
...
0000000000000000      DF *UND*	0000000000000000  CXXABI_1.3.13 _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
0000000000000000      DF *UND*	0000000000000000  CXXABI_1.3.13 _ZNSt15__exception_ptr13exception_ptr10_M_releaseEv
...

On x86_64:

1. dlopen, pthread_create, pthread_join, pthread_once, dlsym, dlerror are using GLIBC_2.2.5
2. _ZSt28__throw_bad_array_new_lengthv, __libc_single_threaded, fstat were not on the table
3. All _exception_ptr are using CXXABI_1.3.3

[samypr100]

Similar issue on armv7 builds:

ldd -v _pywrap_tensorflow_interpreter_wrapper.so
./_pywrap_tensorflow_interpreter_wrapper.so: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /usr/lib/arm-linux-gnueabihf/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.33' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.32' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
./_pywrap_tensorflow_interpreter_wrapper.so: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.34' not found (required by ./_pywrap_tensorflow_interpreter_wrapper.so)
libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x3f4a9000)
libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x3f36e000)
libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x3f345000)
libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x3f246000)
/lib/ld-linux-armhf.so.3 (0x40000000)

Exact same symbols on arm64 build are the ones failing on armv7

[terryheo]

Could you confirm if this only happens in tflite-runtime-nightly? Does tflite-runtime still work?

[samypr100]

Doesn't seem to happen on upstream tflite-runtime.

> ldd -v _pywrap_tensorflow_interpreter_wrapper.so
libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000005501db1000)
libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000005501dc7000)
libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000005501df6000)
libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000005501f81000)
libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x000000550203e000)
libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000005502062000)
/lib/ld-linux-aarch64.so.1 (0x0000005500000000)

I just checked python 3.9 version of tflite-runtime-nightly and it happens on arm64/armv7 builds as well.

It started happening on 2.12.0.dev20230202. #59423 was merged after this nighly build, so I assume the issue comes from a tad bit before? Unless I'm misunderstanding how nightlies are stamped/built.

Judging from the time frame, it was after 2.12.0.dev20230131 and before 2.12.0.dev20230202. So maybe something in these commits: 62b4e83 - 875df16 that switched the build to bazel?

[terryheo]

Right, recently ARM toolchain was updated. #57585
I think we'd better not to use it for Python wheel build. Let me handle it.

[samypr100]

@terryheo Ah I see. I thought we'd be using the toolchains from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/cmake/download_toolchains.sh, but I guess that's only for the cmake build?

download_toolchains.sh could be updated to 9.2.1 if we want to bump it a little to keep glibc versions happy (e.g.):

1. From gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf (8.3.0) to gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf (9.2.1)
2. From gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu to gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu

Although not sure if 9.2.1 would resolve the issues in #57364, I wonder if 10.3-2021.07 would still resolve the issue yet still be compatible. Ubuntu 20.04 was default was gcc-9, Ubuntu 22.04 default is gcc-11, so I'd assume we'd want to stick with the 9.x tools since we're building with 20.04 now. Stricly debian, bullseye defaults to gcc-10, where as buster defaults to gcc-8.

Full file below for posterity in case that also needs to happen...

toolchains.diff

 #!/bin/bash
 # Copyright 2021 The TensorFlow Authors. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
 #     http://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
 
 set -e
 
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 cd "$SCRIPT_DIR/../../../.."
 
 TOOLCHAINS_DIR=$(realpath tensorflow/lite/tools/cmake/toolchains)
 mkdir -p ${TOOLCHAINS_DIR}
 
 case $1 in
 	armhf)
-    if [[ ! -d "${TOOLCHAINS_DIR}/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf" ]]; then
-      curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz >&2
-      tar xvf gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz -C ${TOOLCHAINS_DIR} >&2
+    if [[ ! -d "${TOOLCHAINS_DIR}/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf" ]]; then
+      curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/9.2-2019.12/binrel/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf.tar.xz >&2
+      tar xvf gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf.tar.xz -C ${TOOLCHAINS_DIR} >&2
     fi
-    ARMCC_ROOT=${TOOLCHAINS_DIR}/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf
+    ARMCC_ROOT=${TOOLCHAINS_DIR}/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf
     echo "ARMCC_FLAGS=\"-march=armv7-a -mfpu=neon-vfpv4 -funsafe-math-optimizations \
-      -isystem ${ARMCC_ROOT}/lib/gcc/arm-linux-gnueabihf/8.3.0/include \
-      -isystem ${ARMCC_ROOT}/lib/gcc/arm-linux-gnueabihf/8.3.0/include-fixed \
-      -isystem ${ARMCC_ROOT}/arm-linux-gnueabihf/include/c++/8.3.0 \
-      -isystem ${ARMCC_ROOT}/arm-linux-gnueabihf/libc/usr/include \
+      -isystem ${ARMCC_ROOT}/lib/gcc/arm-none-linux-gnueabihf/9.2.1/include \
+      -isystem ${ARMCC_ROOT}/lib/gcc/arm-none-linux-gnueabihf/9.2.1/include-fixed \
+      -isystem ${ARMCC_ROOT}/arm-none-linux-gnueabihf/include/c++/9.2.1 \
+      -isystem ${ARMCC_ROOT}/arm-none-linux-gnueabihf/libc/usr/include \
       -isystem \"\${CROSSTOOL_PYTHON_INCLUDE_PATH}\" \
       -isystem /usr/include\""
-    echo "ARMCC_PREFIX=${ARMCC_ROOT}/bin/arm-linux-gnueabihf-"
+    echo "ARMCC_PREFIX=${ARMCC_ROOT}/bin/arm-none-linux-gnueabihf-"
 		;;
 	aarch64)
-    if [[ ! -d "${TOOLCHAINS_DIR}/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu" ]]; then
-      curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz >&2
-      tar xvf gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz -C ${TOOLCHAINS_DIR} >&2
+    if [[ ! -d "${TOOLCHAINS_DIR}/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu" ]]; then
+        curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/9.2-2019.12/binrel/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu.tar.xz >&2
+      tar xvf gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu.tar.xz -C ${TOOLCHAINS_DIR} >&2
     fi
-    ARMCC_ROOT=${TOOLCHAINS_DIR}/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu
+    ARMCC_ROOT=${TOOLCHAINS_DIR}/gcc-arm-9.2-2019.12-x86_64-aarch64-none-linux-gnu
     echo "ARMCC_FLAGS=\"-funsafe-math-optimizations \
-      -isystem ${ARMCC_ROOT}/lib/gcc/aarch64-linux-gnu/8.3.0/include \
-      -isystem ${ARMCC_ROOT}/lib/gcc/aaarch64-linux-gnu/8.3.0/include-fixed \
-      -isystem ${ARMCC_ROOT}/aarch64-linux-gnu/include/c++/8.3.0 \
-      -isystem ${ARMCC_ROOT}/aarch64-linux-gnu/libc/usr/include \
+      -isystem ${ARMCC_ROOT}/lib/gcc/aarch64-none-linux-gnu/9.2.1/include \
+      -isystem ${ARMCC_ROOT}/lib/gcc/aarch64-none-linux-gnu/9.2.1/include-fixed \
+      -isystem ${ARMCC_ROOT}/aarch64-none-linux-gnu/include/c++/9.2.1 \
+      -isystem ${ARMCC_ROOT}/aarch64-none-linux-gnu/libc/usr/include \
       -isystem \"\${CROSSTOOL_PYTHON_INCLUDE_PATH}\" \
       -isystem /usr/include\""
-    echo "ARMCC_PREFIX=${ARMCC_ROOT}/bin/aarch64-linux-gnu-"
+    echo "ARMCC_PREFIX=${ARMCC_ROOT}/bin/aarch64-none-linux-gnu-"
 		;;
 	rpi0)
-    if [[ ! -d "${TOOLCHAINS_DIR}/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf" ]]; then
-      curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz >&2
-      tar xvf gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz -C ${TOOLCHAINS_DIR} >&2
+    if [[ ! -d "${TOOLCHAINS_DIR}/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf" ]]; then
+      curl -LO https://storage.googleapis.com/mirror.tensorflow.org/developer.arm.com/media/Files/downloads/gnu-a/9.2-2019.12/binrel/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf.tar.xz >&2
+      tar xvf gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf.tar.xz -C ${TOOLCHAINS_DIR} >&2
     fi
-    ARMCC_ROOT=${TOOLCHAINS_DIR}/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf
+    ARMCC_ROOT=${TOOLCHAINS_DIR}/gcc-arm-9.2-2019.12-x86_64-arm-none-linux-gnueabihf
     echo "ARMCC_FLAGS=\"-march=armv6 -mfpu=vfp -mfloat-abi=hard -funsafe-math-optimizations \
-      -isystem ${ARMCC_ROOT}/lib/gcc/arm-linux-gnueabihf/8.3.0/include \
-      -isystem ${ARMCC_ROOT}/lib/gcc/arm-linux-gnueabihf/8.3.0/include-fixed \
-      -isystem ${ARMCC_ROOT}/arm-linux-gnueabihf/include/c++/8.3.0 \
-      -isystem ${ARMCC_ROOT}/arm-linux-gnueabihf/libc/usr/include \
+      -isystem ${ARMCC_ROOT}/lib/gcc/arm-none-linux-gnueabihf/9.2.1/include \
+      -isystem ${ARMCC_ROOT}/lib/gcc/arm-none-linux-gnueabihf/9.2.1/include-fixed \
+      -isystem ${ARMCC_ROOT}/arm-none-linux-gnueabihf/include/c++/9.2.1 \
+      -isystem ${ARMCC_ROOT}/arm-none-linux-gnueabihf/libc/usr/include \
       -isystem \"\${CROSSTOOL_PYTHON_INCLUDE_PATH}\" \
       -isystem /usr/include\""
-    echo "ARMCC_PREFIX=${ARMCC_ROOT}/bin/arm-linux-gnueabihf-"
+    echo "ARMCC_PREFIX=${ARMCC_ROOT}/bin/arm-none-linux-gnueabihf-"
     ;;
 	*)
 		echo "Usage: download_toolchains.sh [armhf|aarch64|rpi0]" >&2
     exit
 		;;
   esac
 
 echo "download_toolchains.sh completed successfully." >&2

[terryheo]

I've updated internal build script to use CMake to build Python wheels.
So the nightly should be fine now. Please let me know if the issue still exists.

Regarding the toolchain.sh update, I'd better keep it as it is since we need more compatibility. For those who need more up-to-date toolchain, they can use their own toolchain without relying on the script.