opal_wrapper-wrapper-data.txt not found

[thangckt]

Issue type

Build/Install

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

tf 2.13

Custom code

No

OS platform and distribution

Centos 6.9

Mobile device

No response

Python version

3.11

Bazel version

5.2.0

GCC/compiler version

GCC 9.5.0 + OpenMPI 4.1.7

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

compiling error as below:

ERROR: /home1/p001cao/app/miniconda3/envs/py11deepmd/share/bazel/c22f5c495debb6ef458b8cca313d2d01/external/com_google_protobuf/BUILD.bazel:75:11: Compiling src/google/protobuf/compiler/importer.cc failed: (Exit 243): opal_wrapper failed: error executing command
  (cd /home1/p001cao/app/miniconda3/envs/py11deepmd/share/bazel/c22f5c495debb6ef458b8cca313d2d01/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/home1/p001cao/app/miniconda3/envs/py11deepmd/lib:/home1/p001cao/app/mpi/openmpi4.1.7-clang17-IB/lib:/home2/app/compiler/gcc/9.5.0/lib64:/home1/p001cao/app/mpi/openmpi4.1.7-clang17-IB/lib:/home1/p001cao/app/compiler/llvm-17/libexec:/home1/p001cao/app/compiler/llvm-17/lib:/home2/app/compiler/gcc/9.5.0/libexec/gcc/x86_64-pc-linux-gnu/9.5.0:/home2/app/compiler/gcc/9.5.0/lib64:/home2/app/compiler/gcc/9.5.0/lib/gcc/x86_64-pc-linux-gnu/9.5.0:/home1/p001cao/app/tools/binutils-2.41/lib:/home1/p001cao/app/miniconda3/lib \
    PATH=/home2/Modules//bin:/home1/p001cao/app/mpi/openmpi4.1.7-clang17-IB/bin:/home1/p001cao/app/mpi/openmpi4.1.7-clang17-IB/bin:/home1/p001cao/app/compiler/llvm-17/bin:/home2/app/compiler/gcc/9.5.0/bin:/home1/p001cao/app/tools/binutils-2.41/bin:/home1/p001cao/app/miniconda3/envs/py11deepmd/bin:/home1/p001cao/app/miniconda3/condabin:/home1/p001cao/app/miniconda3/bin:/home1/app/sge/bin/linux-x64:/usr/lib64/qt-3.3/bin:/home2/Modules/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ibutils/bin:/home1/p001cao/bin \
    PWD=/proc/self/cwd \
  /home1/p001cao/app/mpi/openmpi4.1.7-clang17-IB/bin/opal_wrapper -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf/importer.d '-frandom-seed=bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf/importer.o' -iquote external/com_google_protobuf -iquote bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-opt-exec-50AE0418/bin/external/zlib -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/src -isystem external/zlib -isystem bazel-out/k8-opt-exec-50AE0418/bin/external/zlib -g0 -w -g0 '-std=c++17' -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_google_protobuf/src/google/protobuf/compiler/importer.cc -o bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf/importer.o)
# Configuration: 25b75394c5b3297b8c64984caa144939808df936b3058e3be498ec8a5b4cebea
# Execution platform: @local_execution_config_platform//:platform
Cannot open configuration file /home1/p001cao/app/mpi/openmpi4.1.7-clang17-IB/share/openmpi/opal_wrapper-wrapper-data.txt
Error parsing data file opal_wrapper: Not found

Standalone code to reproduce the issue

export TF_PYTHON_VERSION=3.11
export PYTHON_VERSION=python3.11
PYTHON_BIN_PATH=$(which $PYTHON_VERSION) \
PYTHON_LIB_PATH=$($PYTHON_VERSION -c "import pip; print(pip.__path__[0].rstrip('/pip'))") \
TF_NEED_MPI=1 \
TF_ENABLE_XLA=1 \
TF_NEED_CUDA=0 \
TF_NEED_ROCM=0 \
TF_CUDA_CLANG=0 \
TF_DOWNLOAD_CLANG=0 \
TF_NEED_TENSORRT=0 \
TF_SET_ANDROID_WORKSPACE=0 \
TF_NEED_OPENCL=0 \
TF_NEED_OPENCL_SYCL=0 \
TF_NEED_AWS=0 \
TF_NEED_GCP=0 \
TF_NEED_S3=0 \
TF_NEED_NCCL=0 \
CC_OPT_FLAGS="-march=native -Wno-sign-compare" \
./configure


```sh
bazel build --jobs=16 -c opt --config=noaws --config=nogcp --config=nonccl --verbose_failures //tensorflow/tools/pip_package:build_pip_package

### Relevant log output

_No response_

[tilakrayal]

@thangckt,
Could you please let us know the specific reason for using tensorflow v2.13 which is pretty old. Could you please try installing tensorflow latest version 2.15 or 2.16.1 and let me know if you are facing the same issue and Could you please confirm the sequence of steps. Tf2.13v supports protobuf >=3.20 versions as per source and make sure you are trying the same. Thank you!

[thangckt]

I have an old Linux, and the maximum bazel version that can run is 5.2.0 (from conda). With this limitation, i can just try up to tf 2.13.