runpath not including //tensorflow/python for _dtensor_device.so

[elfringham]

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): manylinux2014
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: n/a
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): git HEAD
• Python version: 3.7.x
• Bazel version (if compiling from source): 5.0.0
• GCC/Compiler version (if compiling from source): 10.2.1
• CUDA/cuDNN version: n/a
• GPU model and memory: n/a

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior

auditwheel fails with message about cannot find _pywrap_tensorflow_internal.so

Describe the expected behavior

auditwheel passes

Contributing

• Do you want to contribute a PR? (yes/no):
• Briefly describe your candidate solution(if contributing):

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.

auditwheel repair -w ../wheels/ tensorflow-pkg/tensorflow_aarch64-2.9.0-cp38-cp38-linux_aarch64.wh

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

Issue introduced with fd94c26

[elfringham]

@cfRod @nSircombe

[elfringham]

$ auditwheel repair -w ../wheels/ tensorflow-pkg/tensorflow_aarch64-2.9.0-cp38-cp38-linux_aarch64.whl
INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.9.0-cp38-cp38-linux_aarch64.whl
Traceback (most recent call last):
File "/usr/local/bin/auditwheel", line 8, in
sys.exit(main())
File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.9/site-packages/auditwheel/main.py", line 59, in main
rval = args.func(args, p)
File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.9/site-packages/auditwheel/main_repair.py", line 161, in execute
out_wheel = repair_wheel(
File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.9/site-packages/auditwheel/repair.py", line 74, in repair_wheel
raise ValueError(
ValueError: Cannot repair wheel, because required library "_pywrap_tensorflow_internal.so" could not be located

[elfringham]

Running auditwheel with --verbose gives a lot more detail, in particular this section about _dtensor_device shows that tensorflow/python is missing from runpath and so gets a null path for _pywrap_tensorflow_internal.so

"tensorflow/dtensor/python/_dtensor_device.so": {
    "interp": null,
    "path": "tensorflow/dtensor/python/_dtensor_device.so",
    "realpath": "tensorflow/dtensor/python/_dtensor_device.so",
    "needed": [
        "libtensorflow_framework.so.2",
        "_pywrap_tensorflow_internal.so",
        "libstdc++.so.6",
        "libm.so.6",
        "libgcc_s.so.1",
        "libc.so.6"
    ],
    "rpath": [],
    "runpath": [
        "/tmp/tmpp3a0nu2y/tensorflow/dtensor/python",
        "/tmp/tmpp3a0nu2y/tensorflow/dtensor",
        "/tmp/tmpp3a0nu2y/tensorflow"
    ],
    "libs": {
        "libtensorflow_framework.so.2": {
            "realpath": "/tmp/tmpp3a0nu2y/tensorflow/libtensorflow_framework.so.2",
            "path": "/tmp/tmpp3a0nu2y/tensorflow/libtensorflow_framework.so.2",
            "needed": [
                "libdl.so.2",
                "libm.so.6",
                "librt.so.1",
                "libpthread.so.0",
                "libstdc++.so.6",
                "libgcc_s.so.1",
                "libc.so.6",
                "ld-linux-aarch64.so.1"
            ]
        },
        "libdl.so.2": {
            "realpath": "/lib64/libdl-2.17.so",
            "path": "/lib64/libdl.so.2",
            "needed": [
                "libc.so.6",
                "ld-linux-aarch64.so.1"
            ]
        },
        "libc.so.6": {
            "realpath": "/lib64/libc-2.17.so",
            "path": "/lib64/libc.so.6",
            "needed": [
                "ld-linux-aarch64.so.1"
            ]
        },
        "ld-linux-aarch64.so.1": {
            "realpath": "/lib64/ld-2.17.so",
            "path": "/lib/ld-linux-aarch64.so.1",
            "needed": []
        },
        "libm.so.6": {
            "realpath": "/lib64/libm-2.17.so",
            "path": "/lib64/libm.so.6",
            "needed": [
                "libc.so.6"
            ]
        },
        "librt.so.1": {
            "realpath": "/lib64/librt-2.17.so",
            "path": "/lib64/librt.so.1",
            "needed": [
                "libpthread.so.0",
                "libc.so.6"
            ]
        },
        "libpthread.so.0": {
            "realpath": "/lib64/libpthread-2.17.so",
            "path": "/lib64/libpthread.so.0",
            "needed": [
                "libc.so.6",
                "ld-linux-aarch64.so.1"
            ]
        },
        "libstdc++.so.6": {
            "realpath": "/lib64/libstdc++.so.6.0.19",
            "path": "/lib64/libstdc++.so.6",
            "needed": [
                "libm.so.6",
                "libc.so.6",
                "ld-linux-aarch64.so.1",
                "libgcc_s.so.1"
            ]
        },
        "libgcc_s.so.1": {
            "realpath": "/lib64/libgcc_s-4.8.5-20150702.so.1",
            "path": "/lib64/libgcc_s.so.1",
            "needed": [
                "libc.so.6"
            ]
        },
        "_pywrap_tensorflow_internal.so": {
            "realpath": null,
            "path": null,
            "needed": []
        }
    }
},

[elfringham]

Compare the above with the entry for _pywrap_mlir.so which gets a path for _pywrap_tensorflow_internal.so

"tensorflow/python/_pywrap_mlir.so": {
    "interp": null,
    "path": "tensorflow/python/_pywrap_mlir.so",
    "realpath": "tensorflow/python/_pywrap_mlir.so",
    "needed": [
        "libtensorflow_framework.so.2",
        "_pywrap_tensorflow_internal.so",
        "libm.so.6",
        "libdl.so.2",
        "libpthread.so.0",
        "libstdc++.so.6",
        "libgcc_s.so.1",
        "libc.so.6",
        "ld-linux-aarch64.so.1"
    ],
    "rpath": [],
    "runpath": [
        "/tmp/tmpp3a0nu2y/tensorflow/python",
        "/tmp/tmpp3a0nu2y/tensorflow"
    ],
    "libs": {
        "libtensorflow_framework.so.2": {
            "realpath": "/tmp/tmpp3a0nu2y/tensorflow/libtensorflow_framework.so.2",
            "path": "/tmp/tmpp3a0nu2y/tensorflow/libtensorflow_framework.so.2",
            "needed": [
                "libdl.so.2",
                "libm.so.6",
                "librt.so.1",
                "libpthread.so.0",
                "libstdc++.so.6",
                "libgcc_s.so.1",
                "libc.so.6",
                "ld-linux-aarch64.so.1"
            ]
        },
        "libdl.so.2": {
            "realpath": "/lib64/libdl-2.17.so",
            "path": "/lib64/libdl.so.2",
            "needed": [
                "libc.so.6",
                "ld-linux-aarch64.so.1"
            ]
        },
        "libc.so.6": {
            "realpath": "/lib64/libc-2.17.so",
            "path": "/lib64/libc.so.6",
            "needed": [
                "ld-linux-aarch64.so.1"
            ]
        },
        "ld-linux-aarch64.so.1": {
            "realpath": "/lib64/ld-2.17.so",
            "path": "/lib/ld-linux-aarch64.so.1",
            "needed": []
        },
        "libm.so.6": {
            "realpath": "/lib64/libm-2.17.so",
            "path": "/lib64/libm.so.6",
            "needed": [
                "libc.so.6"
            ]
        },
        "librt.so.1": {
            "realpath": "/lib64/librt-2.17.so",
            "path": "/lib64/librt.so.1",
            "needed": [
                "libpthread.so.0",
                "libc.so.6"
            ]
        },
        "libpthread.so.0": {
            "realpath": "/lib64/libpthread-2.17.so",
            "path": "/lib64/libpthread.so.0",
            "needed": [
                "libc.so.6",
                "ld-linux-aarch64.so.1"
            ]
        },
        "libstdc++.so.6": {
            "realpath": "/lib64/libstdc++.so.6.0.19",
            "path": "/lib64/libstdc++.so.6",
            "needed": [
                "libm.so.6",
                "libc.so.6",
                "ld-linux-aarch64.so.1",
                "libgcc_s.so.1"
            ]
        },
        "libgcc_s.so.1": {
            "realpath": "/lib64/libgcc_s-4.8.5-20150702.so.1",
            "path": "/lib64/libgcc_s.so.1",
            "needed": [
                "libc.so.6"
            ]
        },
        "_pywrap_tensorflow_internal.so": {
            "realpath": "/tmp/tmpp3a0nu2y/tensorflow/python/_pywrap_tensorflow_internal.so",
            "path": "/tmp/tmpp3a0nu2y/tensorflow/python/_pywrap_tensorflow_internal.so",
            "needed": [
                "libtensorflow_framework.so.2",
                "libpthread.so.0",
                "libm.so.6",
                "libdl.so.2",
                "librt.so.1",
                "libstdc++.so.6",
                "libgcc_s.so.1",
                "libc.so.6",
                "ld-linux-aarch64.so.1"
            ]
        }
    }
},

[elfringham]

It looks like the runpath is composed of the location of the target shared object and all its parents to the tensorflow root. However _dtensor_device.so has a dependency on _pywrap_tensorflow_internal.so which is located in tensorflow/python. But _dtensor_device.so is located in tensorflow/dtensor/python so its runpath does not include tensorflow/python as it is not a parent and hence the failure to locate _pywrap_tensorflow_internal.so.

[elfringham]

This seems to have been resolved by 9575427

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No