Error loading '_text_similarity_metric_ops.so' when running unit tests

[GeorgeDittmar]

running python 3.7 on mac osx 10.14.6. Not sure if there is some dependency or build step I am missing but I cannot seem to run the unit tests with out the code failing to load this file. Have tried with tensorflow 1.x and 2.x. stack trace is below. Maybe I am just missing something simple?

Traceback (most recent call last):
File "/Applications/PyCharm CE.app/Contents/helpers/pycharm/_jb_unittest_runner.py", line 35, in
main(argv=args, module=None, testRunner=unittestpy.TeamcityTestRunner, buffer=not JB_DISABLE_BUFFERING)
File "/miniconda3/envs/tf2/lib/python3.7/unittest/main.py", line 100, in init
self.parseArgs(argv)
File "/miniconda3/envs/tf2/lib/python3.7/unittest/main.py", line 147, in parseArgs
self.createTests()
File "/miniconda3/envs/tf2/lib/python3.7/unittest/main.py", line 159, in createTests
self.module)
File "/miniconda3/envs/tf2/lib/python3.7/unittest/loader.py", line 220, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/miniconda3/envs/tf2/lib/python3.7/unittest/loader.py", line 220, in
suites = [self.loadTestsFromName(name, module) for name in names]
File "/miniconda3/envs/tf2/lib/python3.7/unittest/loader.py", line 154, in loadTestsFromName
module = import(module_name)
File "/Users/dittmar/Development/text/tensorflow_text/python/ops/bert_tokenizer_test.py", line 32, in
from tensorflow_text.python.ops import bert_tokenizer
File "/Users/dittmar/Development/text/tensorflow_text/init.py", line 21, in
from tensorflow_text.python import metrics
File "/Users/dittmar/Development/text/tensorflow_text/python/metrics/init.py", line 20, in
from tensorflow_text.python.metrics.text_similarity_metric_ops import *
File "/Users/dittmar/Development/text/tensorflow_text/python/metrics/text_similarity_metric_ops.py", line 28, in
gen_text_similarity_metric_ops = load_library.load_op_library(resource_loader.get_path_to_datafile('_text_similarity_metric_ops.so'))
File "/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: dlopen(/Users/dittmar/Development/text/tensorflow_text/python/metrics/_text_similarity_metric_ops.so, 6): image not found

[broken]

Yes; this is an open issue we are looking at. TF updated their build environment to Ubuntu 16, and we have only recently got our builds migrated over. Unfortunately, we haven't gotten the tests moved over yet, but this is a high priority for us.

[GeorgeDittmar]

Are there any possible work arounds to be able to utilize the tests? Was thinking of trying to work on an open issue / feature request around the sliding window but having trouble starting since I want to use the current unit tests to help me workout the logic.

[broken]

I think the tests are in a better state now in the master branch, and should be working.

[GeorgeDittmar]

Hmm I just tried to run the tests locally but still get the same issue. I must have something weird in my setup I need to figure out.

[broken]

I wanted to double-check this worked before closing, and discovered the problem y'all have been experiencing. The custom op unit tests are building the shared libraries to test against. However, most likely the environment is not the same, which creates problems in linking. Our automated tests don't have this same issue since they are being built in the correct environment.

The other option (apart from matching build environments) would be to rebuild TensorFlow, then run the unit tests against that, which is a poor solution. I don't have a good answer to this at the moment, so we'll need to give this more thought on how we wish to handle it.

Note that this is only a problem with running the tests on Linux. MacOS has less build environment varieties for the most part.

[GeorgeDittmar]

@hanneshapke No I kept running into the issue locally on my mac. I sort of gave up for a while. Been meaning to try to look into it again.

[broken]

The solution here is to provide a docker image to run the tests in. This would be similar to what is described in https://github.com/tensorflow/custom-op

Since the automated tests are working though, this is just lower priority at the moment compared to releasing a Keras version of the BERT model, releasing more text models, expanding ragged tensor support, etc.

[GeorgeDittmar]

ah thats a good idea actually.

[Workshopshed]

The error seems to be reoccurring in TF Nightly but is not present in 2.2.0

[evyasonov]

I have the same issue. I've installed tf-nightly, but now I can not import tensorflow_text because of it:

---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-2-cefb365966a1> in <module>()
      1 import tensorflow          as tf
----> 2 import tensorflow_text     as tft
      3 import sys

3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/load_library.py in load_op_library(library_filename)
     56     RuntimeError: when unable to load the library or get the python wrappers.
     57   """
---> 58   lib_handle = py_tf.TF_LoadLibrary(library_filename)
     59   try:
     60     wrappers = _pywrap_python_op_gen.GetPythonWrappers(

NotFoundError: /usr/local/lib/python3.6/dist-packages/tensorflow_text/python/metrics/_text_similarity_metric_ops.so: undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb

See this colab:
https://colab.research.google.com/drive/1X0MhpbPbguscxwY-UDyp9d0OsL5hANR2?usp=sharing

Can you please help me

[hanneshapke]

Hi @evyasonov ,

I just looked at your notebook. The version of tf-text needs to line up with the tf version. Since you use the nightly version, the tf version is "ahead" compared to tf-text.

I tried the import statement in your notebook with the latest stable TF release, and the import of tf-text work.

Python:  3.6.9 (default, Jul 17 2020, 12:50:27) 
[GCC 8.4.0]
TF Version:  2.3.0
Eager mode:  True
GPU is NOT AVAILABLE

Do you need to nightly version of tf?

[evyasonov]

@hanneshapke
The point that I need nightly version of tf because of this issue tensorflow/tensorflow#42673 (comment)

[broken]

Unfortunately, we do not have nightly releases for TF Text at this time. If you actually must use tf-nightly and the latest TF Text release is incompatible, you will need to build the pip package from source (https://github.com/tensorflow/text#build-from-source-steps).

There are a couple things to keep in mind if you build TF Text from source. First, make sure to update the WORKSPACE file with the nightly version of TF that you will be running (and have that version installed). Also, note that the package you build is only guaranteed to be compatible for the system you build it on. If you want to distribute it to other machines, you need to build it in the docker image provided by TF.

[evyasonov]

@broken
Thanks for the advice
I'm trying to update WORKSPACE file and I'm a bit confused.

Is it correct that I should update this part with tf-nightly version:

http_archive(
    name = "org_tensorflow",
    strip_prefix = "tensorflow-2.3.0",
    sha256 = "1a6f24d9e3b1cf5cc55ecfe076d3a61516701bc045925915b26a9d39f4084c34",
    urls = [
        "https://github.com/tensorflow/tensorflow/archive/v2.3.0.zip"
    ],
)

?

If so, what prefix and what url should I use? I have 2.4.0-dev20200908 installed and I tried to use this url " https://github.com/tensorflow/tensorflow/archive/v2.4.0-dev20200908.zip ", bit it failed.

Can you help please

[broken]

That's the right place to update. Unfortunately, TF does not add tags to its nightly builds, so you cannot download the code directly like that. What you need to do is determine the SHA of the commit from the nightly you have. Once you have the nightly version installed, you can find the SHA with tf.__git_version__ - it's the last 10 chars.
Example:

python -c 'import tensorflow as tf; print(tf.__git_version__)' | tail -1 | grep -o '.\{10\}$'

This is only the first 10 characters of the full SHA though. You can probably get the full commit using git checkout or git log, or by navigating to it in your browser (https://github.com/tensorflow/tensorflow/commit/${short_sha_here}).

Once you have the full SHA, you can then use it in the URL of the WORKSPACE file.
https://github.com/tensorflow/tensorflow/archive/${commit_sha}.zip

You can delete the sha256 verification, or use sha256sum on the downloaded zip to determine and update it.

The prefix is the name of the directory when you unzip that file. In this case, it should be: tensorflow-${commit_sha}

[evyasonov]

@broken
Thank you for the answer!

I've finally builded tensorflow-nightly from source:

$ python -c 'import tensorflow as tf; print(tf.__version__); print(tf.__git_version__);'
2.4.0
v1.12.1-43222-g68a6fe0d98

I've cloned tf-text and updated WORKSPACE

http_archive(
    name = "org_tensorflow",
    strip_prefix = "tensorflow-68a6fe0d984377b625b421d34f8c607d6ed73597",
    sha256 = "5f1015e7332de70a8827c9a7412aee747f0230d731fe3f9b98b9b9c674e85065",
    urls = [
        "https://github.com/tensorflow/tensorflow/archive/68a6fe0d984377b625b421d34f8c607d6ed73597.zip"
    ],
)

And I run ./oss_scripts/run_build.sh to build tf-text and it succeed.
I did it for both master and 2.4b branches of tf-text.

BUT when I install tf-text, it downloads tf 2.3.1. And it's not what I expect

$ pip install tensorflow_text-2.4.0b0-cp36-cp36m-linux_x86_64.whl 
Processing ./tensorflow_text-2.4.0b0-cp36-cp36m-linux_x86_64.whl
Collecting tensorflow-hub>=0.8.0
  Using cached tensorflow_hub-0.9.0-py2.py3-none-any.whl (103 kB)
Collecting tensorflow<2.4,>=2.3.0
  Downloading tensorflow-2.3.1-cp36-cp36m-manylinux2010_x86_64.whl (320.4 MB)
...

I need tf-2.4.0 on purpose, but installation of the text removes it. What should I do?

[broken]

Oh, this is because the setup.py of the python package has that version range defined.
https://github.com/tensorflow/text/blob/master/oss_scripts/pip_package/setup.py#L76

That value is just telling pip what to do, so you can either update setuip.py and rebuild so pip accepts your tf version, or reinstall the tensorflow version you desire after installing tensorflow_text.

[evyasonov]

I've just removed 'tensorflow>=2.3.0, <2.4', from setup.py and I've succeed to install tf-text. And finally I've succeed to import both tf-nightly and tf-text!

@broken , thank you very much for you help! Now I can continue with another issue ( tensorflow/tensorflow#42673 (comment) )

[Rmsharks4]

Hi! I'm getting this error at:

import tensorflow_text as text

tensorflow_text==2.4.0rc0 (the only version that exists now)
tensorflow==2.4.0rc2 (because of condition of tf_text: tensorflow<2.5,>=2.4.0rc0)

How can I fix this error - I need to import the library in order to load the Albert Preprocessor Module for Tensorflow-Hub:

preprocessor = hub.KerasLayer("http://tfhub.dev/tensorflow/albert_en_preprocess/1")

Not calling this library above raises a different error.
Help is appreciated - Thanks!

[johnsantamariajr]

Echoing Rmsharks4 , can't go backwards here on Windows, there's only one available version of tensorflow_text.

tensorflow-text==2.4.0rc0
tensorflow==2.4.0rc2
tensorflow-hub==0.1.0````

[broken]

You are probably on Py3.8. With this being the first Windows release, we had a problem with that build. This should be fixed in the next release candidate coming soon. Py3.7 and 3.6 should work though.

[puraminy]

As I found other places, installing tensorflow 2.3 and tensorflow_text 2.3 can resolve this problem. I did the same, but problem was probably due pip and conda conflicts. When I uninstalled them and install both with pip on base environment problem was resolved.

I think since tensorflow_text depends on tensorflow, it's good that it's deployed on conda too.

[lifeiteng]

maybe you are in the repo text/, try python3 -c"import tensorflow_text as tftxt" outside

[milmor]

I have this issue. Tensorflow 2.8 and python 3.8. All packages installed using pip.

NotFoundError: /home/user/anaconda3/envs/tf24/lib/python3.8/site-packages
/tensorflow_text/python/metrics/_text_similarity_metric_ops.so: 
undefined symbol: _ZN10tensorflow15OpKernelContext15allocate_outputEN4absl14lts_2020_09_23
11string_viewERKNS_11TensorShapeEPPNS_6TensorE

[Balive13]

FYI: The following versions are compatible to each other:
tensorflow-text==2.3
tensorflow==2.3
keras==2.3
tensorflow-hub==0.12

[broken]

We finally have instructions on how to use docker images to run our tests on. These images are the same provided by TF. You can find the instructions at the bottom of the README on the home page of this project.