undefined symbol on c5.4xlarge

[yugoren]

I was following BERT example on a c5.4xlarge instance when I came across the following error:

ImportError: /home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/neuron/_whitelist_partition_swig.so: undefined symbol: _ZN10tensorflow10FileSystem20RecursivelyCreateDirERKSs

when I ran bert_model.py. Here's a MWE without BERT example:

(aws_neuron_tensorflow_p36) ubuntu@ip-172-31-95-164:~/aws-neuron-sdk/src/examples/tensorflow/bert_demo$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow.neuron import fuse
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/_api/v1/neuron/__init__.py", line 10, in 
    from tensorflow._api.v1.neuron import graph_util
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/_api/v1/neuron/__init__.py", line 10, in 
    from tensorflow._api.v1.neuron import graph_util
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/_api/v1/neuron/graph_util/__init__.py", line 10, in 
    from tensorflow.python.neuron.python.graph_util import inference_graph_from_session
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/neuron/python/graph_util.py", line 42, in 
    from tensorflow.python.neuron.whitelist_partition_swig import WhitelistPartition
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/neuron/whitelist_partition_swig.py", line 28, in 
    _whitelist_partition_swig = swig_import_helper()
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/neuron/whitelist_partition_swig.py", line 24, in swig_import_helper
    _mod = imp.load_module('_whitelist_partition_swig', fp, pathname, description)
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: /home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/python/neuron/_whitelist_partition_swig.so: undefined symbol: _ZN10tensorflow10FileSystem20RecursivelyCreateDirERKSs

Same missing symbol error is also present for from tensorflow import neuron as tfn. I also found out that this works without a problem on inf1.6xlarge.

[awsrjh]

Hello yugoren

let us have a look and get back to you.

[spring01]

This issue is caused by a mismatch between installed tensorflow and the needed tensorflow-neuron packages, such as with tensorflow-gpu or tensorflow==2.0.

The solution is to reinstall tensorflow-neuron with one of two methods:

Using pip:

pip install tensorflow-neuron --force

Or by using conda:

conda install tensorflow-neuron --force