TF-2+Conda - TensorBoard doesn’t launch

[diego898]

System Information:

• TensorBoard version: 1.14.0a20190301
• TensorFlow version: 2.0.0-alpha and nightly, installed with pip
• Using anaconda python distribution
• OS Platform and version: CentOS 7
• Python version: 3.6

Starting from a basic numpy/scikitlearn anaconda environment, I installed tensorflow 2 via pip. From that enviroment, launching tensorboard gives the following error:

ERROR: Failed to launch TensorBoard (exited with 1).
Contents of stderr:
Traceback (most recent call last):
  File "/home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/bin/tensorboard", line 6, in <module>
    from tensorboard.main import run_main
  File "/home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/lib/python3.6/site-packages/tensorboard/main.py", line 42, in <module>
    import tensorflow as tf  # pylint: disable=unused-import
  File "/home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/lib/python3.6/site-packages/tensorflow/__init__.py", line 27, in <module>
    from tensorflow._api.v2 import audio
  File "/home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/lib/python3.6/site-packages/tensorflow/_api/v2/audio/__init__.py", line 8, in <module>
    from tensorflow.python.ops.gen_audio_ops import decode_wav
  File "/home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/lib/python3.6/site-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/lib/python3.6/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/mesada/.conda/envs/tf-2.0.0-alpha-cpu/lib/python3.6/site-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so)

The confusing part is the last line, where even though Im inside a conda enviroment, the last line indicates trying to use the system `/lib64/libstdc++.so.6:.

I was able to solve this by doing the following (thanks to this related SO question):

(tf-2.0.0-alpha-cpu) ➜  praxis ~ export LD_PRELOAD=$CONDA_PREFIX/lib/libstdc++.so

(tf-2.0.0-alpha-cpu) ➜  praxis ~ tensorboard --logdir /tmp/mnist
TensorBoard 1.14.0a20190301 at http://praxis.emp.vumc.io:6006 (Press CTRL+C to quit)

[wchargin]

Hi @diego898! Thanks for the report. Could you please provide
step-by-step instructions to reproduce the issue? It sounds like you’re
using a combination of Conda and Pip. Does the problem occur if you
create a non-Conda virtualenv and run pip install --pre tensorflow to
set it up?

[diego898]

hey @wchargin - I can confirm that using the systems python 2.7.5, I created a new virtualenv, activated it, and used pip to install tensorflow2 and I was able to import it, and launch tensorboard without error.

I think the problem is specific to conda and tf2+tensorboard

[wchargin]

Okay, thanks! If you would like us to look into this further, please
provide repro steps to help us diagnose the problem.

[diego898]

@wchargin Here are full reproduction steps:

conda env create -f tf-test.yml
conda activate tf-test
which pip
pip install --pre tensorflow
tensorboard --logdir /tmp/mnist

this gives the same error as above:

  File "/home/mesada/.conda/envs/tf-test/bin/tensorboard", line 6, in <module>
    from tensorboard.main import run_main
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorboard/main.py", line 42, in <module>
    import tensorflow as tf  # pylint: disable=unused-import
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/__init__.py", line 27, in <module>
    from tensorflow._api.v2 import audio
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/_api/v2/audio/__init__.py", line 8, in <module>
    from tensorflow.python.ops.gen_audio_ops import decode_wav
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so)

The yml file:

name: tf-test
channels:
  - defaults
dependencies:
  - absl-py=0.7.0=py36_0
  - asn1crypto=0.24.0=py36_0
  - astor=0.7.1=py36_0
  - atomicwrites=1.3.0=py_0
  - attrs=18.2.0=py36h28b3542_0
  - backcall=0.1.0=py36_0
  - blas=1.0=mkl
  - bleach=3.1.0=py36_0
  - bokeh=1.0.4=py36_0
  - c-ares=1.15.0=h7b6447c_1
  - ca-certificates=2019.1.23=0
  - cairo=1.14.12=h8948797_3
  - certifi=2018.11.29=py36_0
  - cffi=1.12.1=py36h2e261b9_0
  - chardet=3.0.4=py36_1
  - cloudpickle=0.8.0=py36_0
  - cryptography=2.5=py36h1ba5d50_0
  - cudatoolkit=9.2=0
  - cudnn=7.3.1=cuda9.2_0
  - cupti=9.2.148=0
  - cycler=0.10.0=py36_0
  - cython=0.29.5=py36he6710b0_0
  - cytoolz=0.9.0.1=py36h14c3975_1
  - dask-core=1.1.2=py_0
  - dbus=1.13.6=h746ee38_0
  - decorator=4.3.2=py36_0
  - dill=0.2.8.2=py36_0
  - entrypoints=0.3=py36_0
  - expat=2.2.6=he6710b0_0
  - fastcache=1.0.2=py36h14c3975_2
  - fontconfig=2.13.0=h9420a91_0
  - freetype=2.9.1=h8a8886c_1
  - fribidi=1.0.5=h7b6447c_0
  - gast=0.2.2=py36_0
  - glib=2.56.2=hd408876_0
  - gmp=6.1.2=h6c8ec71_1
  - gmpy2=2.0.8=py36h10f8cd9_2
  - graphite2=1.3.13=h23475e2_0
  - graphviz=2.40.1=h21bd128_2
  - grpcio=1.16.1=py36hf8bcb03_1
  - gst-plugins-base=1.14.0=hbbd80ab_1
  - gstreamer=1.14.0=hb453b48_1
  - h5py=2.9.0=py36h7918eee_0
  - harfbuzz=1.8.8=hffaf4a1_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=h9c2bf20_1
  - idna=2.8=py36_0
  - imageio=2.5.0=py36_0
  - intel-openmp=2019.1=144
  - ipykernel=5.1.0=py36h39e3cac_0
  - ipython=7.3.0=py36h39e3cac_0
  - ipython_genutils=0.2.0=py36_0
  - ipywidgets=7.4.2=py36_0
  - jedi=0.13.2=py36_0
  - jinja2=2.10=py36_0
  - jpeg=9b=h024ee3a_2
  - jsonschema=2.6.0=py36_0
  - jupyter=1.0.0=py36_7
  - jupyter_client=5.2.4=py36_0
  - jupyter_console=6.0.0=py36_0
  - jupyter_core=4.4.0=py36_0
  - keras-applications=1.0.6=py36_0
  - keras-preprocessing=1.0.5=py36_0
  - kiwisolver=1.0.1=py36hf484d3e_0
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.36=hbc83047_0
  - libprotobuf=3.6.1=hd408876_0
  - libsodium=1.0.16=h1bed415_0
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - libtiff=4.0.10=h2733197_2
  - libuuid=1.0.3=h1bed415_2
  - libxcb=1.13=h1bed415_1
  - libxml2=2.9.9=he19cac6_0
  - llvmlite=0.27.0=py36hd408876_0
  - markdown=3.0.1=py36_0
  - markupsafe=1.1.0=py36h7b6447c_0
  - matplotlib=3.0.2=py36h5429711_0
  - mistune=0.8.4=py36h7b6447c_0
  - mkl=2019.1=144
  - mkl_fft=1.0.10=py36ha843d7b_0
  - mkl_random=1.0.2=py36hd81dba3_0
  - more-itertools=5.0.0=py36_0
  - mpc=1.1.0=h10f8cd9_1
  - mpfr=4.0.1=hdf1c602_3
  - mpmath=1.1.0=py36_0
  - nbconvert=5.3.1=py36_0
  - nbformat=4.4.0=py36_0
  - ncurses=6.1=he6710b0_1
  - networkx=2.2=py36_1
  - nose=1.3.7=py36_2
  - notebook=5.7.4=py36_0
  - numba=0.42.0=py36h962f231_0
  - numpy=1.15.4=py36h7e9f1db_0
  - numpy-base=1.15.4=py36hde5b4d6_0
  - olefile=0.46=py36_0
  - openssl=1.1.1b=h7b6447c_1
  - packaging=19.0=py36_0
  - pandas=0.24.1=py36he6710b0_0
  - pandoc=2.2.3.2=0
  - pandocfilters=1.4.2=py36_1
  - pango=1.42.4=h049681c_0
  - parso=0.3.2=py36_0
  - patsy=0.5.1=py36_0
  - pcre=8.42=h439df22_0
  - pexpect=4.6.0=py36_0
  - pickleshare=0.7.5=py36_0
  - pillow=5.4.1=py36h34e0f95_0
  - pip=19.0.3=py36_0
  - pixman=0.36.0=h7b6447c_0
  - pluggy=0.8.1=py36_0
  - prometheus_client=0.5.0=py36_0
  - prompt_toolkit=2.0.8=py_0
  - protobuf=3.6.1=py36he6710b0_0
  - ptyprocess=0.6.0=py36_0
  - py=1.7.0=py36_0
  - pycparser=2.19=py36_0
  - pydot=1.3.0=py36_1
  - pygments=2.3.1=py36_0
  - pyopenssl=19.0.0=py36_0
  - pyparsing=2.3.1=py36_0
  - pyqt=5.9.2=py36h05f1152_2
  - pysocks=1.6.8=py36_0
  - pytest=4.3.0=py36_0
  - python=3.6.8=h0371630_0
  - python-dateutil=2.7.5=py36_0
  - pytz=2018.9=py36_0
  - pywavelets=1.0.1=py36hdd07704_0
  - pyyaml=3.13=py36h14c3975_0
  - pyzmq=17.1.2=py36he6710b0_2
  - qt=5.9.7=h5867ecd_1
  - qtconsole=4.4.3=py36_0
  - readline=7.0=h7b6447c_5
  - requests=2.21.0=py36_0
  - scikit-image=0.14.1=py36he6710b0_0
  - scikit-learn=0.20.2=py36hd81dba3_0
  - scipy=1.2.1=py36h7c811a0_0
  - seaborn=0.9.0=py36_0
  - send2trash=1.5.0=py36_0
  - setuptools=40.8.0=py36_0
  - sip=4.19.8=py36hf484d3e_0
  - six=1.12.0=py36_0
  - sqlalchemy=1.2.18=py36h7b6447c_0
  - sqlite=3.26.0=h7b6447c_0
  - statsmodels=0.9.0=py36h035aef0_0
  - sympy=1.3=py36_0
  - termcolor=1.1.0=py36_1
  - terminado=0.8.1=py36_1
  - testpath=0.4.2=py36_0
  - tk=8.6.8=hbc83047_0
  - toolz=0.9.0=py36_0
  - tornado=5.1.1=py36h7b6447c_0
  - tqdm=4.31.1=py_0
  - traitlets=4.3.2=py36_0
  - urllib3=1.24.1=py36_0
  - wcwidth=0.1.7=py36_0
  - webencodings=0.5.1=py36_1
  - werkzeug=0.14.1=py36_0
  - wheel=0.33.1=py36_0
  - widgetsnbextension=3.4.2=py36_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zeromq=4.3.1=he6710b0_3
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
  - pip:
    - future==0.17.1
    - googleapis-common-protos==1.5.8
    - promise==2.2.1
    - wrapt==1.11.1

Can you please reopen the issue?

[wchargin]

Great; thanks for the repro steps. I’ve followed those steps with conda
4.5.12 on Debian (not CentOS) and am unable to reproduce the
problem—I can run from google.protobuf.pyext import _message both
before and after pip-installing TensorFlow, and after installing
TensorFlow I can run tensorboard successfully.

I think that you’ll likely get a better response on Stack Overflow,
which has a [tensorboard] tag; I suspect that at least someone
over there will have a CentOS machine and be able to help you out. :-)

[diego898]

@wchargin Respectfully, I disagree with you. I'm reporting what I believe to be bug that appears in TF2+Conda+CentOS7 according to the steps above. I think this report should be opened, and investigated by a member of the tensorboard community willing to try to reproduce this bug completely, on a CentOS7 system. After having done that, if they can't reproduce, perhaps some more discussion/investigation can be carried out, or the bug report can be closed then as not reproducible.

On StackOverflow, I will find workarounds to this that allow me to continue my work. As I said in my original post, I already found a stack overflow answer that allowed me to do just that:

export LD_PRELOAD=$CONDA_PREFIX/lib/libstdc++.so
tensorboard --logdir /tmp/mnist
TensorBoard 1.14.0a20190301 at http://praxis.emp.vumc.io:6006 (Press CTRL+C to quit)

Can you please reopen the issue?

[wchargin]

Sure, that’s fair enough—it would be great to find something that works
better for you than an LD_PRELOAD hack. :-)

It’s a little surprising to me that TensorBoard should fail to work
while TensorFlow works fine, given that the stack trace is inside an
import tensorflow. Are you able to run python -c 'import tensorflow'
from within the problematic conda env?

[diego898]

hey @wchargin - the plot thickens! I am not able to import tensorflow from the terminal as you described. I get the same error:

praxis ~ conda activate tf-test
(tf-test) ➜  praxis ~ python -c 'import tensorflow'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/__init__.py", line 27, in <module>
    from tensorflow._api.v2 import audio
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/_api/v2/audio/__init__.py", line 8, in <module>
    from tensorflow.python.ops.gen_audio_ops import decode_wav
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 52, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/tensorflow/core/framework/graph_pb2.py", line 6, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/mesada/.conda/envs/tf-test/lib/python3.6/site-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so)

The same thing happens if I launch ipython in the terminal and try to import it.

However - If I activate this enviroment, and launch a jupyter notebook I can import tensorflow from within the notebook!