TF2.0: Tensorboard The connection was reset

[daddydrac]

I have the same problem but w/ TF2.0-GPU, and I am on Chrome/Ubuntu, running it inside a docker container w/ juypter, miniconda, cuda-x, tensorrt. The error I get says: The connection was reset when trying to display it in the notebook, despite everything else working fine. (I'm not sure, but even when I goto localhost:6006 it doesn't load either).

Error 1:

Error 2:

Dockerfile:

FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04

# Core Linux Deps
RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y --fix-missing --no-install-recommends apt-utils \
        build-essential \
        curl \
	binutils \
	gdb \
        git \
	freeglut3 \
	freeglut3-dev \
	libxi-dev \
	libxmu-dev \
	gfortran \
        pkg-config \
	python-numpy \
	python-dev \
	python-setuptools \
	libboost-python-dev \
	libboost-thread-dev \
        pbzip2 \
        rsync \
        software-properties-common \
        libblas3 \
        liblapack3 \
        liblapack-dev \
        libblas-dev \
        libboost-all-dev \
        libopenblas-dev \ 
        libtbb2 \
        libtbb-dev \
        libjpeg-dev \
        libpng-dev \
        libtiff-dev \
	libgraphicsmagick1-dev \
        libavresample-dev \
        libavformat-dev \
        libhdf5-dev \
        libpq-dev \
	libgraphicsmagick1-dev \
	libavcodec-dev \
	libgtk2.0-dev \
	liblapack-dev \
        liblapacke-dev \
	libswscale-dev \
	libcanberra-gtk-module \
        libboost-dev \
	libboost-all-dev \
        libeigen3-dev \
	wget \
        vim \
        qt5-default \
        unzip \
	zip \ 
        && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*  && \
    apt-get clean && rm -rf /tmp/* /var/tmp/*


# Install cmake version that supports anaconda python path
RUN wget -O cmake.tar.gz https://github.com/Kitware/CMake/releases/download/v3.15.4/cmake-3.15.4-Linux-x86_64.tar.gz
RUN tar -xvf cmake.tar.gz
WORKDIR /cmake-3.15.4-Linux-x86_64
RUN cp -r bin /usr/
RUN cp -r share /usr/
RUN cp -r doc /usr/share/
RUN cp -r man /usr/share/
WORKDIR /
RUN rm -rf cmake-3.15.4-Linux-x86_64
RUN rm -rf cmake.tar.gz


# Install TensorRT (TPU Access)
RUN apt-get update && \
        apt-get install -y nvinfer-runtime-trt-repo-ubuntu1804-5.0.2-ga-cuda10.0 && \
        apt-get update && \
        apt-get install -y libnvinfer5=5.0.2-1+cuda10.0

RUN file="$(ls -1 /usr/local/)" && echo $file


# Fix conda errors per Anaconda team until they can fix
RUN mkdir ~/.conda


# Install Anaconda
RUN wget --quiet https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
/bin/bash Miniconda3-latest-Linux-x86_64.sh -f -b -p /opt/conda && \
rm Miniconda3-latest-Linux-x86_64.sh
ENV PATH /opt/conda/bin:$PATH


# For CUDA profiling, TensorFlow requires CUPTI.
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

ARG PYTHON=python3
ARG PIP=pip3

# See http://bugs.python.org/issue19846
ENV LANG C.UTF-8

RUN apt-get update && apt-get install -y \
    ${PYTHON} \
    ${PYTHON}-pip

RUN ${PIP} --no-cache-dir install --upgrade \
    pip \
    setuptools \
    hdf5storage \
    h5py \
    matplotlib \
    pyinstrument

# Add auto-complete to Juypter
RUN pip install jupyter-tabnine
RUN pip install pydash

RUN conda update -n base -c defaults conda
RUN conda install -c anaconda jupyter 
RUN conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
RUN conda update conda
RUN conda install numba
RUN conda install -c anaconda cupy 
RUN conda install -c anaconda ipykernel 
RUN conda install -c anaconda seaborn 
RUN conda install -c anaconda ipython 


# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python 

RUN pip install tf-nightly-gpu-2.0-preview
#RUN pip install tensorflow-gpu

#COPY bashrc /etc/bash.bashrc
#RUN chmod a+rwx /etc/bash.bashrc

RUN ${PIP} --no-cache-dir install jupyter  


WORKDIR /
RUN wget -O opencv.zip https://github.com/opencv/opencv/archive/master.zip
RUN wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/master.zip
RUN unzip opencv.zip
RUN unzip opencv_contrib.zip
RUN mv opencv-master opencv
RUN mv opencv_contrib-master opencv_contrib
RUN mkdir /opencv/build
WORKDIR /opencv/build

RUN cmake -DBUILD_TIFF=ON \
		  -DBUILD_opencv_java=OFF \
		  -DWITH_CUDA=ON \
		  -DENABLE_FAST_MATH=1 \
		  -DCUDA_FAST_MATH=1 \
		  -DWITH_CUBLAS=1 \
		  -DENABLE_AVX=ON \
		  -DWITH_OPENGL=ON \
		  -DWITH_OPENCL=OFF \
		  -DWITH_IPP=ON \
		  -DWITH_TBB=ON \
		  -DWITH_EIGEN=ON \
		  -DWITH_V4L=ON \
		#   -DBUILD_TESTS=OFF \
		#   -DBUILD_PERF_TESTS=OFF \
		  -DCMAKE_BUILD_TYPE=RELEASE \
		  -DCMAKE_INSTALL_PREFIX=$(python -c "import sys; print(sys.prefix)") \
		  -D PYTHON3_EXECUTABLE=$(which python3) \
                  -D PYTHON_INCLUDE_DIR=$(python3 -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())") \
                  -D PYTHON_INCLUDE_DIR2=$(python3 -c "from os.path import dirname; from distutils.sysconfig import get_config_h_filename; print(dirname(get_config_h_filename()))") \
                  -D PYTHON_LIBRARY=$(python3 -c "from distutils.sysconfig import get_config_var;from os.path import dirname,join ; print(join(dirname(get_config_var('LIBPC')),get_config_var('LDLIBRARY')))") \
                  -D PYTHON3_PACKAGES_PATH=$(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())") \
                  -DOPENCV_ENABLE_NONFREE=ON \
                  -DOPENCV_EXTRA_MODULES_PATH=/opencv_contrib/modules \
                  -DBUILD_EXAMPLES=ON \
                  -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.0 \
                  -DWITH_QT=ON ..
                 
RUN make -j4 \
        && make install \
	&& rm /opencv.zip \
        && rm /opencv_contrib.zip \
	&& rm -rf /opencv \
        && rm -rf /opencv_contrib


WORKDIR /


# dlib
RUN cd ~ && \
    mkdir -p dlib && \
    git clone -b 'v19.16' --single-branch https://github.com/davisking/dlib.git dlib/ && \
    cd  dlib/ && \
    python3 setup.py install --yes USE_AVX_INSTRUCTIONS --yes DLIB_USE_CUDA --clean


RUN mkdir /.local && chmod a+rwx /.local

WORKDIR /tf
EXPOSE 8888 6006


RUN useradd -ms /bin/bash container_user


RUN ${PYTHON} -m ipykernel.kernelspec


CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root --NotebookApp.custom_display_url='http://localhost:8888'"]

``` #

[rmothukuru]

@joehoeller,
Can you please confirm if you have used tf.summary.trace_export in your code so that Tensorboard uses the Log Files created using that command. If you have used tf.summary.trace_export, can you please share your code so that we can reproduce it and investigate about it. Thanks!

[daddydrac]

@rmothukuru, here is the notebook:

import tensorflow as tf
import datetime, os

fashion_mnist = tf.keras.datasets.fashion_mnist

(x_train, y_train),(x_test, y_test) = fashion_mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

def create_model():
  return tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation='softmax')
  ])

def train_model():
  
  model = create_model()
  model.compile(optimizer='adam',
                loss='sparse_categorical_crossentropy',
                metrics=['accuracy'])

  logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
  tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)

  model.fit(x=x_train, 
            y=y_train, 
            epochs=5, 
            validation_data=(x_test, y_test), 
            callbacks=[tensorboard_callback])

train_model()

# Start TensorBoard within the notebook 
%load_ext tensorboard
%tensorboard --logdir logs

train_model()

from tensorboard import notebook
notebook.list() # View open TensorBoard instance

[rmothukuru]

@joehoeller,
I have created a New Virtual Environment in Anaconda and installed Numpy, Pandas, Matplotlib and Tensorflow-Gpu. After that, I have executed your code and it worked fine. Please find the below screenshot.

.

Can you please try the same. If it is working with a New Virtual Environment, can you please share your Dockerfile so that we can reproduce it. Thanks!

[daddydrac]

I did share the Dockerfile, scroll up. It installs Miniconda btw.

…

On Thu, Oct 10, 2019 at 11:11 PM rmothukuru ***@***.***> wrote: @joehoeller <https://github.com/joehoeller>, I have created a New Virtual Environment in Anaconda and installed Numpy, Pandas, Matplotlib and Tensorflow-Gpu. After that, I have executed your code and it worked fine. Please find the below screenshot. [image: image] <https://user-images.githubusercontent.com/48206667/66622419-23bb9b80-ec05-11e9-83a5-c5e894d2d5b0.png> . Can you please try the same. If it is working with a New Virtual Environment, can you please share your Dockerfile so that we can reproduce it. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33161?email_source=notifications&email_token=ABHVQHFXXJ74P7MW62TSP5TQN74FHA5CNFSM4I62KHQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA6VWFY#issuecomment-540891927>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHVQHDBWIUQZFRT3JDOO73QN74FHANCNFSM4I62KHQA> .

[rmothukuru]

@joehoeller,
Sorry about that. Got the Dockerfile.

[daddydrac]

Nah, your fine. Let me know what I need to change in Docker plz.

…

On Fri, Oct 11, 2019 at 12:31 AM rmothukuru ***@***.***> wrote: @joehoeller <https://github.com/joehoeller>, Sorry about that. Got the Dockerfile. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33161?email_source=notifications&email_token=ABHVQHBTXDTWIMW55QIHLKTQOAFUJA5CNFSM4I62KHQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA62QBI#issuecomment-540911621>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHVQHCEPXJF27Y3NT7WBLTQOAFUJANCNFSM4I62KHQA> .