Histogram Tab Not Showing up with tf.keras Callback

[jmd-0]

Environment information (required)

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py version 30596d3b33f3ff90772dbd75ab5abc3702545a0c

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=6, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='JD-MacBook-Pro.local', release='18.7.0', version='Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: True
INFO: $VIRTUAL_ENV: None

--- check: installed_packages
INFO: installed: tensorboard==1.14.0
INFO: installed: tensorflow==1.14.0
INFO: installed: tensorflow-estimator==1.14.0

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.14.0'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '1.14.0'
INFO: tensorflow.__git_version__: 'unknown'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'~/fluoro/fenv/bin/tensorboard\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): 'JD-MacBook-Pro.local'

--- check: stat_tensorboardinfo
INFO: directory: /var/folders/6p/0r05_kf55273nh_nftm5q9tw0000gn/T/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=8624961842, st_dev=16777222, st_nlink=2, st_uid=501, st_gid=20, st_size=64, st_atime=1569529483, st_mtime=1569531094, st_ctime=1569531094)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['~/fluoro/fenv/lib/python3.6/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.7.1
appnope==0.1.0
apptools==4.4.0
astetik==1.9.8
astor==0.8.0
attrs==19.1.0
autograd==1.3
backcall==0.1.0
bleach==3.1.0
certifi==2019.6.16
chances==0.1.6
chardet==3.0.4
colorama==0.4.1
configobj==5.0.6
cycler==0.10.0
decorator==4.4.0
defusedxml==0.6.0
entrypoints==0.3
envisage==4.7.2
future==0.17.1
gast==0.2.2
geonamescache==1.0.2
gitdb2==2.0.5
GitPython==3.0.2
graphviz==0.11.1
grpcio==1.14.1
h5py==2.9.0
hyperas==0.4.1
hyperopt==0.1.2
idna==2.8
imageio==2.5.0
ipykernel==5.1.1
ipython==7.7.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.15.1
Jinja2==2.10.1
joblib==0.13.2
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
jupyter-tensorboard==0.1.10
jupyterthemes==0.20.0
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kerasplotlib==0.1.4
kiwisolver==1.1.0
lesscpy==0.13.0
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.1
mayavi==4.7.1
mistune==0.8.4
mkl-fft==1.0.14
mkl-random==1.0.2
mkl-service==2.0.2
nbconvert==5.5.0
nbdime==1.0.5
nbformat==4.4.0
networkx==2.3
notebook==6.0.1
numpy==1.16.4
opencv-python==4.1.0.25
pandas==0.24.2
pandocfilters==1.4.2
parso==0.5.1
patsy==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
pip==19.2.2
ply==3.11
prometheus-client==0.7.1
prompt-toolkit==2.0.9
protobuf==3.8.0
psutil==5.6.0
ptyprocess==0.6.0
pydot==1.4.1
pyface==6.1.2
pyglet==1.4.1
Pygments==2.4.2
pymongo==3.9.0
pyparsing==2.4.2
PyQt5==5.13.0
PyQt5-sip==4.19.18
pyrsistent==0.15.4
python-dateutil==2.8.0
pytz==2019.2
PyWavelets==1.0.3
PyYAML==5.1.2
pyzmq==18.1.0
qtconsole==4.5.4
requests==2.22.0
Rtree==0.8.3
scikit-image==0.15.0
scikit-learn==0.21.2
scipy==1.3.0
seaborn==0.9.0
Send2Trash==1.5.0
setuptools==41.0.1
Shapely==1.6.4.post2
six==1.12.0
sklearn==0.0
smmap2==2.0.5
statsmodels==0.10.1
talos==0.6.3
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
tornado==6.0.3
tqdm==4.34.0
traitlets==4.3.2
traits==5.1.2
traitsui==6.1.2
trimesh==2.38.42
urllib3==1.25.3
vtk==8.1.2
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.5
wheel==0.33.4
widgetsnbextension==3.5.1
wrangle==0.6.7
wrapt==1.11.2
xxhash==1.3.0

Issue description

I have been trying to use the basic TensorFlow-Keras callback to get more information about my model. Earlier yesterday, when I first attempted to visualize the Histogram of the model, there was no issue. However, today for whatever reason, as I was updating the model, the histogram tab disappeared from the web browser, when using TensorBoard. After tinkering all day, I have been unable to retrieve the Histogram tab of the web browser.

Here is what TensorBoard shows:

In my search, I stumbled upon the following terminal output:

E0926 16:45:51.255817 123145384722432 directory_watcher.py:242] File ~/fluoro/code/jupyt/vox_fluoro/test_1/tf_logs/run_20190926_164541/events.out.tfevents.1569530742.JD-MacBook-Pro.local updated even though the current file is ~/fluoro/code/jupyt/vox_fluoro/test_1/tf_logs/run_20190926_164541/events.out.tfevents.1569530749.JD-MacBook-Pro.local.profile-empty

Also for completeness, the directory structure for the log files is below:

test_1/
    run_20190926_164541/
        events.out.tfevents.1569530742.JD-MacBook-Pro.local
        events.out.tfevents.1569530749.JD-MacBook-Pro.local.profile-empty
        plugins/
            profile/
                2019-09-26_16-45-49/
                    local.trace

Lastly, just in case it might be useful, here is the code:

import numpy as np
import tensorflow as tf
import os
import datetime



save_dir = os.path.abspath(os.getcwd())


root_logdir = os.path.join(save_dir, 'tf_logs')

run_id = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
log_dir = os.path.join(root_logdir, 'run_' + run_id)

os.makedirs(log_dir, exist_ok=True)



num_of_samples = 6

cali_mat_train = np.random.rand(num_of_samples, 6)
label_mat_train = np.random.rand(num_of_samples, 6)


cali_input_shape = (6,)


input_cali = tf.keras.Input(shape=cali_input_shape, name='input_cali', dtype='float32')

dense_0_cali = tf.keras.layers.Dense(units=10, activation='elu', kernel_initializer='he_uniform', activity_regularizer=None)(input_cali)
bn_0 = tf.keras.layers.BatchNormalization()(dense_0_cali)

dense_1_cali = tf.keras.layers.Dense(units=10, activation='elu', kernel_initializer='he_uniform', activity_regularizer=None)(bn_0)
bn_1 = tf.keras.layers.BatchNormalization()(dense_1_cali)

main_output = tf.keras.layers.Dense(units=6, activation=None, kernel_initializer='he_uniform', activity_regularizer=None, name='main_output')(bn_1)


model = tf.keras.Model(inputs=[input_cali], outputs=main_output)


model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.01), loss='mse', metrics=['mse'])

model.summary()


tensorboard_cb = tf.keras.callbacks.TensorBoard(log_dir, histogram_freq=1, batch_size=10, write_grads=True)
terminate_if_nan = tf.keras.callbacks.TerminateOnNaN()


result = model.fit(x={'input_cali': cali_mat_train}, y=label_mat_train, epochs=20, callbacks=[tensorboard_cb, terminate_if_nan], batch_size=10, shuffle=True, verbose=2)

Thank you for your help!

[nfelt]

Duplicate of #2084

I think the .profile-empty file is the culprit; this is the same issue as #2084. As a workaround, you can try passing --reload_multifile=true to TensorBoard, or you can disable profiling in the Keras callback by passing profile_batch=0 as described in #2084 (comment)

[jmd-0]

Hi @nfelt, thanks for your response!

I just want to be explicit to ensure that I'm addressing the issue. I just tried to run the profile_batch=0 workaround by adding it to the tf.keras callback. Is this what is being suggested (as below)?

tensorboard_cb = tf.keras.callbacks.TensorBoard(log_dir, histogram_freq=1, batch_size=10, write_grads=True, profile_batch=0)

I tried this ^ and was unsuccessful (after clearing my logs directory).

Also, I tried to run:

$ tensorboard --logdir=./tf_logs --reload_multifile=true

and received the following error:

tensorboard: error: unrecognized arguments: --reload_multifile=true

Any advice? Thanks again!

[nfelt]

Yep, that's the suggestion. Can you show the contents of your new log directory when running with that argument passed?

As for --reload_multifile=true I forgot to mention that it's only available in our latest release (TensorBoard 2.0). If you can try pip install tensorboard==2.0.0 the flag should be available, and I believe that version of TB should still be compatible with your TensorFlow 1.14 installation.

[jmd-0]

Yes, definitely.

The directory structure is as follows:

test_1 /
     tf_logs /
          run_20190926_211404 / 
               events.out.tfevents.1569546846.JD-MacBook-Pro.local

I am still getting same result (see below), will try upgrading TensorBoard next.

[jmd-0]

Hey @nfelt ,

Just upgraded to TensorBoard 2.0.0 via pip install tensorboard==2.0.0.

I am still getting the same result with the missing histogram tab (after passing --reload_multifile=true to TensorBoard call). I generated two runs, both with and without profile_batch=0 in the function call for the tf.keras callback, and the problem persists.

For the profile_batch=0 run, directory structure is:

test_1 / 
     tf_logs /
          run_20190926_212632 / 
               events.out.tfevents.1569547593.JD-MacBook-Pro.local

Without the profile_batch argument, directory structure is:

test_1 / 
     tf_logs /
          events.out.tfevents.1569547412.JD-MacBook-Pro.local
          events.out.tfevents.1569547418.JD-MacBook-Pro.local.profile-empty
          plugins / 
               profile /
                    2019-09-26_21-23-38 /
                         local.trace
               

Please let me know if there is anything else I can try! Thank you for the help.

[daviesthomas]

@jmd-0 did you resolve this issue?

[jmd-0]

Have not tried anything since this, so no @daviesthomas