tensorboard not able to read large event file (~600MB)

[chihuahua]

This issue had been migrated from tensorflow/tensorflow#836.

I am training a network on 4 GPUs. The event file is too large to be parsed by tensorboard. How can I increase the limit?

RNING:tensorflow:IOError [Errno 2] No such file or directory: '/home/pc/anaconda2/lib/python2.7/site-packages/tensorflow/tensorboard/TAG' on path /home/pc/anaconda2/lib/python2.7/site-packages/tensorflow/tensorboard/TAG
WARNING:tensorflow:Unable to read TensorBoard tag
Starting TensorBoard on port 6006
(You can navigate to http://0.0.0.0:6006)
[libprotobuf ERROR google/protobuf/src/google/protobuf/io/coded_stream.cc:207] A protocol message was rejected because it was too big (more than 67108864 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/pc/anaconda2/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/home/pc/anaconda2/lib/python2.7/threading.py", line 1073, in run
self.function(*self.args, **self.kwargs)
File "/home/pc/anaconda2/lib/python2.7/site-packages/tensorflow/python/summary/event_accumulator.py", line 242, in Update
self.Reload()
File "/home/pc/anaconda2/lib/python2.7/site-packages/tensorflow/python/summary/event_accumulator.py", line 175, in Reload
for event in self._generator.Load():
File "/home/pc/anaconda2/lib/python2.7/site-packages/tensorflow/python/summary/impl/directory_watcher.py", line 82, in Load
for event in self._loader.Load():
File "/home/pc/anaconda2/lib/python2.7/site-packages/tensorflow/python/summary/impl/event_file_loader.py", line 53, in Load
event.ParseFromString(self._reader.record())
DecodeError: Error parsing message

[teamdandelion]

@digitalsword Could you give us a link to the 600mb event file, or post a script that will generate a repro?

[Sakib-Ar-Rahman]

@decentralion I am facing a similar problem with large event files when trying to visualize embeddings on Tensorboard. Below is a simple script that reproduces the problem. Tensorboard runs perfectly fine if I just reduce the size or change the number of features in the embeddings. This shape produces large checkpoints (over 1GB). The error that I get when running Tensorboard is

Error fetching tensors
/Users/sakibarrahman/static/tf_logs/tests/random_name_model.ckpt-1.data-00000-of-00001; Invalid argument

Script to reproduce similar checkpoints and error:

import tensorflow as tf
import numpy as np
import os
from tensorflow.contrib.tensorboard.plugins import projector


name = "random_name"
LOG_DIR = os.path.join(os.getcwd(), "static/tf_logs/tests")
if not os.path.exists(LOG_DIR):
    os.makedirs(LOG_DIR)

NAME_TO_VISUALISE_VARIABLE = name
path_for_metadata = os.path.join(LOG_DIR, name + '_metadata.tsv')
with open(path_for_metadata, 'w') as f:
    for i in range(482492):
        f.write(f"{repr(i)}\n")
print("Metadata file created")
    
arr_embeds = np.random.rand(482493, 1370)

#  Set to a tf variable
embedding_ph = tf.placeholder(tf.float32, arr_embeds.shape, name='embedding_place_holder')
embedding_var = tf.Variable(tf.zeros(arr_embeds.shape), name=NAME_TO_VISUALISE_VARIABLE)
var_init = embedding_var.assign(embedding_ph)
summary_writer = tf.summary.FileWriter(LOG_DIR)

#  Create projector
config = projector.ProjectorConfig()

#  Add embedding to the projector
embedding = config.embeddings.add()
embedding.tensor_name = embedding_var.name
embedding.metadata_path = path_for_metadata 
projector.visualize_embeddings(summary_writer, config)

print("Saving Data")
# Save the data
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
sess.run(var_init, feed_dict={embedding_ph: arr_embeds})
saver = tf.train.Saver()
saver.save(sess, os.path.join(LOG_DIR, name + "_model.ckpt"), 1)

Is there any way to load large files onto Tensorboard?

[jart]

Could this be the same issue as tensorflow/tensorflow#582?

[nfelt]

@Sakib-Ar-Rahman that looks like maybe a different issue since it actually involves a checkpoint file, not an events file. Could you repost into a new github issue (https://github.com/tensorflow/tensorboard/issues/new) and include the information requested in the issue template? In particular, could you include any stacktrace that comes with the error message if there is one?

[Sakib-Ar-Rahman]

Just created a new issue for checkpoint files at #1214

[pindinagesh]

@chihuahua

Could you please move this issue to closed status since PR is merged?

Thanks

[pindinagesh]

@chihuahua

Closing this issue. Please feel free to reopen if this still exist. Thanks