Chrome timeline support on tensorflow

[stalagmite7]

I've been looking at some profiling tools available, the chrome timeline from tensorflow looks super useful this stackoverflow post, but its not documented and the only examples I can find to piece together how to run it are tensorflow models.
I am running keras with a tf backend, and my sequential models are all built in keras. For instance, this is how my model is being built:

   model.fit_generator( \
	    generator= data_gen(args, 1), \
 	    steps_per_epoch=tr_steps, \
 	    epochs=args.epochs, \
 	    validation_data=data_gen(args, 2), \
 	    validation_steps=val_steps, \
 	    verbose=1, \
 	    callbacks=[checkpointer])

I am at a loss for how I would try to generate a timeline trace for this model, and wanted to know if Keras supports this? And how would I generate a trace for this?

• Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[fchollet]

I have never used it. But you could try something like this:

from keras import backend as K

# Run the graph with full trace option
with K.get_session()  as sess:
    run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
    run_metadata = tf.RunMetadata()
    
    model.fit_generator(...)

    # Create the Timeline object, and write it to a json
    tl = timeline.Timeline(run_metadata.step_stats)
    ctf = tl.generate_chrome_trace_format()
    with open('timeline.json', 'w') as f:
        f.write(ctf)

[stalagmite7]

Thankd for the quick response! I tried that out but my timeline.json is not recording anything, this is all there is in the file:

{
	"traceEvents": [
		{
			"ph": "M",
			"args": {
			    "name": "Allocators"
			},
			"pid": 0,
			"name": "process_name"
		}
	]
}

So my time line trace on chrome is actually empty.

[al626]

The issue is that run_options and run_metadata need to be passed into tf.Session.run. I've got some code which allows you to do this in Keras here

[mas-dse-greina]

Is this feature in the latest release? I'm getting the same results as stalagmite7. timeline.json is not recording anything. Can anyone provide an example program? Thanks.

[ghost]

I'm also stuck on this, can you please provide an example to clarify? Thanks

[al626]

Sorry for the delay, something like the below should work:

import keras
from keras.layers.core import Dense
from keras.models import Sequential
import tensorflow as tf
from tensorflow.python.client import timeline

x = np.random.randn(10000, 2)
y = (x[:, 0] * x[:, 1]) > 0 # xor
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
model = Sequential()
model.add(Dense(units=64, activation='relu', input_dim=2))
model.add(Dense(units=2, activation='softmax'))
model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              options=run_options,
              run_metadata=run_metadata)
model.fit(x, keras.utils.to_categorical(y), epochs=1)
trace = timeline.Timeline(step_stats=run_metadata.step_stats)
with open('/tmp/timeline.ctf.json', 'w') as f:
    f.write(trace.generate_chrome_trace_format())

Then go to chrome://tracing in either Google Chrome or Chromium and open /tmp/timeline.ctf.json.

As far as I'm aware, the canonical documentation for how to use this is in this comment.

[shensheng27]

model.fit_generator didn't save anything for trace. Is this keras timeline only working on model.fit?

[cjmielke]

@shensheng27 thank you for this suggestion! fit_generator indeed doesnt seem to save anything, but fit() does. Nice catch