Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Diagnostics tools

A collection of tools to help triage and diagnose misbehaving networks.

Notation used:

  • TensorFlow: TF
  • TensorBoard: TB
  • protobuf binary: pb
  • protobuf string text: pbtxt
  • NGraph-TF: NGTF

What to do if your network fails

TODO: List steps to generate logs or run diagnostic tools

Capturing logs in text file

NGTF uses the std error to output its logs, so it is necessary to pipe it correctly to capture all logs. python > log.txt 2>&1

A full dump


Debug flags

  • NGRAPH_ENABLE_SERIALIZE=1: Generate nGraph level serialized graphs .json
  • NGRAPH_CPU_TRACING=1: Generate nGraph level function timelines
  • NGRAPH_TF_VLOG_LEVEL=5: Generate ngraph-tf logging info for different passes
  • NGRAPH_TF_LOG_PLACEMENT=1: Generates op placement log at stdout
  • NGRAPH_TF_DUMP_CLUSTERS=1: Dumps Encapsulated TF Graphs: ngraph_cluster_<cluster_num>
  • NGRAPH_TF_DUMP_GRAPHS=1: Dumps TF graphs for different passes : precapture, capture, unmarked, marked, clustered, declustered, encapsulated
  • TF_CPP_MIN_VLOG_LEVEL=1: Enables TF CPP Logs
  • NGRAPH_TF_DUMP_DECLUSTERED_GRAPHS=1: Dumps graphs with final clusters assigned. Use this to view TF computation graph with colored nodes indicating clusters

Visualizing encapsulates using TB

  • Run your script with this flag: NGRAPH_TF_DUMP_DECLUSTERED_GRAPHS=1 python
  • Change directory to this diagnostics folder
  • Run this script to parse the dumped graphs to know which encapsulate a node belongs to. At this step nodemap.pkl is created: python ./path/to/folder/where/ nodemap.pkl
  • Modify the graphdef and dump TB file in ./vis using encapsulate information in nodemap.pkl: python -c nodemap.pkl ./path/to/original_network_pbtxtfile.pbtxt ./vis. If you do not have the pbtxt of the original tensorflow graph, you can dump it from your script using write_graph
  • View the original network with encapsulate information by running tensorboard, using the files created in ./vis.
  • Note: you may need to preprocess your protobuffers (pbtxt) to remove the _class attribute if any of the above steps result in an error such as {NodeType} expects to be colocated with unknown node .... In that case, first run python -d /path/to/pbtxt/to/process/ and then follow the above steps again.

Disable NGRAPH in python

  • In your script, import ngraph_bridge by using: import ngraph_bridge
  • Disable ngraph by calling: ngraph_bridge.disable()
  • Enable ngraph by calling: ngraph_bridge.enable()
  • Checking whether ngraph is enabled by calling: ngraph.is_enabled()
  • You need to enable ngraph every time you called ngraph_bridge.disable(), so it is good to check if ngraph is enabled by calling ngraph.is_enabled()
  • Caution: The above functions are only effective at the beginning of the execution. Once the session is created and run is called, the above functions will not be able to disable ngraph.
  • For example usage, take a look at the model_test/ in the diagnostics folder

Protobuf Visualization

The python script can convert a protobuf (pb or pbtxt) into a dot file or a TB log, which can be viewed using TB. If the input is a pbtxt then ngtf_graph_viewer can also sanitize node names to remove underscores from the front of node names (which indicate they are internal nodes and might cause TB to complain). It can also prepend strings in front of certain node names, a feature which can be used to append encapsulate information for clustering nodes together

ngtf_graph_viewer has been tested on Python 2 TF-1.9, but should work with Python 3 and other versions of TF.

Run the following for detailed help:

python -h

Some commandline samples/usecases

  • pbtxt to TB: python pbtxtfile.pbtxt ./vis
  • pbtxt to dot: python -v 0 pbtxtfile.pbtxt ./vis
  • pb to TB: python -b pbtxtfile.pb ./vis
  • pb to dot: python -b -v 0 pbtxtfile.pb ./vis
  • pbtxt to TB after prepending cluster information. See Visualizing encapsulates using TB: python -c nodemap.pkl pbtxtfile.pbtxt ./vis

Some other usecases

  • graphdef to dot: from ngtf_graph_viewer import graphdef_to_dot
  • graphdef to TB: from ngtf_graph_viewer import graphdef_to_tensorboard
  • pbtxt to TB: from ngtf_graph_viewer import protobuf_to_grouped_tensorboardt; protobuf_to_grouped_tensorboard(input_filename, dot_dir, input_binary=False)
  • pbtxt to dot: from ngtf_graph_viewer import protobuf_to_dot; protobuf_to_dot(input_filename, dot_dir, input_binary=False)
  • pb to TB: from ngtf_graph_viewer import protobuf_to_grouped_tensorboardt; protobuf_to_grouped_tensorboard(input_filename, dot_dir, input_binary=True)
  • pb to dot: from ngtf_graph_viewer import protobuf_to_dot; protobuf_to_dot(input_filename, dot_dir, input_binary=True)
  • pb to graphdef: from ngtf_graph_viewer import load_file; load_file(input_filename, input_binary=True)
  • pbtxt to graphdef: from ngtf_graph_viewer import load_file; load_file(input_filename, input_binary=False)
  • modify a graphdef's nodes names: from ngtf_graph_viewer import modify_node_names; modify_node_names(graph_def, node_map={"net1/node1":"e1/net1/node1"})
You can’t perform that action at this time.