Add Document about the Graph Optimization in TensorFlow #511

nuka137 · 2019-04-22T22:02:22Z

This pull request adds the document about the graph optimization performing internally in TensorFlow.
This document shows what optimizations are performed in TensorFlow, and how to enable/disable these optimizations from Python API.
I think most of TensorFlow beginners does not interests, but this document helps the advanced user who wants to control the graph optimizations manually.

P.S. I could not figure out where should I add this document, so I added to site/en/guide/graphs.md for temporary.

lamberta · 2019-04-24T00:25:22Z

Thanks!
Could potentially go in /site/en/guide/performance, but will wait until after review to see where it should land.

MarkDaoust · 2019-04-25T21:55:30Z

site/en/guide/graphs.md

+Below optimizations are typical optimizations performed in TensorFlow.
+
+* **Constant Folding** evaluates the operations whose input tensors are all constant.
+* **Memory Optimizer** analyzes the TensorFlow graph to inspect the peak memory usage at each operations. To reduce a peak memory usage, Memory Optimizer inserts CPU-GPU memory copy operations for swapping GPU memory to CPU, or rewrites TensorFlow graph in order to the recomputation.


Missing verb here.

Thanks, I fixed it.

MarkDaoust · 2019-04-25T21:56:42Z

site/en/guide/graphs.md

+* Enable Debug Stripper
+
+**Debug Stripper** which is disabled by default, strips the operations used for the debug purpose (Assert, CheckNumerics, Print).
+To enable Debug Stripper, you need to write codes as follows.


Suggested change

To enable Debug Stripper, you need to write codes as follows.

To enable Debug Stripper, you need to write code as follows.

Thanks for a suggestion. I fixed it.

MarkDaoust · 2019-04-25T21:57:19Z

site/en/guide/graphs.md

+**Debug Stripper** which is disabled by default, strips the operations used for the debug purpose (Assert, CheckNumerics, Print).
+To enable Debug Stripper, you need to write codes as follows.
+
+Only you need to do is changing `RewriteOptions::debug_stripper` enabled.


Suggested change

Only you need to do is changing `RewriteOptions::debug_stripper` enabled.

The only thing you need to do is enable `RewriteOptions::debug_stripper`.

I fixed it.

MarkDaoust · 2019-04-25T22:00:09Z

site/en/guide/graphs.md

+To disable Constant Folding, you need to write code as follows.
+
+**Constant Folding** is performed more than two times on the graph optimization process in TensorFlow.
+You need to change not only `RewriteOptions::constant_folding`, but also `OptimizerOptions::opt_level`.


Suggested change

You need to change not only `RewriteOptions::constant_folding`, but also `OptimizerOptions::opt_level`.

You need to change both`RewriteOptions::constant_folding`, and `OptimizerOptions::opt_level`.

Thanks. I fixed it.

MarkDaoust · 2019-04-25T22:01:15Z

site/en/guide/graphs.md

+To disable Constant Folding, you need to write code as follows.
+
+**Constant Folding** is performed more than two times on the graph optimization process in TensorFlow.
+You need to change not only `RewriteOptions::constant_folding`, but also `OptimizerOptions::opt_level`.


Since this is a python doc, should we be referring to these by their python names?

like graph_options.rewrite_options.constant_folding

OK, I replaced all parts like this issue in the document.

MarkDaoust · 2019-04-25T22:03:49Z

@suharshs, you know about graph optimizations. Can you validate this Docs PR, or suggest someone who can?

Is anything here relevant to the tf2 docs?

MarkDaoust · 2019-04-25T22:04:21Z

site/en/guide/graphs.md

+cfg.graph_options.optimizer_options.opt_level = config_pb2.OptimizerOptions.L0
+
+# build the computation graph
+train_op = ...


It's best if code-examples are copy-paste runnable.

Added simple code-example.

MarkDaoust · 2019-04-25T22:04:51Z

site/en/guide/graphs.md

+cfg.graph_options.rewrite_options.debug_stripper = rewriter_config_pb2.RewriterConfig.ON
+
+# build the computation graph
+train_op = ...


Can you put something here so the example is copy-paste runnable?

Added simple code-example too.

MarkDaoust · 2019-04-25T22:06:48Z

site/en/guide/graphs.md

+train_op = ...
+
+# execute the computation graph with Debug Stripper ON
+with tf.Session(config=cfg) as sess:


Can we do something here to show that the optimization is really applied?

Maybe include an assert or check-numerics above, and then pass it bad data here?

Added the instructions how to inspect the optimized graph as Inspect the Optimized Graph section. Do these instructions match your intention?

MarkDaoust · 2019-04-25T22:07:42Z

site/en/guide/graphs.md

+train_op = ...
+
+# execute the computation graph with Debug Stripper ON
+with tf.Session(config=cfg) as sess:


Is there a way we can verify that this does what's expected? is there a way to inspect the optimized graph?

Same comment as "Constant Folding".

suharshs · 2019-04-25T22:20:36Z

I think this is grappler specific, so @rmlarsen may be a better fit.

rmlarsen · 2019-04-29T23:49:25Z

We just released a slide deck about this used for a presentation given to the CS245 class at Stanford University on April 24, 2019. A recording of the talk should be available as well.

nuka137 · 2019-05-08T00:55:38Z

@MarkDaoust

Thanks for the review.
I fixed all issues you mentioned.
Could you review it again?

@rmlarsen @MarkDaoust

This slide has lots of useful information. Thanks for sharing.
However, I'm not sure whether I should write all information of this slide to this document.

At the beginning, I assumed that this document is for the beginner of the graph optimization.
But this slide will be great information for the more advanced user.
How about adding a reference of this slide to the document at this first release?
Any suggestions?

lamberta · 2019-06-24T22:54:10Z

Hey @alextp, can you take a quick look at this pull request?
Is this still relevant for TF2? If so, is it accurate and how should we surface this content? Should it remain a developer guide as part of the main TF2 guides or moved into GitHub as kind of an internals doc? WDYT? Thanks

alextp · 2019-06-25T16:02:11Z

@lamberta documenting the graph optimizations is a good idea. TF 2 has changed somewhat the syntax to control them, now it's in tf.config.optimizer: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/config/optimizer

I assume you'll proof-read for English mistakes and whatnot before merging, though.

@ezhulenev @rmlarsen for FYI since you work on these parts of tf and might have strong opinions about the content in this document.

nuka137 · 2019-07-10T22:18:09Z

@lamberta @alextp

Recently, I found that tensorflow.python.eager.context.export_run_metadata is a way in TF 2.0 to inspect the optimized (function) graph.
And also, graph optimization can be applied via tf.config.optimizer in TF 2.0.
Should I rewrite whole documents to TF 2.0?

lamberta · 2019-07-11T04:09:41Z

Hi @nuka137,
Our focus now is on TF2 docs (we'll switch the site over soon enough), so, yes, updating this doc to TF2 would be very helpful 😀

nuka137 · 2019-09-26T03:56:46Z

@lamberta

Hi, I'm so sorry for late reply. I'm going to resume this pull request.
But I found that there is no document site/en/guide/graphs.md in master branch.
Where should I move this document?
Graph optimization is also effective in TF2 as tf.function.

lamberta · 2019-09-27T22:29:57Z

Thanks, @nuka137
I think the best way forward for this is to work it into the autograph reference that @mdanatg is working on: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/autograph/g3doc/reference

I don't think we're going to have an separate graphs.mddoc for TF2, and this will require a considerable rewrite. Can we get the relevant performance optimization info into the autograph reference first, then later we'll look at creating a separate user guide on tf.org.

Sound ok?

nuka137 · 2019-09-28T00:44:21Z

@lamberta

Thanks!
I understood your instruction. It is great idea to work in autograph document.

Add: Document about the Graph Optimization in TensorFlow

79a23bd

nuka137 requested review from lamberta and MarkDaoust as code owners April 22, 2019 22:02

googlebot added the cla: yes CLA has been signed label Apr 22, 2019

lamberta added the awaiting-technical-review label Apr 24, 2019

MarkDaoust requested a review from suharshs April 25, 2019 22:01

MarkDaoust reviewed Apr 25, 2019

View reviewed changes

Fix: review

197d52c

lamberta closed this Sep 27, 2019

nuka137 mentioned this pull request Nov 20, 2019

Add Document about the Graph Optimization in TensorFlow #1206

Merged

	To enable Debug Stripper, you need to write codes as follows.
	To enable Debug Stripper, you need to write code as follows.

	Only you need to do is changing `RewriteOptions::debug_stripper` enabled.
	The only thing you need to do is enable `RewriteOptions::debug_stripper`.

	You need to change not only `RewriteOptions::constant_folding`, but also `OptimizerOptions::opt_level`.
	You need to change both`RewriteOptions::constant_folding`, and `OptimizerOptions::opt_level`.

Add Document about the Graph Optimization in TensorFlow #511

Add Document about the Graph Optimization in TensorFlow #511

Uh oh!

Conversation

nuka137 commented Apr 22, 2019

Uh oh!

lamberta commented Apr 24, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarkDaoust commented Apr 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suharshs commented Apr 25, 2019

Uh oh!

rmlarsen commented Apr 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nuka137 commented May 8, 2019

Uh oh!

lamberta commented Jun 24, 2019

Uh oh!

alextp commented Jun 25, 2019

Uh oh!

nuka137 commented Jul 10, 2019

Uh oh!

lamberta commented Jul 11, 2019

Uh oh!

nuka137 commented Sep 26, 2019

Uh oh!

lamberta commented Sep 27, 2019

Uh oh!

nuka137 commented Sep 28, 2019

Uh oh!

Uh oh!

rmlarsen commented Apr 29, 2019 •

edited

Loading