TF-TRT C++ conversion

[tfeher]

TF-TRT C++ interface to convert models.

Differences compared to the Python API:

• It is assumed that the input graph is frozen. This assumption will be removed in a follow up PR.
• convert_to_static_engine conversion param allows to convert dynamic engines to static engines.
• Currently we do not provide a way to save the engine files as assets.

The steps for model conversion by ConvertAndBuild

1. Inline functions
2. Freeze the graph - omitted since we assume frozen input graph
3. Run Grappler with TRTOptimizer pass
4. Infer the graph to provide shape information (only in dynamic shape mode).
5. Infer the graph to build the engines
6. Convert the graph_def to have static engines

(On the Python side, steps 4-5 are done by a separate build function.)

Related PRs:

• Load and select optimization profiles for static TRT engine #52047
• Example code for the tensorflow/tensorrt repository TF-TRT C++ conversion example tensorrt#271

[bixia1]

We can avoid the change to the utilities for freezing the graph by updating out SavedModelBundle with the converted GraphDef. Something like this:

MetaGraphDef* meta_graph_def = &saved_model_bundle->meta_graph_def; 
*meta_graph_def->mutable_graph_def() = graph_def;

[tfeher]

@bixia1 ready for review.

[bixia1]

I am still working on reviewing trt_convert.cc .

[bixia1]

Still working on trt_convert.cc.

[bixia1]

If we will go with the API where users provide input/output names, we should setup ItemConfig fetch_nodes/feed_nodes.
If we will go with the API where users provide SignatureDef, we need to clear the SignatureDef in the input meta_graph_def and keep only the one we need.

Also, if we will need to run grappler a few times, we probably want to avoid constructing GrapplerItem each time by somehow reusing the GrapplerItem, see TF code here.

[tfeher]

• I have set up feed_nodes and fetch_nodes.
• If a previous conversion had RunMetaOptimizer(item, ..., out_graph_def), can we just replace the item's graph item.graph = out_graph_def before the next RunMetaOptimizer call? For now we have only a single grappler pass, but it might matter in the follow up PR where we need to inline the graph before freezing it.

[tfeher]

Thanks @bixia1 for the review I have addressed most of the issues. Pleas have a look.

[tfeher]

• I have set up feed_nodes and fetch_nodes.
• If a previous conversion had RunMetaOptimizer(item, ..., out_graph_def), can we just replace the item's graph item.graph = out_graph_def before the next RunMetaOptimizer call? For now we have only a single grappler pass, but it might matter in the follow up PR where we need to inline the graph before freezing it.

[tfeher]

@bixia1 I have removed the circular dependency, and added trt_convert_api as a dependency of trt_op_libs. Let me know if there are other issues that we want to address in this PR.

[bixia1]

Can you fix the PR description, such as to remove the information that is obsolete? The PR description will become part of the commit message.

[tfeher]

Thanks @bixia1 for the review, I have addressed the issues.