Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Auto Graph Fusion][Modelzoo] After enable Auto op fusion feature in WDL of modelzoo, but get an error. #126

Closed
Duyi-Wang opened this issue Mar 22, 2022 · 1 comment
Closed

[Auto Graph Fusion][Modelzoo] After enable Auto op fusion feature in WDL of modelzoo, but get an error. #126

Duyi-Wang opened this issue Mar 22, 2022 · 1 comment
Assignees
@JackMoriarty

Comments

@Duyi-Wang
Copy link
Contributor

Duyi-Wang commented Mar 22, 2022 •
edited
Loading

I want to enable Auto Graph Fusion feature in WDL and follow the steps in DeepRec Docs, but I get an error.

Code to reproduce the issue
I use following codes to enable Auto Graph Fusion. The full code please see Full code

        if args.op_fusion and not args.tf:
            '''Auto Graph Fusion'''
            sess_config.graph_options.optimizer_options.do_op_fusion = True

Run python train.py --steps 1000 --no_eval --op_fusion True can reproduce error. Use WDL dataset.

logs

INFO:tensorflow:Parsing ./data/train.csv
INFO:tensorflow:Parsing ./data/eval.csv
INFO:tensorflow:Graph was finalized.
2022-03-22 14:10:30.688360: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3000000000 Hz
2022-03-22 14:10:30.707518: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5622edbbebf0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-03-22 14:10:30.707558: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
INFO:tensorflow:run without loading checkpoint
2022-03-22 14:10:30.786850: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_else_const head/gradients/head/loss/xentropy/Select_grad/zeros_like
2022-03-22 14:10:30.787074: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/head/loss/xentropy/Select_grad/Select_1
2022-03-22 14:10:30.787248: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_else_const head/gradients_1/head/loss/xentropy/Select_grad/zeros_like
2022-03-22 14:10:30.787437: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/head/loss/xentropy/Select_grad/Select_1
2022-03-22 14:10:30.787920: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C10_embedding/C10_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.788067: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C10_embedding/C10_embedding_weights_grad/Select
2022-03-22 14:10:30.788089: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C11_embedding/C11_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.788232: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C11_embedding/C11_embedding_weights_grad/Select
2022-03-22 14:10:30.788255: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C12_embedding/C12_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.788394: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C12_embedding/C12_embedding_weights_grad/Select
2022-03-22 14:10:30.788415: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C13_embedding/C13_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.788554: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C13_embedding/C13_embedding_weights_grad/Select
2022-03-22 14:10:30.788575: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C14_embedding/C14_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.788714: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C14_embedding/C14_embedding_weights_grad/Select
2022-03-22 14:10:30.788735: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C15_embedding/C15_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.788875: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C15_embedding/C15_embedding_weights_grad/Select
2022-03-22 14:10:30.788895: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C16_embedding/C16_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.789049: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C16_embedding/C16_embedding_weights_grad/Select
2022-03-22 14:10:30.789071: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C17_embedding/C17_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.789214: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C17_embedding/C17_embedding_weights_grad/Select
2022-03-22 14:10:30.789234: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C18_embedding/C18_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.789374: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C18_embedding/C18_embedding_weights_grad/Select
2022-03-22 14:10:30.789395: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C19_embedding/C19_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.789534: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C19_embedding/C19_embedding_weights_grad/Select
2022-03-22 14:10:30.789554: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C1_embedding/C1_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.789693: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C1_embedding/C1_embedding_weights_grad/Select
2022-03-22 14:10:30.789715: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C20_embedding/C20_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.789854: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C20_embedding/C20_embedding_weights_grad/Select
2022-03-22 14:10:30.789874: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C21_embedding/C21_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.790014: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C21_embedding/C21_embedding_weights_grad/Select
2022-03-22 14:10:30.790035: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C22_embedding/C22_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.790177: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C22_embedding/C22_embedding_weights_grad/Select
2022-03-22 14:10:30.790197: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C23_embedding/C23_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.790338: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C23_embedding/C23_embedding_weights_grad/Select
2022-03-22 14:10:30.790359: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C24_embedding/C24_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.790504: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C24_embedding/C24_embedding_weights_grad/Select
2022-03-22 14:10:30.790529: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C25_embedding/C25_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.790687: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C25_embedding/C25_embedding_weights_grad/Select
2022-03-22 14:10:30.790708: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C26_embedding/C26_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.790851: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C26_embedding/C26_embedding_weights_grad/Select
2022-03-22 14:10:30.790872: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C2_embedding/C2_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.791015: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C2_embedding/C2_embedding_weights_grad/Select
2022-03-22 14:10:30.791036: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C3_embedding/C3_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.791183: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C3_embedding/C3_embedding_weights_grad/Select
2022-03-22 14:10:30.791204: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C4_embedding/C4_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.791348: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C4_embedding/C4_embedding_weights_grad/Select
2022-03-22 14:10:30.791369: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C5_embedding/C5_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.791513: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C5_embedding/C5_embedding_weights_grad/Select
2022-03-22 14:10:30.791544: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C6_embedding/C6_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.791686: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C6_embedding/C6_embedding_weights_grad/Select
2022-03-22 14:10:30.791706: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C7_embedding/C7_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.791848: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C7_embedding/C7_embedding_weights_grad/Select
2022-03-22 14:10:30.791874: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C8_embedding/C8_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.792021: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C8_embedding/C8_embedding_weights_grad/Select
2022-03-22 14:10:30.792042: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients/dnn/input_from_feature_columns/input_layer/C9_embedding/C9_embedding_weights_grad/zeros_like
2022-03-22 14:10:30.792188: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients/dnn/input_from_feature_columns/input_layer/C9_embedding/C9_embedding_weights_grad/Select
2022-03-22 14:10:30.792258: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C1/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.792436: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C1/weighted_sum_grad/Select
2022-03-22 14:10:30.792456: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C10/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.792632: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C10/weighted_sum_grad/Select
2022-03-22 14:10:30.792654: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C11/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.792827: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C11/weighted_sum_grad/Select
2022-03-22 14:10:30.792848: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C12/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.793022: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C12/weighted_sum_grad/Select
2022-03-22 14:10:30.793043: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C13/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.793221: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C13/weighted_sum_grad/Select
2022-03-22 14:10:30.793243: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C14/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.793417: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C14/weighted_sum_grad/Select
2022-03-22 14:10:30.793439: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C15/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.793612: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C15/weighted_sum_grad/Select
2022-03-22 14:10:30.793633: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C16/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.793807: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C16/weighted_sum_grad/Select
2022-03-22 14:10:30.793829: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C17/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.794006: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C17/weighted_sum_grad/Select
2022-03-22 14:10:30.794027: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C18/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.794205: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C18/weighted_sum_grad/Select
2022-03-22 14:10:30.794227: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C19/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.794401: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C19/weighted_sum_grad/Select
2022-03-22 14:10:30.794422: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C2/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.794596: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C2/weighted_sum_grad/Select
2022-03-22 14:10:30.794618: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C20/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.794793: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C20/weighted_sum_grad/Select
2022-03-22 14:10:30.794813: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C21/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.794988: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C21/weighted_sum_grad/Select
2022-03-22 14:10:30.795011: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C22/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.795189: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C22/weighted_sum_grad/Select
2022-03-22 14:10:30.795210: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C23/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.795385: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C23/weighted_sum_grad/Select
2022-03-22 14:10:30.795407: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C24/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.795582: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C24/weighted_sum_grad/Select
2022-03-22 14:10:30.795602: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C25/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.795778: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C25/weighted_sum_grad/Select
2022-03-22 14:10:30.795799: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C26/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.795974: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C26/weighted_sum_grad/Select
2022-03-22 14:10:30.795999: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C3/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.796178: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C3/weighted_sum_grad/Select
2022-03-22 14:10:30.796200: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C4/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.796375: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C4/weighted_sum_grad/Select
2022-03-22 14:10:30.796396: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C5/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.796572: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C5/weighted_sum_grad/Select
2022-03-22 14:10:30.796593: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C6/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.796768: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C6/weighted_sum_grad/Select
2022-03-22 14:10:30.796790: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C7/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.796966: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C7/weighted_sum_grad/Select
2022-03-22 14:10:30.796991: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C8/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.797170: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C8/weighted_sum_grad/Select
2022-03-22 14:10:30.797192: I ./tensorflow/core/graph/template_select_pruning_base.h:69] Found match op by select_pruning_then_const head/gradients_1/linear/linear_model_1/linear_model/C9/weighted_sum_grad/zeros_like
2022-03-22 14:10:30.797368: I ./tensorflow/core/graph/template_select_pruning_base.h:75] remove node: head/gradients_1/linear/linear_model_1/linear_model/C9/weighted_sum_grad/Select
2022-03-22 14:10:30.797492: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C10_embedding/C10_embedding_weights]
2022-03-22 14:10:30.797542: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C11_embedding/C11_embedding_weights]
2022-03-22 14:10:30.797564: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C12_embedding/C12_embedding_weights]
2022-03-22 14:10:30.797583: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C13_embedding/C13_embedding_weights]
2022-03-22 14:10:30.797602: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C14_embedding/C14_embedding_weights]
2022-03-22 14:10:30.797621: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C15_embedding/C15_embedding_weights]
2022-03-22 14:10:30.797645: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C16_embedding/C16_embedding_weights]
2022-03-22 14:10:30.797664: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C17_embedding/C17_embedding_weights]
2022-03-22 14:10:30.797682: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C18_embedding/C18_embedding_weights]
2022-03-22 14:10:30.797701: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C19_embedding/C19_embedding_weights]
2022-03-22 14:10:30.797720: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C1_embedding/C1_embedding_weights]
2022-03-22 14:10:30.797739: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C20_embedding/C20_embedding_weights]
2022-03-22 14:10:30.797757: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C21_embedding/C21_embedding_weights]
2022-03-22 14:10:30.797776: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C22_embedding/C22_embedding_weights]
2022-03-22 14:10:30.797794: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C23_embedding/C23_embedding_weights]
2022-03-22 14:10:30.797813: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C24_embedding/C24_embedding_weights]
2022-03-22 14:10:30.797832: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C25_embedding/C25_embedding_weights]
2022-03-22 14:10:30.797850: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C26_embedding/C26_embedding_weights]
2022-03-22 14:10:30.797868: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C2_embedding/C2_embedding_weights]
2022-03-22 14:10:30.797886: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C3_embedding/C3_embedding_weights]
2022-03-22 14:10:30.797905: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C4_embedding/C4_embedding_weights]
2022-03-22 14:10:30.797923: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C5_embedding/C5_embedding_weights]
2022-03-22 14:10:30.797940: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C6_embedding/C6_embedding_weights]
2022-03-22 14:10:30.797958: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C7_embedding/C7_embedding_weights]
2022-03-22 14:10:30.797976: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C8_embedding/C8_embedding_weights]
2022-03-22 14:10:30.797994: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[dnn/input_from_feature_columns/input_layer/C9_embedding/C9_embedding_weights]
2022-03-22 14:10:30.798035: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C1/weighted_sum]
2022-03-22 14:10:30.798054: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C10/weighted_sum]
2022-03-22 14:10:30.798072: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C11/weighted_sum]
2022-03-22 14:10:30.798091: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C12/weighted_sum]
2022-03-22 14:10:30.798113: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C13/weighted_sum]
2022-03-22 14:10:30.798132: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C14/weighted_sum]
2022-03-22 14:10:30.798150: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C15/weighted_sum]
2022-03-22 14:10:30.798169: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C16/weighted_sum]
2022-03-22 14:10:30.798188: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C17/weighted_sum]
2022-03-22 14:10:30.798206: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C18/weighted_sum]
2022-03-22 14:10:30.798223: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C19/weighted_sum]
2022-03-22 14:10:30.798242: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C2/weighted_sum]
2022-03-22 14:10:30.798261: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C20/weighted_sum]
2022-03-22 14:10:30.798279: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C21/weighted_sum]
2022-03-22 14:10:30.798297: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C22/weighted_sum]
2022-03-22 14:10:30.798316: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C23/weighted_sum]
2022-03-22 14:10:30.798334: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C24/weighted_sum]
2022-03-22 14:10:30.798353: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C25/weighted_sum]
2022-03-22 14:10:30.798371: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C26/weighted_sum]
2022-03-22 14:10:30.798389: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C3/weighted_sum]
2022-03-22 14:10:30.798408: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C4/weighted_sum]
2022-03-22 14:10:30.798426: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C5/weighted_sum]
2022-03-22 14:10:30.798445: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C6/weighted_sum]
2022-03-22 14:10:30.798467: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C7/weighted_sum]
2022-03-22 14:10:30.798485: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C8/weighted_sum]
2022-03-22 14:10:30.798503: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar] match op[linear/linear_model_1/linear_model/C9/weighted_sum]
2022-03-22 14:10:30.798945: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_else_scalar] match op[head/loss/xentropy/Select]
2022-03-22 14:10:30.799421: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_else_scalar_in_grad] match op[head/gradients/head/loss/xentropy/Select_grad/Select]
2022-03-22 14:10:30.799453: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_else_scalar_in_grad] match op[head/gradients/head/loss/xentropy/Select_1_grad/Select]
2022-03-22 14:10:30.799528: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_else_scalar_in_grad] match op[head/gradients_1/head/loss/xentropy/Select_grad/Select]
2022-03-22 14:10:30.799546: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_else_scalar_in_grad] match op[head/gradients_1/head/loss/xentropy/Select_1_grad/Select]
2022-03-22 14:10:30.799958: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/head/loss/xentropy/Select_1_grad/Select_1]
2022-03-22 14:10:30.799996: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C10_embedding/C10_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800013: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C11_embedding/C11_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800028: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C12_embedding/C12_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800043: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C13_embedding/C13_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800057: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C14_embedding/C14_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800072: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C15_embedding/C15_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800087: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C16_embedding/C16_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800101: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C17_embedding/C17_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800120: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C18_embedding/C18_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800136: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C19_embedding/C19_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800155: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C1_embedding/C1_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800169: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C20_embedding/C20_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800184: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C21_embedding/C21_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800199: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C22_embedding/C22_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800213: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C23_embedding/C23_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800228: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C24_embedding/C24_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800242: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C25_embedding/C25_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800256: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C26_embedding/C26_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800271: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C2_embedding/C2_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800286: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C3_embedding/C3_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800300: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C4_embedding/C4_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800314: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C5_embedding/C5_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800329: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C6_embedding/C6_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800344: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C7_embedding/C7_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800358: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C8_embedding/C8_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800373: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients/dnn/input_from_feature_columns/input_layer/C9_embedding/C9_embedding_weights_grad/Select_1]
2022-03-22 14:10:30.800428: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/head/loss/xentropy/Select_1_grad/Select_1]
2022-03-22 14:10:30.800455: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C1/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800471: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C10/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800493: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C11/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800508: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C12/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800523: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C13/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800538: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C14/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800552: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C15/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800567: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C16/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800582: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C17/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800596: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C18/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800610: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C19/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800625: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C2/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800639: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C20/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800654: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C21/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800670: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C22/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800685: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C23/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800699: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C24/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800714: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C25/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800728: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C26/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800745: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C3/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800760: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C4/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800774: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C5/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800790: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C6/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800804: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C7/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800819: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C8/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.800833: I ./tensorflow/core/graph/template_select_base.h:36] Fusion template[select_then_scalar_in_grad] match op[head/gradients_1/linear/linear_model_1/linear_model/C9/weighted_sum_grad/Select_1]
2022-03-22 14:10:30.919577: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] model_pruner failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1' has missing fanin 'head/gradients/head/loss/xentropy/Select_grad/Select_1'.
2022-03-22 14:10:30.941287: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] function_optimizer failed: Invalid argument: Node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1': Unknown input node 'head/gradients/head/loss/xentropy/Select_grad/Select_1'
2022-03-22 14:11:15.303107: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] model_pruner failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1' has missing fanin 'head/gradients/head/loss/xentropy/Select_grad/Select_1'.
2022-03-22 14:11:15.324636: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] function_optimizer failed: Invalid argument: Node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1': Unknown input node 'head/gradients/head/loss/xentropy/Select_grad/Select_1'
INFO:tensorflow:Running local_init_op.
2022-03-22 14:11:23.962906: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] model_pruner failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1' has missing fanin 'head/gradients/head/loss/xentropy/Select_grad/Select_1'.
2022-03-22 14:11:23.985236: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] function_optimizer failed: Invalid argument: Node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1': Unknown input node 'head/gradients/head/loss/xentropy/Select_grad/Select_1'
INFO:tensorflow:Done running local_init_op.
2022-03-22 14:11:32.103417: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] model_pruner failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1' has missing fanin 'head/gradients/head/loss/xentropy/Select_grad/Select_1'.
2022-03-22 14:11:32.125529: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] function_optimizer failed: Invalid argument: Node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1': Unknown input node 'head/gradients/head/loss/xentropy/Select_grad/Select_1'
2022-03-22 14:11:40.871891: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] model_pruner failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1' has missing fanin 'head/gradients/head/loss/xentropy/Select_grad/Select_1'.
2022-03-22 14:11:40.894088: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:583] function_optimizer failed: Invalid argument: Node 'head/gradients/head/loss/xentropy/Select_grad/tuple/control_dependency_1': Unknown input node 'head/gradients/head/loss/xentropy/Select_grad/Select_1'
Using TensorFlow version 1.15.5
Checking dataset...
Numbers of training dataset is 8000000
Numbers of test dataset is 2000000
The training steps is 100
The testing steps is 3907
Saving model checkpoints to ./result/model_WIDE_AND_DEEP_1647929426
Traceback (most recent call last):
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected begin[0] == 0 (got 1) and size[0] == 0 (got -1) when input.dim_size(0) == 0
	 [[{{node linear/linear_model_1/linear_model/C20/weighted_sum/Slice_2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_rebuild.py", line 737, in <module>
    main()
  File "train_rebuild.py", line 537, in main
    checkpoint_dir, tf_config, server)
  File "train_rebuild.py", line 414, in train
    sess.run([model.loss, model.train_op])
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 804, in run
    run_metadata=run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1309, in run
    run_metadata=run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1410, in run
    raise six.reraise(*original_exc_info)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1395, in run
    return self._sess.run(*args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1468, in run
    run_metadata=run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1226, in run
    return self._sess.run(*args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expected begin[0] == 0 (got 1) and size[0] == 0 (got -1) when input.dim_size(0) == 0
	 [[node linear/linear_model_1/linear_model/C20/weighted_sum/Slice_2 (defined at /home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'linear/linear_model_1/linear_model/C20/weighted_sum/Slice_2':
  File "train_rebuild.py", line 737, in <module>
    main()
  File "train_rebuild.py", line 517, in main
    dense_layer_partitioner=dense_layer_partitioner)
  File "train_rebuild.py", line 116, in __init__
    self._create_model()
  File "train_rebuild.py", line 187, in _create_model
    trainable=True)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column.py", line 504, in linear_model
    retval = linear_model_layer(features)  # pylint: disable=not-callable
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 871, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 234, in wrapper
    return converted_call(f, options, args, kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 439, in converted_call
    return _call_unconverted(f, args, kwargs, options)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 330, in _call_unconverted
    return f(*args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column.py", line 696, in call
    weighted_sum = layer(builder)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/layers/base.py", line 564, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 915, in __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column.py", line 588, in call
    weight_var=self._weight_var)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column.py", line 1938, in _create_weighted_sum
    weight_var=weight_var)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column.py", line 2081, in _create_categorical_column_weighted_sum
    name='weighted_sum')
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/embedding_ops.py", line 1338, in safe_embedding_lookup_sparse
    array_ops.slice(array_ops.shape(result), [1], [-1])
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 855, in slice
    return gen_array_ops._slice(input_, begin, size, name=name)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 9272, in _slice
    "Slice", input=input, begin=begin, size=size, name=name)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JackMoriarty JackMoriarty

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@liutongxuan @JackMoriarty @Duyi-Wang