[Adaptive Embedding] After enable Adaptive embedding, it fails to evaluate model with modelzoo.

[Duyi-Wang]

After enable Adaptive embedding, it fails to evaluate model with modelzoo after completing training.

Code to reproduce the issue
With WDL in modelzoo, run python train.py --steps 100 --adaptive_emb true

Other info / logs

Training completed.                                                                                                                                                                                                                                                     
INFO:tensorflow:Graph was finalized.                                                                                                                                                                                                                                    
INFO:tensorflow:run with loading checkpoint                                                                                                                                                                                                                             
INFO:tensorflow:Restoring parameters from ./result/model_BST_1653893703/model.ckpt-100                                                                                                                                                                                  
2022-05-30 14:56:04.871953: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/adgroup_id_embedding/adgroup_id_embedding_weights][new_name:fused_op_1_select_then_scalar]      
2022-05-30 14:56:04.872043: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/age_level_embedding/age_level_embedding_weights][new_name:fused_op_2_select_then_scalar]        
2022-05-30 14:56:04.872552: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/brand_embedding/brand_embedding_weights][new_name:fused_op_3_select_then_scalar]                
2022-05-30 14:56:04.872924: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/campaign_id_embedding/campaign_id_embedding_weights][new_name:fused_op_4_select_then_scalar]    
2022-05-30 14:56:04.873322: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/cate_id_embedding/cate_id_embedding_weights][new_name:fused_op_5_select_then_scalar]            
2022-05-30 14:56:04.873678: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/cms_group_id_embedding/cms_group_id_embedding_weights][new_name:fused_op_6_select_then_scalar]  
2022-05-30 14:56:04.874156: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/cms_segid_embedding/cms_segid_embedding_weights][new_name:fused_op_7_select_then_scalar]        
2022-05-30 14:56:04.874631: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/customer_embedding/customer_embedding_weights][new_name:fused_op_8_select_then_scalar]          
2022-05-30 14:56:04.875088: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/new_user_class_level_embedding/new_user_class_level_embedding_weights][new_name:fused_op_9_sele$
t_then_scalar]                                                                                                                                                                                                                                                          
2022-05-30 14:56:04.875571: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/occupation_embedding/occupation_embedding_weights][new_name:fused_op_10_select_then_scalar]     
2022-05-30 14:56:04.875981: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/pid_embedding/pid_embedding_weights][new_name:fused_op_11_select_then_scalar]                   
2022-05-30 14:56:04.876455: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/price_embedding/price_embedding_weights][new_name:fused_op_12_select_then_scalar]               
2022-05-30 14:56:04.876896: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/pvalue_level_embedding/pvalue_level_embedding_weights][new_name:fused_op_13_select_then_scalar] 
2022-05-30 14:56:04.877411: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/shopping_level_embedding/shopping_level_embedding_weights][new_name:fused_op_14_select_then_scal
ar]
2022-05-30 14:56:04.877944: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar] match op[input_layer/unseq_input_layer/input_layer/user_id_embedding/user_id_embedding_weights][new_name:fused_op_15_select_then_scalar]
2022-05-30 14:56:04.880237: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/adgroup_id_embedding/adgroup_id_embedding_weights_grad/Select][new_name:f
used_op_1_select_else_scalar_in_grad]
2022-05-30 14:56:04.880278: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/age_level_embedding/age_level_embedding_weights_grad/Select][new_name:fus
ed_op_2_select_else_scalar_in_grad]
2022-05-30 14:56:04.880297: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/campaign_id_embedding/campaign_id_embedding_weights_grad/Select[101/4331$
:fused_op_3_select_else_scalar_in_grad]                                                                                                                                                                                                                                 
2022-05-30 14:56:04.880316: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/cms_group_id_embedding/cms_group_id_embedding_weights_grad/Select][new_na
me:fused_op_4_select_else_scalar_in_grad]                                                                                                                                                                                                                               
2022-05-30 14:56:04.880335: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/cms_segid_embedding/cms_segid_embedding_weights_grad/Select][new_name:fus
ed_op_5_select_else_scalar_in_grad]                                                                                                                                                                                                                                     
2022-05-30 14:56:04.880351: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/customer_embedding/customer_embedding_weights_grad/Select][new_name:fused
_op_6_select_else_scalar_in_grad]                                                                                                                                                                                                                                       
2022-05-30 14:56:04.880367: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/new_user_class_level_embedding/new_user_class_level_embedding_weights_gra
d/Select][new_name:fused_op_7_select_else_scalar_in_grad]                                                                                                                                                                                                               
2022-05-30 14:56:04.880383: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/occupation_embedding/occupation_embedding_weights_grad/Select][new_name:f
used_op_8_select_else_scalar_in_grad]                                                                                                                                                                                                                                   
2022-05-30 14:56:04.880398: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/pid_embedding/pid_embedding_weights_grad/Select][new_name:fused_op_9_sele
ct_else_scalar_in_grad]                                                                                                                                                                                                                                                 
2022-05-30 14:56:04.880413: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/price_embedding/price_embedding_weights_grad/Select][new_name:fused_op_10
_select_else_scalar_in_grad]                                                                                                                                                                                                                                            
2022-05-30 14:56:04.880428: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/pvalue_level_embedding/pvalue_level_embedding_weights_grad/Select][new_na
me:fused_op_11_select_else_scalar_in_grad]                                                                                                                                                                                                                              
2022-05-30 14:56:04.880443: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/shopping_level_embedding/shopping_level_embedding_weights_grad/Select][ne
w_name:fused_op_12_select_else_scalar_in_grad]                                                                                                                                                                                                                          
2022-05-30 14:56:04.880458: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/user_id_embedding/user_id_embedding_weights_grad/Select][new_name:fused_o
p_13_select_else_scalar_in_grad]                                                                                                                                                                                                                                        
2022-05-30 14:56:04.880521: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/cate_id_embedding/cate_id_embedding_weights_grad/Select][new_name:fused_o
p_14_select_else_scalar_in_grad]                                                                                                                                                                                                                                        
2022-05-30 14:56:04.880537: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_else_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/brand_embedding/brand_embedding_weights_grad/Select][new_name:fused_op_15
_select_else_scalar_in_grad]                                                                                                                                                                                                                                            
2022-05-30 14:56:04.881665: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/adgroup_id_embedding/adgroup_id_embedding_weights_grad/Select_1][new_nam$
:fused_op_1_select_then_scalar_in_grad]                                                                                                                                                                                                                                
2022-05-30 14:56:04.881789: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/age_level_embedding/age_level_embedding_weights_grad/Select_1][new_name:$
used_op_2_select_then_scalar_in_grad]                                                                                                                                                                                                                                  
2022-05-30 14:56:04.882280: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/campaign_id_embedding/campaign_id_embedding_weights_grad/Select_1][new_n$
me:fused_op_3_select_then_scalar_in_grad]                                                                                                                                                                                                                              
2022-05-30 14:56:04.882307: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/cms_group_id_embedding/cms_group_id_embedding_weights_grad/Select_1][new$
name:fused_op_4_select_then_scalar_in_grad]                                                                                                  
2022-05-30 14:56:04.882325: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/cms_segid_embedding/cms_segid_embedding_weights_grad/Select_1][new_name:$
used_op_5_select_then_scalar_in_grad]                                                                                                                                                                                                                                  
2022-05-30 14:56:04.882343: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/customer_embedding/customer_embedding_weights_grad/Select_1][new_name:fu$
ed_op_6_select_then_scalar_in_grad]                                                                                                                                                                                                                                    
2022-05-30 14:56:04.882359: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/new_user_class_level_embedding/new_user_class_level_embedding_weights_gr$
d/Select_1][new_name:fused_op_7_select_then_scalar_in_grad]                                                                                                                                                                                                            
2022-05-30 14:56:04.882375: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/occupation_embedding/occupation_embedding_weights_grad/Select_1][new_nam$
:fused_op_8_select_then_scalar_in_grad]                                                                                                      
2022-05-30 14:56:04.882391: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/pid_embedding/pid_embedding_weights_grad/Select_1][new_name:fused_op_9_s$
lect_then_scalar_in_grad]                                                                
2022-05-30 14:56:04.882408: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/price_embedding/price_embedding_weights_grad/Select_1][new_name:fused_op$
10_select_then_scalar_in_grad]                                                                                                                                                                                                                                         
2022-05-30 14:56:04.882423: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/pvalue_level_embedding/pvalue_level_embedding_weights_grad/Select_1][new$
name:fused_op_11_select_then_scalar_in_grad]                                                                                                                                                                                                                           
2022-05-30 14:56:04.882440: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/shopping_level_embedding/shopping_level_embedding_weights_grad/Select_1]$
new_name:fused_op_12_select_then_scalar_in_grad]
2022-05-30 14:56:04.882456: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/user_id_embedding/user_id_embedding_weights_grad/Select_1][new_name:fuse$
_op_13_select_then_scalar_in_grad]        
2022-05-30 14:56:04.882515: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/cate_id_embedding/cate_id_embedding_weights_grad/Select_1][new_name:fuse$
_op_14_select_then_scalar_in_grad]
2022-05-30 14:56:04.882533: I ./tensorflow/core/graph/template_select_base.h:41] Fusion template[select_then_scalar_in_grad] match op[head/gradients/input_layer/unseq_input_layer/input_layer/brand_embedding/brand_embedding_weights_grad/Select_1][new_name:fused_op$
15_select_then_scalar_in_grad]             
2022-05-30 14:56:05.593580: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:05.598049: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:06.180909: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
INFO:tensorflow:Running local_init_op.                                                                                                                                                                                                                                 
2022-05-30 14:56:06.308357: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:06.309002: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:06.309340: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:06.336448: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
INFO:tensorflow:Done running local_init_op.                                                                                                                                                                                                                   
2022-05-30 14:56:06.649402: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                           
2022-05-30 14:56:06.768921: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:06.882066: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:07.804632: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:07.812211: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:08.707163: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                          
2022-05-30 14:56:10.078551: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                           
2022-05-30 14:56:10.496578: I tensorflow/core/common_runtime/tensorpool_allocator.cc:146] TensorPoolAllocator enabled
INFO:tensorflow:Prefetching was closed.                                                                                                                                                                                                                      
INFO:tensorflow:Prefetching was closed.                                                                                                                                                                                                                                 
INFO:tensorflow:Prefetching was closed.
INFO:tensorflow:Prefetching was closed.                                                                                                                                                                                                                                 
INFO:tensorflow:Prefetching was closed.
INFO:tensorflow:Prefetching was closed.
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, data.shape must start with partitions.shape, got data.shape = [272], partitions.shape = [512]
         [[{{node input_layer/unseq_input_layer/input_layer/price_embedding/DynamicPartition_1}}]]
ERROR:tensorflow:Prefetching was cancelled unexpectedly:  
                                                                                                                                                   
data.shape must start with partitions.shape, got data.shape = [272], partitions.shape = [512]
         [[{{node input_layer/unseq_input_layer/input_layer/price_embedding/DynamicPartition_1}}]]                                                        
Exception in thread PrefetchThread-PrefetchRunner-4:
Traceback (most recent call last):                                                                                                               
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()                                                                                                                                   
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)                                                                                            
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run
    run_fetch(*feed)                                                                                
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run
    self._call_tf_sessionrun(None, {}, [], target_list, None)                                                                          
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)                                                                                                                                    
tensorflow.python.framework.errors_impl.InvalidArgumentError: data.shape must start with partitions.shape, got data.shape = [272], partitions.shape = [512]
         [[{{node input_layer/unseq_input_layer/input_layer/price_embedding/DynamicPartition_1}}]]                                                 
                 
2022-05-30 14:56:10.811248: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200
2022-05-30 14:56:14.841705: I ./tensorflow/core/common_runtime/kernel_stat.h:74] User collect node stats, start_step is 100, stop_step is 200                                                                                                                   [0/4331]
Traceback (most recent call last):
  File "train.py", line 573, in eval
    [model.acc_op, model.auc_op, merged])
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 804, in run                                                                                                                         
    run_metadata=run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1309, in run                                                                                                                        
    run_metadata=run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1408, in run                                                                                                                        
    raise six.reraise(*original_exc_info)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1395, in run                                                                                                                        
    return self._sess.run(*args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1468, in run                                                                                                                        
    run_metadata=run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1226, in run                                                                                                                        
    return self._sess.run(*args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run                                                                                                                                
    run_metadata)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call                                                                                                                               
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Session was closed.
         [[node prefetch_2/TensorBufferTake (defined at /home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]                                                                                                

Original stack trace for 'prefetch_2/TensorBufferTake':
  File "train.py", line 907, in <module>
    main()
  File "train.py", line 653, in main
    next_element = tf.staged(next_element, num_threads=8, capacity=40)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch.py", line 140, in staged
    shared_threads=num_clients)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_tensor_buffer_ops.py", line 535, in tensor_buffer_take                                                                                                           
    shared_threads=shared_threads, name=name)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper                                                                                                              
    op_def=op_def)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func                                                                                                                              
    return func(*args, **kwargs)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op                                                                                                                               
    attrs, op_def, compute_device)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal                                                                                                                     
    op_def=op_def)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__                                                                                                                                
    self._traceback = tf_stack.extract_stack()


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 907, in <module>
    main()
  File "train.py", line 683, in main
    checkpoint_dir)
  File "train.py", line 576, in eval
    print("ACC = {}\nAUC = {}".format(eval_acc, eval_auc))
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 911, in __exit__                                                                                                                    
    self._close_internal(exception_type)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 949, in _close_internal                                                                                                             
    self._sess.close()
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1216, in close                                                                                                                      
    self._sess.close()
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1384, in close                                                                                                                      
    ignore_live_threads=True)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/training/coordinator.py", line 389, in join                                                                                                                              
    six.reraise(*self._exc_info_to_raise)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/ops/prefetch_runner.py", line 236, in run                                                                                                                                
    run_fetch(*feed)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1287, in _single_operation_run                                                                                                                  
    self._call_tf_sessionrun(None, {}, [], target_list, None)
  File "/home/duyi/miniconda3/envs/deeprec/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun                                                                                                                    
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: data.shape must start with partitions.shape, got data.shape = [272], partitions.shape = [512]                                                                                                            
         [[{{node input_layer/unseq_input_layer/input_layer/price_embedding/DynamicPartition_1}}]]