Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Quantization AssertionError "inputs len must equal with input_tensor" #34

Closed
dmsuehir opened this issue Oct 11, 2021 · 15 comments
Closed

Quantization AssertionError "inputs len must equal with input_tensor" #34

dmsuehir opened this issue Oct 11, 2021 · 15 comments

Comments

@dmsuehir
Copy link

I'm following the TensorFlow BERT MRPC example to run the neural compressor with a saved model that I exported after fine tuning BERT from the Intel Model Zoo using the IMDB movie review sentiment analysis dataset. The training task for this was "cola" instead of "mrpc", but I still used run_classifier.py to train the model.
I used the same Dataset class definition and collate_fn from the example and my yaml has:

model:
  name: bert
  framework: tensorflow
  inputs: input_file, batch_size
  outputs: Cast_147:0, loss/Mean:0, loss/Neg:0, loss/Cast:0

My python code looks like this:

from neural_compressor.metric import METRICS
class Accuracy(object):
    def __init__(self):
        self.metric = METRICS('tensorflow')['Accuracy']()
          
    # it's ugly that the label is in the iterator
    def update(self, preds, label):
        logits, labels = preds
        self.metric.update(logits, labels)

    def reset(self):
        self.metric.reset()

    def result(self):
        return self.metric.result()

# Using run_classifier from the Intel model zoo
from run_classifier import file_based_input_fn_builder

eval_file = os.path.join(output_dir, "eval.tf_record")
estimator_input_fn = file_based_input_fn_builder(
          input_file=eval_file,
          seq_length=max_seq_length,
          is_training=False,
          drop_remainder=False)

quantizer.model = common.Model(os.path.join(output_dir, "frozen"), input_fn=estimator_input_fn)
quantizer.calib_dataloader = common.DataLoader(dataset, collate_fn=collate_fn)
quantizer.eval_dataloader = common.DataLoader(dataset, collate_fn=collate_fn)

quantizer.metric = common.Metric(metric_cls=Accuracy, name="bert_metric")
q_model = quantizer()

This is failing with the following error:

2021-10-11 20:32:11 [INFO] Start to evaluate the TensorFlow model.
2021-10-11 20:32:11 [ERROR] Unexpected exception AssertionError('inputs len must equal with input_tensor') happened during tuning.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/neural_compressor/experimental/quantization.py", line 151, in execute
    self.strategy.traverse()
  File "/usr/local/lib/python3.8/dist-packages/neural_compressor/strategy/strategy.py", line 307, in traverse
    self.baseline = self._evaluate(self.model)
  File "/usr/local/lib/python3.8/dist-packages/neural_compressor/strategy/strategy.py", line 446, in _evaluate
    val = self.objective.evaluate(eval_func, model)
  File "/usr/local/lib/python3.8/dist-packages/neural_compressor/objective.py", line 213, in evaluate
    acc = eval_func(model)
  File "/usr/local/lib/python3.8/dist-packages/neural_compressor/utils/create_obj_from_config.py", line 132, in eval_func
    return adaptor.evaluate(model, dataloader, postprocess,
  File "/usr/local/lib/python3.8/dist-packages/neural_compressor/adaptor/tensorflow.py", line 291, in evaluate
    assert len(input_tensor) == len(inputs), \
AssertionError: inputs len must equal with input_tensor

The input_tensor for my model looks like this so I'm assuming that length is 4. What is other inputs that's being checked in the error above that has a different length than the input tensor?

[<tf.Tensor 'input_mask:0' shape=(None, 128) dtype=int32>,
 <tf.Tensor 'input_ids:0' shape=(None, 128) dtype=int32>,
 <tf.Tensor 'label_ids:0' shape=(None,) dtype=int32>,
 <tf.Tensor 'segment_ids:0' shape=(None, 128) dtype=int32>]

Any suggestions on how to resolve this issue?
@dmsuehir
Copy link
Author

dmsuehir commented Oct 16, 2021 •
edited

I think I figured out the error with the inputs. I think it might have had to do with the fact that I'm starting from a saved_model.pb which has inputs different than the example. I ended up using a Datasets class more similar the TFBERTDataSets class from examples/engine/nlp/bert_base_mrpc/utils.py.

Now, my output looks like this:

2021-10-15 23:12:44 [INFO] Pass Quantization elapsed time: 5634.64 ms
2021-10-15 23:16:35 [INFO] Pass QuantizedRNNConverter elapsed time: 33.85 ms
2021-10-15 23:16:37 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 80.63 ms
2021-10-15 23:16:38 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 32.54 ms
2021-10-15 23:16:38 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 26.37 ms
2021-10-15 23:16:38 [INFO] Pass MetaOpOptimizer elapsed time: 14.78 ms
2021-10-15 23:16:44 [INFO] Pass PostCseOptimizer elapsed time: 4606.32 ms
2021-10-15 23:16:46 [INFO] |********Mixed Precision Statistics*******|
2021-10-15 23:16:46 [INFO] +---------------+---------+-------+-------+
2021-10-15 23:16:46 [INFO] |    Op Type    |  Total  |  INT8 |  FP32 |
2021-10-15 23:16:46 [INFO] +---------------+---------+-------+-------+
2021-10-15 23:16:46 [INFO] |     MatMul    |    63   |   62  |   1   |
2021-10-15 23:16:46 [INFO] |   QuantizeV2  |    38   |   38  |   0   |
2021-10-15 23:16:46 [INFO] |      Cast     |    1    |   0   |   1   |
2021-10-15 23:16:46 [INFO] +---------------+---------+-------+-------+
2021-10-15 23:16:46 [INFO] Pass quantize model elapsed time: 275332.59 ms
2021-10-15 23:16:46 [INFO] Tune 1 result is: [accuracy: 1.0000, duration (seconds): 0.0000], Best tune result is: [accuracy: 1.0000, duration (seconds): 0.0000]
2021-10-15 23:16:46 [INFO] Save tuning history to /localdisk/dmsuehir/frameworks.ai.models.intel-models/docs/notebooks/bert_classifier_fine_tuning/nc_workspace/2021-10-15_23-11-36/./history.snapshot.
2021-10-15 23:16:46 [INFO] Specified timeout or max trials is reached! Found a quantized model which meet accuracy goal. Exit.
2021-10-15 23:16:46 [INFO] Save deploy yaml to /localdisk/dmsuehir/frameworks.ai.models.intel-models/docs/notebooks/bert_classifier_fine_tuning/nc_workspace/2021-10-15_23-11-36/deploy.yaml
2021-10-15 23:16:47 [WARNING] From /usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:200: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
2021-10-15 23:16:47 [INFO] No assets to save.
2021-10-15 23:16:47 [INFO] No assets to write.
2021-10-15 23:16:47 [INFO] SavedModel written to: /localdisk/dmsuehir/bert_fp32_training_output/quantized/saved_model.pb
2021-10-15 23:16:47 [INFO] Save quantized model to /localdisk/dmsuehir/bert_fp32_training_output/quantized.

Toward the middle of that output it's saying Tune 1 result is: [accuracy: 1.0000, duration (seconds): 0.0000], Best tune result is: [accuracy: 1.0000, duration (seconds): 0.0000]. I'm wondering if the accuracy 1 and duration of 0 seconds means that something went wrong? I see the saved_model.pb in my output directory. Does this look like a successful run or is there anything wrong?

@ftian1
Copy link
Contributor

ftian1 commented Oct 18, 2021

could you pls paste the whole log?

As you specified the metric and eval_dataloader, neural compressor should construct the evaluation process in internal. the duration should not be 0. but in your log, the duration is 0. looks like fake_eval func is invoked. if it's true, "Generate a fake evaluation function" debug message will be printed. did you see that?

@dmsuehir
Copy link
Author

@ftian1 Thanks, your response helped me to figure out a bit what was going on. Since I switched to follow the example from neural-compressor/examples/engine/nlp/bert_base_mrpc instead of the example from neural-compressor/examples/tensorflow/nlp/bert_base_mrpc, I commented out that line of code that was specifying the metric, since I didn't see the metric being set in the run_engine.py example. Is that because this example instead specifies the metric in the yaml instead of in the code? How do I know if I need to write code versus define things in the yaml?

I put back in the metric setting to my script, where I'm defining the Accuracy class like:

class Accuracy(object):
    def __init__(self):
        self.pred_list = []
        self.label_list = []
        self.num_samples = 0

    def update(self, preds, label):
        self.pred_list.extend(np.argmax(preds[0], axis=1))
        self.label_list.extend(list(label))
        self.num_samples += len(label)

    def reset(self):
        self.pred_list = []
        self.label_list = []
        self.num_samples = 0

    def result(self):
        correct_num = np.sum(np.array(self.pred_list) == np.array(self.label_list))
        return correct_num / self.num_samples

and then setting the metric like:

quantizer.metric = common.Metric(metric_cls=Accuracy)

Now, my full output looks like this:

2021-10-18 21:33:06.489220: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

User settings:

   KMP_AFFINITY=granularity=fine,verbose,compact,1,0
   KMP_BLOCKTIME=1
   KMP_DUPLICATE_LIB_OK=True
   KMP_INIT_AT_FORK=FALSE
   KMP_SETTINGS=1

Effective settings:

   KMP_ABORT_DELAY=0
   KMP_ADAPTIVE_LOCK_PROPS='1,1024'
   KMP_ALIGN_ALLOC=64
   KMP_ALL_THREADPRIVATE=448
   KMP_ATOMIC_MODE=2
   KMP_BLOCKTIME=1
   KMP_CPUINFO_FILE: value is not defined
   KMP_DETERMINISTIC_REDUCTION=false
   KMP_DEVICE_THREAD_LIMIT=2147483647
   KMP_DISP_NUM_BUFFERS=7
   KMP_DUPLICATE_LIB_OK=true
   KMP_ENABLE_TASK_THROTTLING=true
   KMP_FORCE_REDUCTION: value is not defined
   KMP_FOREIGN_THREADS_THREADPRIVATE=true
   KMP_FORKJOIN_BARRIER='2,2'
   KMP_FORKJOIN_BARRIER_PATTERN='hyper,hyper'
   KMP_GTID_MODE=3
   KMP_HANDLE_SIGNALS=false
   KMP_HOT_TEAMS_MAX_LEVEL=1
   KMP_HOT_TEAMS_MODE=0
   KMP_INIT_AT_FORK=true
   KMP_LIBRARY=throughput
   KMP_LOCK_KIND=queuing
   KMP_MALLOC_POOL_INCR=1M
   KMP_NUM_LOCKS_IN_BLOCK=1
   KMP_PLAIN_BARRIER='2,2'
   KMP_PLAIN_BARRIER_PATTERN='hyper,hyper'
   KMP_REDUCTION_BARRIER='1,1'
   KMP_REDUCTION_BARRIER_PATTERN='hyper,hyper'
   KMP_SCHEDULE='static,balanced;guided,iterative'
   KMP_SETTINGS=true
   KMP_SPIN_BACKOFF_PARAMS='4096,100'
   KMP_STACKOFFSET=64
   KMP_STACKPAD=0
   KMP_STACKSIZE=8M
   KMP_STORAGE_MAP=false
   KMP_TASKING=2
   KMP_TASKLOOP_MIN_TASKS=0
   KMP_TASK_STEALING_CONSTRAINT=1
   KMP_TEAMS_THREAD_LIMIT=112
   KMP_TOPOLOGY_METHOD=all
   KMP_USE_YIELD=1
   KMP_VERSION=false
   KMP_WARNINGS=true
   OMP_AFFINITY_FORMAT='OMP: pid %P tid %i thread %n bound to OS proc set {%A}'
   OMP_ALLOCATOR=omp_default_mem_alloc
   OMP_CANCELLATION=false
   OMP_DEFAULT_DEVICE=0
   OMP_DISPLAY_AFFINITY=false
   OMP_DISPLAY_ENV=false
   OMP_DYNAMIC=false
   OMP_MAX_ACTIVE_LEVELS=1
   OMP_MAX_TASK_PRIORITY=0
   OMP_NESTED: deprecated; max-active-levels-var=1
   OMP_NUM_THREADS: value is not defined
   OMP_PLACES: value is not defined
   OMP_PROC_BIND='intel'
   OMP_SCHEDULE='static'
   OMP_STACKSIZE=8M
   OMP_TARGET_OFFLOAD=DEFAULT
   OMP_THREAD_LIMIT=2147483647
   OMP_WAIT_POLICY=PASSIVE
   KMP_AFFINITY='verbose,warnings,respect,granularity=fine,compact,1,0'

2021-10-18 21:33:06.496606: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(30522, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/kernel:0' shape=(768, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/bias:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(30522, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/kernel:0' shape=(768, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/bias:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:06.840062: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-10-18 21:33:06.867602: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2500000000 Hz
2021-10-18 21:33:07 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(30522, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:07 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:07 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:07 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/kernel:0' shape=(768, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:07 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/bias:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(30522, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/kernel:0' shape=(768, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/bias:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2021-10-18 21:33:08 [WARNING] Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2021-10-18 21:33:08.641390: I tensorflow/core/grappler/devices.cc:78] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2021-10-18 21:33:08.641520: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-18 21:33:08.670966: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: graph_to_optimize
  function_optimizer: function_optimizer did nothing. time = 0.008ms.
  function_optimizer: function_optimizer did nothing. time = 0.002ms.

2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/word_embeddings:0' shape=(30522, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/token_type_embeddings:0' shape=(2, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/embeddings/position_embeddings:0' shape=(512, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/kernel:0' shape=(768, 768) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:08 [WARNING] Unable to create a python object for variable <tf.Variable 'bert/encoder/layer_0/attention/self/query/bias:0' shape=(768,) dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().
2021-10-18 21:33:20.867571: I tensorflow/core/grappler/devices.cc:78] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2021-10-18 21:33:20.868582: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-18 21:33:23.025591: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: tf_graph
  model_pruner: Graph size after: 1838 nodes (-201), 2103 edges (-201), time = 12.427ms.
  implementation_selector: Graph size after: 1838 nodes (0), 2103 edges (0), time = 4.367ms.
  function_optimizer: function_optimizer did nothing. time = 0.881ms.
  common_subgraph_elimination: Graph size after: 1066 nodes (-772), 1828 edges (-275), time = 355.321ms.
  constant_folding: Graph size after: 1084 nodes (18), 1864 edges (36), time = 193.268ms.
  shape_optimizer: shape_optimizer did nothing. time = 1.519ms.
  arithmetic_optimizer: Graph size after: 1084 nodes (0), 1864 edges (0), time = 17.43ms.
  layout: Graph size after: 1084 nodes (0), 1864 edges (0), time = 17.871ms.
  remapper: Graph size after: 1060 nodes (-24), 1840 edges (-24), time = 13.581ms.
  loop_optimizer: Graph size after: 1060 nodes (0), 1840 edges (0), time = 4.857ms.
  dependency_optimizer: Graph size after: 1010 nodes (-50), 1740 edges (-100), time = 9.544ms.
  memory_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 36.621ms.
  model_pruner: Graph size after: 1010 nodes (0), 1740 edges (0), time = 4.992ms.
  implementation_selector: Graph size after: 1010 nodes (0), 1740 edges (0), time = 2.712ms.
  function_optimizer: function_optimizer did nothing. time = 0.786ms.
  common_subgraph_elimination: Graph size after: 1010 nodes (0), 1740 edges (0), time = 344.281ms.
  constant_folding: Graph size after: 1010 nodes (0), 1740 edges (0), time = 72.343ms.
  shape_optimizer: shape_optimizer did nothing. time = 1.974ms.
  arithmetic_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 17.573ms.
  remapper: Graph size after: 1010 nodes (0), 1740 edges (0), time = 13.674ms.
  dependency_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 9.7ms.

2021-10-18 21:33:26 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:33:26 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:33:26 [INFO] ConvertLayoutOptimizer elapsed time: 0.6 ms
2021-10-18 21:33:27.370606: I tensorflow/core/grappler/devices.cc:78] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2021-10-18 21:33:27.371164: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-18 21:33:27.797627: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: graph_to_optimize
  model_pruner: Graph size after: 1010 nodes (0), 1740 edges (0), time = 4.466ms.
  shape_optimizer: shape_optimizer did nothing. time = 0.627ms.
  dependency_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 8.039ms.
  debug_stripper: Graph size after: 1010 nodes (0), 1740 edges (0), time = 1.74ms.
  loop_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 4.277ms.
  model_pruner: Graph size after: 1005 nodes (-5), 1735 edges (-5), time = 6.991ms.
  shape_optimizer: shape_optimizer did nothing. time = 0.679ms.
  dependency_optimizer: Graph size after: 1004 nodes (-1), 1733 edges (-2), time = 9.312ms.
  debug_stripper: debug_stripper did nothing. time = 0.861ms.

2021-10-18 21:33:28 [INFO] Pass GrapplerOptimizer elapsed time: 1786.18 ms
2021-10-18 21:33:28 [INFO] Pass SwitchOptimizer elapsed time: 21.32 ms
2021-10-18 21:33:28 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 15.84 ms
2021-10-18 21:33:28 [INFO] Pass SplitSharedInputOptimizer elapsed time: 28.18 ms
2021-10-18 21:33:28 [INFO] Pass GraphFoldConstantOptimizer elapsed time: 11.73 ms
2021-10-18 21:33:28 [INFO] Pass FuseColumnWiseMulOptimizer elapsed time: 21.29 ms
2021-10-18 21:33:28 [WARNING] From /usr/local/lib/python3.8/dist-packages/neural_compressor/adaptor/tf_utils/util.py:317: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2021-10-18 21:33:28 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 60.2 ms
2021-10-18 21:33:28 [INFO] Pass GraphCseOptimizer elapsed time: 20.78 ms
2021-10-18 21:33:28 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 18.95 ms
2021-10-18 21:33:28 [INFO] Pass UpdateEnterOptimizer elapsed time: 9.73 ms
2021-10-18 21:33:28 [INFO] Pass ConvertLeakyReluOptimizer elapsed time: 18.78 ms
2021-10-18 21:33:28 [INFO] Pass InjectDummyBiasAddOptimizer elapsed time: 21.04 ms
2021-10-18 21:33:28 [INFO] Pass ConvertAddToBiasAddOptimizer elapsed time: 19.52 ms
2021-10-18 21:33:28 [INFO] Pass FuseTransposeReshapeOptimizer elapsed time: 20.19 ms
2021-10-18 21:33:28 [INFO] Pass FuseConvWithMathOptimizer elapsed time: 19.24 ms
2021-10-18 21:33:30 [INFO] Pass Pre Optimization elapsed time: 4323.41 ms
2021-10-18 21:33:32 [INFO] Get FP32 model baseline.
2021-10-18 21:33:33 [INFO] Start to evaluate the TensorFlow model.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-111
OMP: Info #156: KMP_AFFINITY: 112 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 28 cores/pkg x 2 threads/core (56 total cores)
OMP: Info #213: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 56 maps to package 0 core 0 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 57 maps to package 0 core 1 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 58 maps to package 0 core 2 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 59 maps to package 0 core 3 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 4 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 60 maps to package 0 core 4 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 5 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 61 maps to package 0 core 5 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 6 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 62 maps to package 0 core 6 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 8 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 63 maps to package 0 core 8 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 9 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 64 maps to package 0 core 9 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 10 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 65 maps to package 0 core 10 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 11 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 66 maps to package 0 core 11 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 12 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 67 maps to package 0 core 12 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to package 0 core 13 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 68 maps to package 0 core 13 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to package 0 core 14 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 69 maps to package 0 core 14 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to package 0 core 16 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 70 maps to package 0 core 16 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to package 0 core 17 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 71 maps to package 0 core 17 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 16 maps to package 0 core 18 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 72 maps to package 0 core 18 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 17 maps to package 0 core 19 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 73 maps to package 0 core 19 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 18 maps to package 0 core 20 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 74 maps to package 0 core 20 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 19 maps to package 0 core 21 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 75 maps to package 0 core 21 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 20 maps to package 0 core 22 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 76 maps to package 0 core 22 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 21 maps to package 0 core 24 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 77 maps to package 0 core 24 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 22 maps to package 0 core 25 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 78 maps to package 0 core 25 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 23 maps to package 0 core 26 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 79 maps to package 0 core 26 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 24 maps to package 0 core 27 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 80 maps to package 0 core 27 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 25 maps to package 0 core 28 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 81 maps to package 0 core 28 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 26 maps to package 0 core 29 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 82 maps to package 0 core 29 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 27 maps to package 0 core 30 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 83 maps to package 0 core 30 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 28 maps to package 1 core 0 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 84 maps to package 1 core 0 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 29 maps to package 1 core 1 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 85 maps to package 1 core 1 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 30 maps to package 1 core 2 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 86 maps to package 1 core 2 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 31 maps to package 1 core 3 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 87 maps to package 1 core 3 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 32 maps to package 1 core 4 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 88 maps to package 1 core 4 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 33 maps to package 1 core 5 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 89 maps to package 1 core 5 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 34 maps to package 1 core 6 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 90 maps to package 1 core 6 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 35 maps to package 1 core 8 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 91 maps to package 1 core 8 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 36 maps to package 1 core 9 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 92 maps to package 1 core 9 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 37 maps to package 1 core 10 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 93 maps to package 1 core 10 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 38 maps to package 1 core 11 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 94 maps to package 1 core 11 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 39 maps to package 1 core 12 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 95 maps to package 1 core 12 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 40 maps to package 1 core 13 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 96 maps to package 1 core 13 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 41 maps to package 1 core 14 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 97 maps to package 1 core 14 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 42 maps to package 1 core 16 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 98 maps to package 1 core 16 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 43 maps to package 1 core 17 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 99 maps to package 1 core 17 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 44 maps to package 1 core 18 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 100 maps to package 1 core 18 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 45 maps to package 1 core 19 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 101 maps to package 1 core 19 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 46 maps to package 1 core 20 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 102 maps to package 1 core 20 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 47 maps to package 1 core 21 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 103 maps to package 1 core 21 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 48 maps to package 1 core 22 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 104 maps to package 1 core 22 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 49 maps to package 1 core 24 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 105 maps to package 1 core 24 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 50 maps to package 1 core 25 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 106 maps to package 1 core 25 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 51 maps to package 1 core 26 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 107 maps to package 1 core 26 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 52 maps to package 1 core 27 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 108 maps to package 1 core 27 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 53 maps to package 1 core 28 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 109 maps to package 1 core 28 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 54 maps to package 1 core 29 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 110 maps to package 1 core 29 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 55 maps to package 1 core 30 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 111 maps to package 1 core 30 thread 1 
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20563 thread 0 bound to OS proc set 0
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20563 thread 1 bound to OS proc set 1
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20565 thread 2 bound to OS proc set 2
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20566 thread 3 bound to OS proc set 3
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20567 thread 4 bound to OS proc set 4
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20569 thread 6 bound to OS proc set 6
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20568 thread 5 bound to OS proc set 5
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20570 thread 7 bound to OS proc set 7
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20571 thread 8 bound to OS proc set 8
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20572 thread 9 bound to OS proc set 9
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20573 thread 10 bound to OS proc set 10
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20575 thread 12 bound to OS proc set 12
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20574 thread 11 bound to OS proc set 11
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20576 thread 13 bound to OS proc set 13
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20577 thread 14 bound to OS proc set 14
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20578 thread 15 bound to OS proc set 15
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20579 thread 16 bound to OS proc set 16
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20580 thread 17 bound to OS proc set 17
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20581 thread 18 bound to OS proc set 18
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20582 thread 19 bound to OS proc set 19
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20583 thread 20 bound to OS proc set 20
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20584 thread 21 bound to OS proc set 21
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20585 thread 22 bound to OS proc set 22
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20586 thread 23 bound to OS proc set 23
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20587 thread 24 bound to OS proc set 24
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20588 thread 25 bound to OS proc set 25
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20589 thread 26 bound to OS proc set 26
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20590 thread 27 bound to OS proc set 27
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20591 thread 28 bound to OS proc set 28
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20592 thread 29 bound to OS proc set 29
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20593 thread 30 bound to OS proc set 30
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20594 thread 31 bound to OS proc set 31
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20595 thread 32 bound to OS proc set 32
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20596 thread 33 bound to OS proc set 33
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20597 thread 34 bound to OS proc set 34
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20598 thread 35 bound to OS proc set 35
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20599 thread 36 bound to OS proc set 36
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20600 thread 37 bound to OS proc set 37
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20601 thread 38 bound to OS proc set 38
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20602 thread 39 bound to OS proc set 39
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20603 thread 40 bound to OS proc set 40
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20604 thread 41 bound to OS proc set 41
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20605 thread 42 bound to OS proc set 42
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20606 thread 43 bound to OS proc set 43
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20607 thread 44 bound to OS proc set 44
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20608 thread 45 bound to OS proc set 45
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20609 thread 46 bound to OS proc set 46
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20611 thread 48 bound to OS proc set 48
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20610 thread 47 bound to OS proc set 47
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20612 thread 49 bound to OS proc set 49
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20613 thread 50 bound to OS proc set 50
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20614 thread 51 bound to OS proc set 51
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20615 thread 52 bound to OS proc set 52
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20616 thread 53 bound to OS proc set 53
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20617 thread 54 bound to OS proc set 54
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20618 thread 55 bound to OS proc set 55
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20619 thread 56 bound to OS proc set 56
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20620 thread 57 bound to OS proc set 57
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20621 thread 58 bound to OS proc set 58
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20622 thread 59 bound to OS proc set 59
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20623 thread 60 bound to OS proc set 60
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20624 thread 61 bound to OS proc set 61
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20625 thread 62 bound to OS proc set 62
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20626 thread 63 bound to OS proc set 63
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20627 thread 64 bound to OS proc set 64
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20628 thread 65 bound to OS proc set 65
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20629 thread 66 bound to OS proc set 66
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20631 thread 68 bound to OS proc set 68
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20630 thread 67 bound to OS proc set 67
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20632 thread 69 bound to OS proc set 69
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20633 thread 70 bound to OS proc set 70
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20634 thread 71 bound to OS proc set 71
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20635 thread 72 bound to OS proc set 72
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20636 thread 73 bound to OS proc set 73
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20637 thread 74 bound to OS proc set 74
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20639 thread 76 bound to OS proc set 76
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20638 thread 75 bound to OS proc set 75
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20640 thread 77 bound to OS proc set 77
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20642 thread 79 bound to OS proc set 79
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20641 thread 78 bound to OS proc set 78
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20644 thread 81 bound to OS proc set 81
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20643 thread 80 bound to OS proc set 80
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20646 thread 83 bound to OS proc set 83
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20645 thread 82 bound to OS proc set 82
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20647 thread 84 bound to OS proc set 84
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20648 thread 85 bound to OS proc set 85
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20650 thread 87 bound to OS proc set 87
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20649 thread 86 bound to OS proc set 86
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20651 thread 88 bound to OS proc set 88
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20652 thread 89 bound to OS proc set 89
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20653 thread 90 bound to OS proc set 90
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20655 thread 92 bound to OS proc set 92
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20654 thread 91 bound to OS proc set 91
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20656 thread 93 bound to OS proc set 93
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20657 thread 94 bound to OS proc set 94
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20658 thread 95 bound to OS proc set 95
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20659 thread 96 bound to OS proc set 96
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20660 thread 97 bound to OS proc set 97
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20661 thread 98 bound to OS proc set 98
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20662 thread 99 bound to OS proc set 99
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20663 thread 100 bound to OS proc set 100
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20664 thread 101 bound to OS proc set 101
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20665 thread 102 bound to OS proc set 102
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20666 thread 103 bound to OS proc set 103
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20667 thread 104 bound to OS proc set 104
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20668 thread 105 bound to OS proc set 105
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20669 thread 106 bound to OS proc set 106
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20670 thread 107 bound to OS proc set 107
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20671 thread 108 bound to OS proc set 108
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20673 thread 110 bound to OS proc set 110
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20672 thread 109 bound to OS proc set 109
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20674 thread 111 bound to OS proc set 111
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 20675 thread 112 bound to OS proc set 0
2021-10-18 21:41:14 [INFO] Save tuning history to /localdisk/dmsuehir/frameworks.ai.models.intel-models/docs/notebooks/bert_classifier_fine_tuning/nc_workspace/2021-10-18_21-32-58/./history.snapshot.
2021-10-18 21:41:14 [INFO] FP32 baseline is: [accuracy: 0.7110, duration (seconds): 462.0222]
2021-10-18 21:41:14 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:14 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:14 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:14 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:17 [WARNING] Found possible input node names: ['label_ids', 'input_mask', 'input_ids', 'segment_ids'], output node names: ['loss/Softmax', 'loss/Mean'].
2021-10-18 21:41:17 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:18 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:18 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:18 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2021-10-18 21:41:20 [WARNING] Found possible input node names: ['label_ids', 'input_mask', 'input_ids', 'segment_ids'], output node names: ['loss/Softmax', 'loss/Mean'].
2021-10-18 21:41:24 [WARNING] Issue encountered when serializing variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2021-10-18 21:41:24 [WARNING] Issue encountered when serializing trainable_variables.
Type is unsupported, or the types of the items don't match field type in CollectionDef. Note this is a warning and probably safe to ignore.
to_proto not supported in EAGER mode.
2021-10-18 21:41:24.636831: I tensorflow/core/grappler/devices.cc:78] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2021-10-18 21:41:24.636961: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-18 21:41:24.666438: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: graph_to_optimize
  function_optimizer: function_optimizer did nothing. time = 0.007ms.
  function_optimizer: function_optimizer did nothing. time = 0.002ms.

2021-10-18 21:41:35.914857: I tensorflow/core/grappler/devices.cc:78] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2021-10-18 21:41:35.916193: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-18 21:41:38.084844: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: tf_graph
  model_pruner: Graph size after: 1838 nodes (-201), 2103 edges (-201), time = 11.428ms.
  implementation_selector: Graph size after: 1838 nodes (0), 2103 edges (0), time = 4.487ms.
  function_optimizer: function_optimizer did nothing. time = 0.892ms.
  common_subgraph_elimination: Graph size after: 1066 nodes (-772), 1828 edges (-275), time = 363.568ms.
  constant_folding: Graph size after: 1084 nodes (18), 1864 edges (36), time = 199.097ms.
  shape_optimizer: shape_optimizer did nothing. time = 1.543ms.
  arithmetic_optimizer: Graph size after: 1084 nodes (0), 1864 edges (0), time = 17.516ms.
  layout: Graph size after: 1084 nodes (0), 1864 edges (0), time = 18.154ms.
  remapper: Graph size after: 1060 nodes (-24), 1840 edges (-24), time = 13.961ms.
  loop_optimizer: Graph size after: 1060 nodes (0), 1840 edges (0), time = 4.954ms.
  dependency_optimizer: Graph size after: 1010 nodes (-50), 1740 edges (-100), time = 9.725ms.
  memory_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 36.34ms.
  model_pruner: Graph size after: 1010 nodes (0), 1740 edges (0), time = 5.089ms.
  implementation_selector: Graph size after: 1010 nodes (0), 1740 edges (0), time = 2.764ms.
  function_optimizer: function_optimizer did nothing. time = 0.81ms.
  common_subgraph_elimination: Graph size after: 1010 nodes (0), 1740 edges (0), time = 346.283ms.
  constant_folding: Graph size after: 1010 nodes (0), 1740 edges (0), time = 73.912ms.
  shape_optimizer: shape_optimizer did nothing. time = 1.992ms.
  arithmetic_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 17.818ms.
  remapper: Graph size after: 1010 nodes (0), 1740 edges (0), time = 13.989ms.
  dependency_optimizer: Graph size after: 1010 nodes (0), 1740 edges (0), time = 9.905ms.

2021-10-18 21:41:41.942240: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-10-18 21:41:41 [WARNING] From /usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:206: quantize_v2 (from tensorflow.python.ops.array_ops) is deprecated and will be removed after 2017-10-25.
Instructions for updating:
`tf.quantize_v2` is deprecated, please use `tf.quantization.quantize` instead.
2021-10-18 21:41:47 [INFO] Pass Quantization elapsed time: 5593.17 ms
2021-10-18 21:45:38 [INFO] Pass QuantizedRNNConverter elapsed time: 39.7 ms
2021-10-18 21:45:39 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 80.33 ms
2021-10-18 21:45:39 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 32.24 ms
2021-10-18 21:45:39 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 27.39 ms
2021-10-18 21:45:40 [INFO] Pass MetaOpOptimizer elapsed time: 15.34 ms
2021-10-18 21:45:46 [INFO] Pass PostCseOptimizer elapsed time: 4423.77 ms
2021-10-18 21:45:47 [INFO] |********Mixed Precision Statistics*******|
2021-10-18 21:45:47 [INFO] +---------------+---------+-------+-------+
2021-10-18 21:45:47 [INFO] |    Op Type    |  Total  |  INT8 |  FP32 |
2021-10-18 21:45:47 [INFO] +---------------+---------+-------+-------+
2021-10-18 21:45:47 [INFO] |     MatMul    |    63   |   62  |   1   |
2021-10-18 21:45:47 [INFO] |   QuantizeV2  |    38   |   38  |   0   |
2021-10-18 21:45:47 [INFO] |      Cast     |    1    |   0   |   1   |
2021-10-18 21:45:47 [INFO] +---------------+---------+-------+-------+
2021-10-18 21:45:47 [INFO] Pass quantize model elapsed time: 272596.25 ms
2021-10-18 21:45:47 [INFO] Start to evaluate the TensorFlow model.
OMP: Info #249: KMP_AFFINITY: pid 19955 tid 21038 thread 113 bound to OS proc set 1
2021-10-18 21:54:33 [INFO] Tune 1 result is: [accuracy: 0.7240, duration (seconds): 526.5596], Best tune result is: [accuracy: 0.7240, duration (seconds): 526.5596]
2021-10-18 21:54:33 [INFO] Save tuning history to /localdisk/dmsuehir/frameworks.ai.models.intel-models/docs/notebooks/bert_classifier_fine_tuning/nc_workspace/2021-10-18_21-32-58/./history.snapshot.
2021-10-18 21:54:33 [INFO] Specified timeout or max trials is reached! Found a quantized model which meet accuracy goal. Exit.
2021-10-18 21:54:33 [INFO] Save deploy yaml to /localdisk/dmsuehir/frameworks.ai.models.intel-models/docs/notebooks/bert_classifier_fine_tuning/nc_workspace/2021-10-18_21-32-58/deploy.yaml
2021-10-18 21:54:34 [WARNING] From /usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:200: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
2021-10-18 21:54:34 [INFO] No assets to save.
2021-10-18 21:54:34 [INFO] No assets to write.
2021-10-18 21:54:35 [INFO] SavedModel written to: /localdisk/dmsuehir/bert_fp32_training_output/quantized/saved_model.pb
2021-10-18 21:54:35 [INFO] Save quantized model to /localdisk/dmsuehir/bert_fp32_training_output/quantized.

Does this look correct? I did see some warnings in the log, so not sure if there are other things that I'm doing wrong.

deb-intel pushed a commit to deb-intel/lp-opt-tool that referenced this issue Nov 4, 2021
@bmyrcha
ILITV-629 Add install_requirements.sh script to pytorch huggingface m…
d875f11 
…odels (intel#34)
@GuoGuiRong
Copy link

GuoGuiRong commented Mar 23, 2022 •
edited

I have the same error? but I don't know how to reslove

(tf2) [root@VM-117-232-centos /data/home/guirongguo/neural-compressor/examples/tensorflow/image_recognition/SavedModel/quantization/ptq_2]# bash run_tuning.sh --input_model=origin_model.pb --output_model=origin_model_tune.pb --dataset_location=./
+ main --input_model=origin_model.pb --output_model=origin_model_tune.pb --dataset_location=./
+ init_params --input_model=origin_model.pb --output_model=origin_model_tune.pb --dataset_location=./
+ for var in '"$@"'
+ case $var in
++ echo --input_model=origin_model.pb
++ cut -f2 -d=
+ input_model=origin_model.pb
+ for var in '"$@"'
+ case $var in
++ echo --output_model=origin_model_tune.pb
++ cut -f2 -d=
+ output_model=origin_model_tune.pb
+ for var in '"$@"'
+ case $var in
++ echo --dataset_location=./
++ cut -f2 -d=
+ dataset_location=./
+ run_tuning
+ python inference.py --input_graph origin_model.pb --evaluation_data_location .//eval_processed_data.tfrecords --calibration_data_location .//train_processed_data.tfrecords --accuracy_only --batch_size 100 --output_graph origin_model_tune.pb --config origin_model.yaml --tune
2022-03-23 17:32:47 [WARNING] Force convert framework model to neural_compressor model.
2022-03-23 17:32:47 [WARNING] Output tensor names should not be empty.
2022-03-23 17:32:47 [WARNING] Input tensor names should not be empty.
WARNING:tensorflow:From inference.py:326: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
2022-03-23 17:32:47 [WARNING] From inference.py:326: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and: 
`tf.data.TFRecordDataset(path)`
batch size is 100,2000 iteration
2022-03-23 17:32:47.643748: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-03-23 17:32:47 [WARNING] Found possible input node names: ['dense_input', 'sparse_ids_input', 'sparse_wgt_input', 'seq_50_input'], output node names: ['dense/Sigmoid'].
2022-03-23 17:32:47 [INFO] ConvertLayoutOptimizer elapsed time: 0.49 ms
2022-03-23 17:32:48.100305: I tensorflow/core/grappler/devices.cc:75] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2022-03-23 17:32:48.100779: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-03-23 17:32:48.202772: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1149] Optimization results for grappler item: graph_to_optimize
  model_pruner: Graph size after: 573 nodes (-90), 675 edges (-90), time = 3.993ms.
  shape_optimizer: shape_optimizer did nothing. time = 0.494ms.
  dependency_optimizer: Graph size after: 573 nodes (0), 675 edges (0), time = 3.865ms.
  debug_stripper: debug_stripper did nothing. time = 0.399ms.
  loop_optimizer: Graph size after: 573 nodes (0), 675 edges (0), time = 1.866ms.
  model_pruner: Graph size after: 573 nodes (0), 675 edges (0), time = 2.296ms.
  shape_optimizer: shape_optimizer did nothing. time = 0.398ms.
  dependency_optimizer: Graph size after: 573 nodes (0), 675 edges (0), time = 3.795ms.
  debug_stripper: debug_stripper did nothing. time = 0.408ms.

2022-03-23 17:32:48 [INFO] Pass GrapplerOptimizer elapsed time: 349.62 ms
2022-03-23 17:32:48 [INFO] Pass SwitchOptimizer elapsed time: 7.0 ms
2022-03-23 17:32:48 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 6.21 ms
2022-03-23 17:32:48 [INFO] Pass SplitSharedInputOptimizer elapsed time: 7.8 ms
2022-03-23 17:32:48 [INFO] Pass GraphFoldConstantOptimizer elapsed time: 8.53 ms
2022-03-23 17:32:48 [INFO] Pass FuseColumnWiseMulOptimizer elapsed time: 6.59 ms
WARNING:tensorflow:From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/adaptor/tf_utils/util.py:318: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2022-03-23 17:32:48 [WARNING] From /root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/adaptor/tf_utils/util.py:318: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2022-03-23 17:32:48 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 19.01 ms
2022-03-23 17:32:48 [INFO] Pass GraphCseOptimizer elapsed time: 6.49 ms
2022-03-23 17:32:48 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 5.71 ms
2022-03-23 17:32:48 [INFO] Pass UpdateEnterOptimizer elapsed time: 3.16 ms
2022-03-23 17:32:48 [INFO] Pass ConvertLeakyReluOptimizer elapsed time: 5.85 ms
2022-03-23 17:32:48 [INFO] Pass ConvertAddToBiasAddOptimizer elapsed time: 6.02 ms
2022-03-23 17:32:48 [INFO] Pass FuseTransposeReshapeOptimizer elapsed time: 6.02 ms
2022-03-23 17:32:48 [INFO] Pass FuseConvWithMathOptimizer elapsed time: 6.09 ms
2022-03-23 17:32:48 [INFO] Pass ExpandDimsOptimizer elapsed time: 8.47 ms
2022-03-23 17:32:48 [INFO] Pass InjectDummyBiasAddOptimizer elapsed time: 12.61 ms
2022-03-23 17:32:48 [INFO] Pass MoveSqueezeAfterReluOptimizer elapsed time: 6.11 ms
2022-03-23 17:32:48 [INFO] Pass Pre Optimization elapsed time: 975.82 ms
2022-03-23 17:32:49 [INFO] Get FP32 model baseline.
Run inference
Set inter and intra for mkl: 
intra_op_parallelism_threads =  0
inter_op_parallelism_threads =  0
2022-03-23 17:32:49.599309: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2022-03-23 17:32:50.412966: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
--------------------------------------------------
Total test records: 20000
Number of batches: 200
Batch size = 100
Latency: 27.968 ms
Throughput: 3575.517 records/sec
Accuracy: 98.87500
--------------------------------------------------
2022-03-23 17:32:56 [INFO] Save tuning history to /data/home/guirongguo/neural-compressor/examples/tensorflow/image_recognition/SavedModel/quantization/ptq_2/nc_workspace/2022-03-23_17-32-46/./history.snapshot.
2022-03-23 17:32:56 [INFO] FP32 baseline is: [Accuracy: 98.8750, Duration (seconds): 6.6118]
2022-03-23 17:32:56 [CRITICAL] Please set environment variable TF_ENABLE_ONEDNN_OPTS=1 when Tensorflow 2.6.x installed.
2022-03-23 17:32:56 [WARNING] Found possible input node names: ['dense_input', 'seq_50_input', 'sparse_ids_input', 'sparse_wgt_input'], output node names: ['dense/Sigmoid'].
2022-03-23 17:32:56 [WARNING] Found possible input node names: ['dense_input', 'seq_50_input', 'sparse_ids_input', 'sparse_wgt_input'], output node names: ['dense/Sigmoid'].
2022-03-23 17:32:58 [INFO] Pass Quantization elapsed time: 843.25 ms
Traceback (most recent call last):
  File "/root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 547, in quantize
    self._inference(self._sampling_model)
  File "/root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 155, in _inference
    'inputs len must equal with input_tensor'
AssertionError: inputs len must equal with input_tensor
2022-03-23 17:33:00 [ERROR] Fail to quantize graph due to inputs len must equal with input_tensor.
2022-03-23 17:33:00 [ERROR] Unexpected exception AttributeError("'NoneType' object has no attribute 'graph_def'") happened during tuning.
Traceback (most recent call last):
  File "/root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/experimental/quantization.py", line 151, in execute
    self.strategy.traverse()
  File "/root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/strategy/strategy.py", line 347, in traverse
    tune_cfg, self.model, self.calib_dataloader, self.q_func)
  File "/root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/utils/utility.py", line 240, in fi
    res = func(*args, **kwargs)
  File "/root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/adaptor/tensorflow.py", line 539, in quantize
    data_loader=data_loader).convert()
  File "/root/anaconda3/envs/tf2/lib/python3.7/site-packages/neural_compressor/adaptor/tf_utils/graph_converter.py", line 249, in convert
    post_cse_graph_def = PostCseOptimizer(model.graph_def).do_transformation()
AttributeError: 'NoneType' object has no attribute 'graph_def'
2022-03-23 17:33:00 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit.
Traceback (most recent call last):
  File "inference.py", line 345, in <module>
    evaluate_opt_graph.run()
  File "inference.py", line 309, in run
    q_model.save(self.args.output_graph)
AttributeError: 'NoneType' object has no attribute 'save'

@hshen14
Copy link
Collaborator

hshen14 commented Jul 2, 2022

cc @chensuyue

@chensuyue
Copy link
Contributor

@GuoGuiRong Sorry for the late reply, may I know if the issue still there?

@asirvaiy
Copy link
Contributor

Hi @chensuyue, I am seeing this same error for MaskRCNN_resnet50. I am providing publicly available mask_rcnn_coco weights(frozen .pb file) as input_model.

@chensuyue
Copy link
Contributor

Hi @chensuyue, I am seeing this same error for MaskRCNN_resnet50. I am providing publicly available mask_rcnn_coco weights(frozen .pb file) as input_model.

Thanks for your feedback, could you provide the public link of the MaskRCNN_resnet50 model?

@asirvaiy
Copy link
Contributor

asirvaiy commented Jul 13, 2022 •
edited

Hi @chensuyue, I am seeing this same error for MaskRCNN_resnet50. I am providing publicly available mask_rcnn_coco weights(frozen .pb file) as input_model.

Thanks for your feedback, could you provide the public link of the MaskRCNN_resnet50 model?

https://github.com/matterport/Mask_RCNN/releases/tag/v2.0 Here is the model file in H5 format. I converted it to frozen form(pb).
https://github.com/matterport/Mask_RCNN is the main repo for MaskRCNN TF model.

@chensuyue
Copy link
Contributor

@zehao-intel pls check.

@zehao-intel
Copy link
Collaborator

Yes. I will track it.

@zehao-intel
Copy link
Collaborator

Hi @chensuyue, I am seeing this same error for MaskRCNN_resnet50. I am providing publicly available mask_rcnn_coco weights(frozen .pb file) as input_model.

Hi @asirvaiy, the h5 file in the link you provided only contains weights. You need to use the main repo for MaskRCNN TF model to load this weight file to make it a complete model. Did you try this?
And if you have done so, could you provide the whole working flow regarding how did you generate the pb file?
Additionally, could you please paste the complete log for this error?

@asirvaiy
Copy link
Contributor

Hi @chensuyue, I am seeing this same error for MaskRCNN_resnet50. I am providing publicly available mask_rcnn_coco weights(frozen .pb file) as input_model.

Hi @asirvaiy, the h5 file in the link you provided only contains weights. You need to use the main repo for MaskRCNN TF model to load this weight file to make it a complete model. Did you try this? And if you have done so, could you provide the whole working flow regarding how did you generate the pb file? Additionally, could you please paste the complete log for this error?

Hi @zehao-intel, I have used this https://github.com/Amith4504/Frozen-Graph-Inference-MaskRCNN to convert .h5 to .pb
I can also send you the .pb file. Tried to use these instructions: https://github.com/intel/neural-compressor/tree/master/examples/tensorflow/object_detection/tensorflow_models/quantization/ptq

@zehao-intel
Copy link
Collaborator

Hi @asirvaiy.
Thanks for your reply!
I will check it ASAP.

@zehao-intel
Copy link
Collaborator

This "inputs len must equal with input_tensor" error occurs when the number of input data is not equal to the number of input tensors.
In this case, asirvairy's MaskRCNN pb file has three input tensors: image_image, input_image_meta, input_anchors
Hence, there should be three inputs to be feed into these tensors by constructing a feed_dict.

I have used correct input settings and preprocessing from the MaskRCNN repo to feed correct data. The model can be quantized with real data now.

VincyZhang pushed a commit that referenced this issue Feb 12, 2023
@PenghuiCheng
fixed nightly test error for examples (#34)
18a5835 
* fixed nightly test error for examples

* change the folder for qat and ptq examples
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dmsuehir @ftian1 @GuoGuiRong @hshen14 @chensuyue @asirvaiy @zehao-intel