custom data loader and metric not working for quantization of TF saved model

[mesolmaz]

I am using helloworld tf_example 2 as baseline to create custom dataset loader from local image folders, and the same classification accuracy as metric. But during quantization, I see a message saying "[INFO] Neither evaluation function nor metric is defined. Generate a quantized model with default quantization configuration. [INFO] Generate a fake evaluation function."
Here is the code snippet for Dataset class constructor. MyMetric class definition and member functions are the same as the example.

class Dataset(object):
  def __init__(self, image_dir):
      
    #   (train_images, train_labels), (test_images,
    #              test_labels) = keras.datasets.fashion_mnist.load_data()
      datagen = keras.preprocessing.image.ImageDataGenerator()
      train_generator = datagen.flow_from_directory(
        image_dir,
        target_size=(300, 300),
        color_mode ='grayscale',
        batch_size=32,
        class_mode='categorical'
      )
      x=np.concatenate([train_generator.next()[0] for i in range(train_generator.__len__())])
      y=np.concatenate([train_generator.next()[1] for i in range(train_generator.__len__())])

      self.test_images = x / 255.0
      self.labels = y
      pass

and here is the output during quantization.

Found 10731 images belonging to 3 classes.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 04:54:53 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 04:54:54.146247: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-02-23 04:54:54.148159: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 04:56:00 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 04:57:14 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 04:58:12.338695: I tensorflow/core/grappler/devices.cc:75] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2022-02-23 04:58:12.338923: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-02-23 04:58:12.362167: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1149] Optimization results for grappler item: graph_to_optimize
function_optimizer: function_optimizer did nothing. time = 0.021ms.
function_optimizer: function_optimizer did nothing. time = 0.001ms.

2022-02-23 04:58:17.228588: I tensorflow/core/grappler/devices.cc:75] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2022-02-23 04:58:17.228792: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-02-23 04:58:17.444025: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1149] Optimization results for grappler item: tf_graph
constant_folding: Graph size after: 1118 nodes (-612), 1787 edges (-612), time = 80.122ms.
constant_folding: Graph size after: 1118 nodes (0), 1787 edges (0), time = 64.691ms.

WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 04:58:18 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 04:59:14 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:00:09 [INFO] ConvertLayoutOptimizer elapsed time: 2.27 ms
2022-02-23 05:00:11.809891: I tensorflow/core/grappler/devices.cc:75] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2022-02-23 05:00:11.810752: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-02-23 05:00:11.940530: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1149] Optimization results for grappler item: graph_to_optimize
model_pruner: Graph size after: 1115 nodes (-3), 1784 edges (-3), time = 18.656ms.
shape_optimizer: shape_optimizer did nothing. time = 1.215ms.
dependency_optimizer: Graph size after: 1114 nodes (-1), 1171 edges (-613), time = 18.54ms.
debug_stripper: debug_stripper did nothing. time = 1.425ms.
loop_optimizer: Graph size after: 1114 nodes (0), 1171 edges (0), time = 9.038ms.
model_pruner: Graph size after: 1114 nodes (0), 1171 edges (0), time = 10.288ms.
shape_optimizer: shape_optimizer did nothing. time = 1.118ms.
dependency_optimizer: Graph size after: 1114 nodes (0), 1171 edges (0), time = 13.384ms.
debug_stripper: debug_stripper did nothing. time = 1.266ms.

2022-02-23 05:00:11 [INFO] Pass GrapplerOptimizer elapsed time: 1987.65 ms
2022-02-23 05:00:12 [INFO] Pass SwitchOptimizer elapsed time: 242.62 ms
2022-02-23 05:00:12 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 244.49 ms
2022-02-23 05:00:12 [INFO] Pass SplitSharedInputOptimizer elapsed time: 31.42 ms
2022-02-23 05:00:12 [INFO] Pass GraphFoldConstantOptimizer elapsed time: 220.24 ms
2022-02-23 05:00:12 [INFO] Pass FuseColumnWiseMulOptimizer elapsed time: 243.88 ms
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/neural_compressor/adaptor/tf_utils/util.py:317: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2022-02-23 05:00:13 [WARNING] From /usr/local/lib/python3.8/dist-packages/neural_compressor/adaptor/tf_utils/util.py:317: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2022-02-23 05:00:13 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 912.96 ms
2022-02-23 05:00:14 [INFO] Pass GraphCseOptimizer elapsed time: 249.38 ms
2022-02-23 05:00:14 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 781.72 ms
2022-02-23 05:00:15 [INFO] Pass UpdateEnterOptimizer elapsed time: 105.81 ms
2022-02-23 05:00:15 [INFO] Pass ConvertLeakyReluOptimizer elapsed time: 126.88 ms
2022-02-23 05:00:15 [INFO] Pass ConvertAddToBiasAddOptimizer elapsed time: 128.69 ms
2022-02-23 05:00:15 [INFO] Pass FuseTransposeReshapeOptimizer elapsed time: 131.34 ms
2022-02-23 05:00:15 [INFO] Pass FuseConvWithMathOptimizer elapsed time: 129.63 ms
2022-02-23 05:00:15 [INFO] Pass ExpandDimsOptimizer elapsed time: 125.77 ms
2022-02-23 05:00:15 [INFO] Pass InjectDummyBiasAddOptimizer elapsed time: 171.51 ms
2022-02-23 05:00:16 [INFO] Pass MoveSqueezeAfterReluOptimizer elapsed time: 123.17 ms
2022-02-23 05:00:17 [INFO] Pass Pre Optimization elapsed time: 119182.96 ms
2022-02-23 05:00:18 [INFO] Neither evaluation function nor metric is defined. Generate a quantized model with default quantization configuration.
2022-02-23 05:00:18 [INFO] Generate a fake evaluation function.
2022-02-23 05:00:18 [INFO] Get FP32 model baseline.
2022-02-23 05:00:18 [INFO] Save tuning history to /workspaces/Neural_Compressor/nc_workspace/2022-02-23_04-51-36/./history.snapshot.
2022-02-23 05:00:18 [INFO] FP32 baseline is: [Accuracy: 1.0000, Duration (seconds): 0.0001]
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:00:19 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:01:14 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:02:09 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:03:05 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:04:00 [WARNING] Found possible input node names: ['InputLayer'], output node names: ['dense_3'].
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:04:01 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:04:57 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:05:52 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:06:47 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:07:43 [WARNING] Found possible input node names: ['InputLayer'], output node names: ['dense_3'].
WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:07:44 [WARNING] SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), NOT tf.saved_model.save(). To confirm, there should be a file named "keras_metadata.pb" in the SavedModel directory.
2022-02-23 05:08:41.231079: I tensorflow/core/grappler/devices.cc:75] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2022-02-23 05:08:41.231264: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-02-23 05:08:41.254906: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1149] Optimization results for grappler item: graph_to_optimize
function_optimizer: function_optimizer did nothing. time = 0.013ms.
function_optimizer: function_optimizer did nothing. time = 0.001ms.

2022-02-23 05:08:46.726701: I tensorflow/core/grappler/devices.cc:75] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2022-02-23 05:08:46.726891: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-02-23 05:08:46.895748: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1149] Optimization results for grappler item: tf_graph
constant_folding: Graph size after: 1118 nodes (-612), 1787 edges (-612), time = 63.491ms.
constant_folding: Graph size after: 1118 nodes (0), 1787 edges (0), time = 53.424ms.

2022-02-23 05:09:21 [INFO] Pass Quantization elapsed time: 33568.48 ms
2022-02-23 05:10:22 [INFO] Pass QuantizedRNNConverter elapsed time: 76.27 ms
2022-02-23 05:10:26 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 143.55 ms
2022-02-23 05:10:26 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 57.56 ms
2022-02-23 05:10:26 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 63.12 ms
2022-02-23 05:10:26 [INFO] Pass MetaOpOptimizer elapsed time: 28.79 ms
2022-02-23 05:10:27 [INFO] Pass PostCseOptimizer elapsed time: 403.61 ms
2022-02-23 05:10:28 [INFO] |*Mixed Precision Statistics|
2022-02-23 05:10:28 [INFO] +---------------+---------+-------+-------+
2022-02-23 05:10:28 [INFO] | Op Type | Total | INT8 | FP32 |
2022-02-23 05:10:28 [INFO] +---------------+---------+-------+-------+
2022-02-23 05:10:28 [INFO] | Conv2D | 120 | 120 | 0 |
2022-02-23 05:10:28 [INFO] | MatMul | 4 | 4 | 0 |
2022-02-23 05:10:28 [INFO] | ConcatV2 | 58 | 0 | 58 |
2022-02-23 05:10:28 [INFO] | MaxPool | 1 | 1 | 0 |
2022-02-23 05:10:28 [INFO] | AvgPool | 3 | 3 | 0 |
2022-02-23 05:10:28 [INFO] | QuantizeV2 | 64 | 64 | 0 |
2022-02-23 05:10:28 [INFO] | Dequantize | 64 | 64 | 0 |
2022-02-23 05:10:28 [INFO] +---------------+---------+-------+-------+
2022-02-23 05:10:28 [INFO] Pass quantize model elapsed time: 609884.64 ms
2022-02-23 05:10:28 [INFO] Tune 1 result is: [Accuracy (None|fp32): 1.0000|1.0000, Duration (seconds) (None|fp32): 0.0000|0.0001], Best tune result is: [Accuracy: 1.0000, Duration (seconds): 0.0000]
2022-02-23 05:10:28 [INFO] Save tuning history to /workspaces/Neural_Compressor/nc_workspace/2022-02-23_04-51-36/./history.snapshot.
2022-02-23 05:10:29 [INFO] Specified timeout or max trials is reached! Found a quantized model which meet accuracy goal. Exit.
2022-02-23 05:10:29 [INFO] Save deploy yaml to /workspaces/Neural_Compressor/nc_workspace/2022-02-23_04-51-36/deploy.yaml
2022-02-23 05:10:29.169157: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:207: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
2022-02-23 05:10:29 [WARNING] From /usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:207: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
INFO:tensorflow:No assets to save.
2022-02-23 05:10:29 [INFO] No assets to save.
INFO:tensorflow:No assets to write.
2022-02-23 05:10:29 [INFO] No assets to write.
INFO:tensorflow:SavedModel written to: /workspaces/Neural_Compressor/models/int8/saved_model.pb
2022-02-23 05:10:30 [INFO] SavedModel written to: /workspaces/Neural_Compressor/models/int8/saved_model.pb
2022-02-23 05:10:30 [INFO] Save quantized model to /workspaces/Neural_Compressor/models/int8.

[NeoZhangJianyu]

@mesolmaz
The root cause is the data format:
The shape of x and y must be same as that in trained model.

In your code, the x shape is (n, 300, 300, 1).
Please change it as same as your model if needed.

The default metric hope the label is like: [0, 2, 3].
In your dataset code, the label is [[1,0,0,0], [0,0,1,0], [0,0,0,1]].

Please change your code to:

class Dataset(object):
  def __init__(self, image_dir):

      datagen = keras.preprocessing.image.ImageDataGenerator()
      train_generator = datagen.flow_from_directory(
        image_dir,
        target_size=(300, 300),
        color_mode ='grayscale',
        batch_size=32,
        #class_mode='categorical'
        class_mode='binary'
      )
      x=np.concatenate([train_generator.next()[0] for i in range(train_generator.__len__())])
      y=np.concatenate([train_generator.next()[1] for i in range(train_generator.__len__())])
      y=[int(i) for i in y]
      self.test_images = x / 255.0
      self.labels = y
      pass
  ....

[mesolmaz]

I made the changes to the code, but I still see the same message about fake evaluation function. I also see the same Mixed Precision Statistics and tune result:

Tune 1 result is: [Accuracy (None|fp32): 1.0000|1.0000, Duration (seconds) (None|fp32): 0.0000|0.0001], Best tune result is: [Accuracy: 1.0000, Duration (seconds): 0.0000]

Here is the code for fitting the quantizer:

quantizer = Quantization('/workspaces/Neural_Compressor/neural-compressor/examples/helloworld/tf_example2/conf.yaml')
train_parent_dir = "/workspaces/Neural_Compressor/imagenet/regular_labels"
dataset = Dataset(image_dir = train_parent_dir)
quantizer.metric = MyMetric()
quantizer.calib_dataloader = common.DataLoader(dataset, batch_size=1)
quantizer.eval_dataloader = common.DataLoader(dataset, batch_size=1)
quantizer.model = common.Model('/workspaces/Neural_Compressor/models/tensorflow/model')
q_model = quantizer.fit()

q_model.save('./models/int8')

[mesolmaz]

The problem was the Metric class. I replaced the line:

quantizer.metric = MyMetric()

with

quantizer.metric = common.Metric(metric_cls=MyMetric)

and I was able to get 2 iterations of tuning. Accuracy was 0.2975 but I can try to improve that.

[mengniwang95]

Thank you for finding this issue, we will fix it asap.