RuntimeError: Given shapes, [1,784] and [784,100], are not broadcastable.Node number 1 (MUL) failed to prepare.

[aanujdu]

1. System information

I am using Google Colab. Gives the output Linux a0c9eeb98a07 5.10.147+ #1 SMP Sat Dec 10 16:00:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux, when I run uname -a.

Tensorflow Version: 2.12.0

2. Code

Reference TensorFlow Model Colab: The basic conversion to TFLite without any representative dataset works just fine. However, the problematic portions are cells 4, and 5, where I am trying to perform a TFLite conversion with a representative dataset.

3. Failure after conversion

It throws the following error.

WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /tmp/tmpz4fb25rp/assets
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-6-3bd48f77d98d>](https://localhost:8080/#) in <cell line: 10>()
      8 converter.inference_output_type = tf.uint8
      9 
---> 10 tflite_model_quant = converter.convert()
     11 
     12 interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)

13 frames
[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in convert(self)
   1895         Invalid quantization parameters.
   1896     """
-> 1897     return super(TFLiteConverterV2, self).convert()
   1898 
   1899 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in wrapper(self, *args, **kwargs)
    960   def wrapper(self, *args, **kwargs):
    961     # pylint: disable=protected-access
--> 962     return self._convert_and_export_metrics(convert_func, *args, **kwargs)
    963     # pylint: enable=protected-access
    964 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _convert_and_export_metrics(self, convert_func, *args, **kwargs)
    938     self._save_conversion_params_metric()
    939     start_time = time.process_time()
--> 940     result = convert_func(self, *args, **kwargs)
    941     elapsed_time_ms = (time.process_time() - start_time) * 1000
    942     if result:

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in convert(self)
   1534     """
   1535     if self.experimental_lower_to_saved_model:
-> 1536       saved_model_convert_result = self._convert_as_saved_model()
   1537       if saved_model_convert_result:
   1538         return saved_model_convert_result

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _convert_as_saved_model(self)
   1514       if self.saved_model_dir:
   1515         self._validate_inputs(graph_def, input_tensors)
-> 1516         return self._convert_from_saved_model(graph_def)
   1517     finally:
   1518       shutil.rmtree(temp_dir, True)

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _convert_from_saved_model(self, graph_def)
   1129 
   1130     result = _convert_saved_model(**converter_kwargs)
-> 1131     return self._optimize_tflite_model(
   1132         result, quant_mode, quant_io=self.experimental_new_quantizer)
   1133 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    213       except Exception as error:
    214         report_error_message(str(error))
--> 215         raise error from None  # Re-throws the exception.
    216 
    217     return wrapper

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    203     def wrapper(*args, **kwargs):
    204       try:
--> 205         return func(*args, **kwargs)
    206       except ConverterError as converter_error:
    207         if converter_error.errors:

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _optimize_tflite_model(self, model, quant_mode, quant_io)
    897         q_allow_float = quant_mode.is_allow_float()
    898         q_variable_quantization = quant_mode.enable_mlir_variable_quantization
--> 899         model = self._quantize(model, q_in_type, q_out_type, q_activations_type,
    900                                q_bias_type, q_allow_float,
    901                                q_variable_quantization)

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _quantize(self, result, input_type, output_type, activations_type, bias_type, allow_float, enable_variable_quantization)
    636                                                 custom_op_registerers_by_func)
    637     if self._experimental_calibrate_only or self.experimental_new_quantizer:
--> 638       calibrated = calibrate_quantize.calibrate(
    639           self.representative_dataset.input_gen)
    640 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    213       except Exception as error:
    214         report_error_message(str(error))
--> 215         raise error from None  # Re-throws the exception.
    216 
    217     return wrapper

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    203     def wrapper(*args, **kwargs):
    204       try:
--> 205         return func(*args, **kwargs)
    206       except ConverterError as converter_error:
    207         if converter_error.errors:

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/optimize/calibrator.py](https://localhost:8080/#) in calibrate(self, dataset_gen)
    224       dataset_gen: A generator that generates calibration samples.
    225     """
--> 226     self._feed_tensors(dataset_gen, resize_input=True)
    227     return self._calibrator.Calibrate()

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/optimize/calibrator.py](https://localhost:8080/#) in _feed_tensors(self, dataset_gen, resize_input)
    127                                      signature_key)
    128           else:
--> 129             self._calibrator.Prepare([list(s.shape) for s in input_array])
    130         else:
    131           if signature_key is not None:

RuntimeError: Given shapes, [1,784] and [784,100], are not broadcastable.Node number 1 (MUL) failed to prepare.

Following is the netron image and the problematic node info.

I do not suspect the issue to be with the broadcasting of scalar r in the multiplication with W1_1, because the code here works just fine. The only difference in this new code is that the call implementation only takes r as the input argument and returns the product of r and W1_1, which does not emit the error shown above, even though there is a broadcast.

[pjpratik]

Hi @aanujdu Thanks for reporting the issue.

@sachinprasadhs I was able to reproduce this issue. Please find the gist here.

Thanks.

[sachinprasadhs]

The issue seems to be with the representative dataset which you are using.

Below is the corrected way to use the representative_data_gen.

def representative_data_gen():
    for _ in range(100):
        yield {'image': tf.random.normal(shape=(1, 784), dtype=tf.float32),'r1': tf.Variable(np.random.randn(), dtype=tf.float32)}

With the above changes, I'm able to convert the model without any issues. Attaching the Gist for reference here https://gist.github.com/sachinprasadhs/fb4b62e7e37b5308217addbe9c2edd5f.

[aanujdu]

Thanks for the response @sachinprasadhs . I can reproduce a working conversion with your code. However, I am still not sure what is the actual issue. Are you saying I need to specify the output as a dictionary with keys as the names of the variables I specify in the input signature? If that is the case, could you please let me know how I write the code if I want to use the train_images_flat array, and the random array as my representative dataset? The example you have given is for random inputs, which works for sure, but I want to know how to extend it for an actual usecase with real variables.

I tried the following code using the dictionary format, but it still gives me a similar error.

def representative_data_gen():
      for (input_value, randomness) in zip(tf.data.Dataset.from_tensor_slices(train_images_flat).batch(1).take(100),
                                        r[0:100, :]):
          yield {'image': input_value, 'r1': tf.Variable(randomness, dtype=tf.float32)}

Error:

WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.
INFO:tensorflow:Assets written to: /tmp/tmp_c8clq3s/assets
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-7-a7311f42f11a>](https://localhost:8080/#) in <cell line: 14>()
     12 converter.representative_dataset = representative_data_gen
     13 
---> 14 tflite_model_quant = converter.convert()
     15 
     16 interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)

13 frames
[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in convert(self)
   1895         Invalid quantization parameters.
   1896     """
-> 1897     return super(TFLiteConverterV2, self).convert()
   1898 
   1899 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in wrapper(self, *args, **kwargs)
    960   def wrapper(self, *args, **kwargs):
    961     # pylint: disable=protected-access
--> 962     return self._convert_and_export_metrics(convert_func, *args, **kwargs)
    963     # pylint: enable=protected-access
    964 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _convert_and_export_metrics(self, convert_func, *args, **kwargs)
    938     self._save_conversion_params_metric()
    939     start_time = time.process_time()
--> 940     result = convert_func(self, *args, **kwargs)
    941     elapsed_time_ms = (time.process_time() - start_time) * 1000
    942     if result:

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in convert(self)
   1534     """
   1535     if self.experimental_lower_to_saved_model:
-> 1536       saved_model_convert_result = self._convert_as_saved_model()
   1537       if saved_model_convert_result:
   1538         return saved_model_convert_result

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _convert_as_saved_model(self)
   1514       if self.saved_model_dir:
   1515         self._validate_inputs(graph_def, input_tensors)
-> 1516         return self._convert_from_saved_model(graph_def)
   1517     finally:
   1518       shutil.rmtree(temp_dir, True)

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _convert_from_saved_model(self, graph_def)
   1129 
   1130     result = _convert_saved_model(**converter_kwargs)
-> 1131     return self._optimize_tflite_model(
   1132         result, quant_mode, quant_io=self.experimental_new_quantizer)
   1133 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    213       except Exception as error:
    214         report_error_message(str(error))
--> 215         raise error from None  # Re-throws the exception.
    216 
    217     return wrapper

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    203     def wrapper(*args, **kwargs):
    204       try:
--> 205         return func(*args, **kwargs)
    206       except ConverterError as converter_error:
    207         if converter_error.errors:

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _optimize_tflite_model(self, model, quant_mode, quant_io)
    897         q_allow_float = quant_mode.is_allow_float()
    898         q_variable_quantization = quant_mode.enable_mlir_variable_quantization
--> 899         model = self._quantize(model, q_in_type, q_out_type, q_activations_type,
    900                                q_bias_type, q_allow_float,
    901                                q_variable_quantization)

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/lite.py](https://localhost:8080/#) in _quantize(self, result, input_type, output_type, activations_type, bias_type, allow_float, enable_variable_quantization)
    636                                                 custom_op_registerers_by_func)
    637     if self._experimental_calibrate_only or self.experimental_new_quantizer:
--> 638       calibrated = calibrate_quantize.calibrate(
    639           self.representative_dataset.input_gen)
    640 

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    213       except Exception as error:
    214         report_error_message(str(error))
--> 215         raise error from None  # Re-throws the exception.
    216 
    217     return wrapper

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/convert_phase.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    203     def wrapper(*args, **kwargs):
    204       try:
--> 205         return func(*args, **kwargs)
    206       except ConverterError as converter_error:
    207         if converter_error.errors:

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/optimize/calibrator.py](https://localhost:8080/#) in calibrate(self, dataset_gen)
    224       dataset_gen: A generator that generates calibration samples.
    225     """
--> 226     self._feed_tensors(dataset_gen, resize_input=True)
    227     return self._calibrator.Calibrate()

[/usr/local/lib/python3.9/dist-packages/tensorflow/lite/python/optimize/calibrator.py](https://localhost:8080/#) in _feed_tensors(self, dataset_gen, resize_input)
    127                                      signature_key)
    128           else:
--> 129             self._calibrator.Prepare([list(s.shape) for s in input_array])
    130         else:
    131           if signature_key is not None:

RuntimeError: Given shapes, [2] and [784,100], are not broadcastable.Node number 1 (MUL) failed to prepare.

Maybe, what might help is, if you can clearly explain the set of rules I need to follow when I am using the representative_data_gen function, instead of a working example.

Thanks,

[sachinprasadhs]

Well in that case, according to the data which you have, you should provide in the below format.

def representative_data_gen():
      for (input_value, randomness) in zip(tf.data.Dataset.from_tensor_slices(train_images_flat).batch(1).take(100),
                                        r[0:100, :]):
          yield {'image': input_value, 'r1': tf.Variable(randomness[0], dtype=tf.float32)}

Here is the working Gist.

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No