[ERROR] Unexpected exception InternalError() happened during turing!

[peiwenhuang27]

Framework: Tensorflow 2.5.0, Intel-Tensorflow 2.5.0
Environment: Google Colab
Model format: saved model (pre-trained under TF 2.5.0)

1. I encountered the following error

[ERROR] Unexpected exception InternalError() happened during turing!

several times, and I don't know what is the cause since it is not printing the source of error.

2. For the inputs outputs node suggestion

Found possible input node names: ['input_noisy'], output node names: ['dense']

I already set my inputs/outputs to the ones it suggested in my config yaml, but the warning keeps showing, as if ignoring the config file.

3. The warning keeps showing:

Please set environment variable TF_ENABLE_MKL_NATIVE_FORMAT=0 when Tensorflow 2.5.0 installed.

even after I specifically set it via

!export TF_ENABLE_MKL_NATIVE_FORMAT=0

4. I believe there is a typo in the error message, it's supposed to say 'tuning', no? 😃

Traceback

2021-07-19 02:52:17 [INFO] Start to run model quantization...
2021-07-19 02:52:17 [INFO] |Mixed Precision Statistics|
2021-07-19 02:52:17 [INFO] |INT8 MatMul: 3 |
2021-07-19 02:52:17 [INFO] |INT8 Conv2D: 5 |
2021-07-19 02:52:17 [INFO] |Overall: INT8 100.00% (8/8) BF16 0.00% (0/8) |
2021-07-19 02:52:17 [INFO] |**************************************************|
2021-07-19 02:52:18 [WARNING] Please set environment variable TF_ENABLE_MKL_NATIVE_FORMAT=0 when Tensorflow 2.5.0 installed.
2021-07-19 02:52:23 [WARNING] Found possible input node names: ['input_noisy'], output node names: ['dense']
2021-07-19 02:52:23 [WARNING] Found possible input node names: ['input_noisy'], output node names: ['dense']
2021-07-19 02:52:26 [INFO] loading session....
2021-07-19 02:52:29.701188: I tensorflow/core/grappler/devices.cc:78] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2021-07-19 02:52:29.701454: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-07-19 02:52:29.717618: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: graph_to_optimize
function_optimizer: Graph size after: 270 nodes (185), 364 edges (264), time = 4.316ms.
function_optimizer: function_optimizer did nothing. time = 0.183ms.

2021-07-19 02:52:30.057810: I tensorflow/core/grappler/devices.cc:78] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0 (Note: TensorFlow was not compiled with CUDA or ROCm support)
2021-07-19 02:52:30.058012: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-07-19 02:52:30.190360: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1144] Optimization results for grappler item: tf_graph
constant_folding: Graph size after: 185 nodes (-42), 218 edges (-46), time = 80.757ms.
constant_folding: Graph size after: 185 nodes (0), 218 edges (0), time = 26.106ms.

2021-07-19 02:52:30 [INFO] Unknown match MatMul
2021-07-19 02:52:30 [INFO] Unknown match MatMul
2021-07-19 02:52:30.842607: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-07-19 02:52:32 [INFO] Pass Quantization elapsed time: 1876.04 ms
2021-07-19 02:53:01 [INFO] Pass QuantizedRNNConverter elapsed time: 100.07 ms
2021-07-19 02:53:02 [INFO] Pass StripUnusedNodesOptimizer elapsed time: 300.89 ms
2021-07-19 02:53:02 [INFO] Pass RemoveTrainingNodesOptimizer elapsed time: 99.77 ms
2021-07-19 02:53:02 [INFO] Pass FoldBatchNormNodesOptimizer elapsed time: 100.88 ms
2021-07-19 02:53:02 [INFO] Pass MetaOpOptimizer elapsed time: 99.0 ms
2021-07-19 02:53:02 [INFO] Node name unused_control_flow_input_23 doesn't exist in the model, please check the yaml.
2021-07-19 02:53:02 [WARNING] Found possible input node names: ['input_noisy'], output node names: ['dense']
2021-07-19 02:53:05 [INFO] Pass PostCseOptimizer elapsed time: 3148.93 ms
2021-07-19 02:53:05 [INFO] Pass quantize model elapsed time: 47932.33 ms
2021-07-19 02:53:05 [INFO] Start to evaluate Tensorflow model...
2021-07-19 02:53:06 [ERROR] Unexpected exception InternalError() happened during turing!
2021-07-19 02:53:06 [ERROR] Specified timeout or max trials is reached! Not found any quantized model which meet accuracy goal. Exit...
Traceback (most recent call last):
File "main.py", line 94, in
evaluate_opt_graph.run()
File "main.py", line 79, in run
q_model.save(self.args.output_graph)
AttributeError: 'NoneType' object has no attribute 'save'

[ftian1]

@peiwenhuang27
The InternalError() is raised by TensorFlow and means there is graph exection exception. it's caused by not setting TF_ENABLE_MKL_NATIVE_FORMAT=0 env variable.

it's quite weird the message of "Please set environment variable TF_ENABLE_MKL_NATIVE_FORMAT=0 when Tensorflow 2.5.0 installed." keeping printed after you have set this env TF_ENABLE_MKL_NATIVE_FORMAT being 0.

if you take a look https://github.com/intel/lpot/blob/master/lpot/adaptor/tf_utils/graph_converter.py#L177, you should know it shouldn't happen...

[peiwenhuang27]

I ran a small test to check whether this variable has been set:

!export TF_ENABLE_MKL_NATIVE_FORMAT=0

import os
print(os.getenv('TF_ENABLE_MKL_NATIVE_FORMAT'))

The printed result shows None, literally right after I set the variable.

[peiwenhuang27]

I used another method to set the variable

os['TF_ENABLE_MKL_NATIVE_FORMAT']=0

and it worked! (seems like export does not work well in Colab)

Still don't know why the inputs/outputs warning keeps showing, though it seems like it doesn't effect the result.

Anyway, thank you for your explanation!

[guomingz]

The export VARIABLE_NAME=VALUE doesn't take effect on the Colab process which u want to update the enviroment variables. It's why the warning keeps showing. If you try Conda instead of Colab, the issue won't exist.

The only way to solve it for Colab is to call os.environ['VARIABLE_NAME']=VALUE.