Model optimizer does not accept float16 model as input

[ed-alertedh]

Since I'm targeting an NCS 2 device that only supports FP16 inference, I thought I would try training the model as FP16 in the first place. I trained a relatively simple model with tensorflow and tried to convert from checkpoint with this command:
python $mo_path --input_meta_graph eval_graph/checkpoint-data.meta --output_dir ir_model --data_type FP16

I then got this error message:

Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      None
        - Path for generated IR:        /mnt/d/dev/singulariti/oscillot-classifier/edgetpu/intel_model/
        - IR output name:       checkpoint-data
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP16
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  None
Model Optimizer version:        2019.1.0-341-gc9b66a2
[ ERROR ]  Unexpected exception happened during extracting attributes for node inputs.
Original exception message: Data type is unsupported: 19.
 For more information please refer to Model Optimizer FAQ (<INSTALL_DIR>/deployment_tools/documentation/docs/MO_FAQ.html), question #50.

According to the proto definition data type 19 is float16. So it seems like the converter does not support fp16->fp16 and will only do fp32->fp16. Am I doing something wrong? If this use-case is unsupported are there plans to support it?

edit: I have done a small amount of experimenting, making notes here in case I need to prepare a PR later

• added a mapping to model_optimizer/mo/front/tf/common.py: ('DT_HALF', np.float16, lambda pb: pb.half_val, lambda x: np.float16(x)),
• Seem to have triggered a weird edge case converting constants: ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (288,) and requested shape (9,1,32). Editing model_optimizer/mo/front/tf/extractors/utils.py seemed to make it work enough to produce an output IR model, unsure if this is valid for all cases though:

#return np.broadcast_to(value, shape=shape).copy()
return np.reshape(value, shape).copy()

[shubha-ramani]

Dear @ed-alertedh
Actually FP16 works fine, see the help output for python mo_tf.py --h (you are doing Tensorflow but they all have this switch):
--data_type {FP16,FP32,half,float}]
Just out of curiosity, can you try half ? Please add --log_level DEBUG to your mo command and post it here. Maybe there is some incompatibility between FP16->FP16 like you said. This is odd. I can certainly file a bug on it.

According to that document https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/types.proto
I see this :
DT_HALF = 19;
So maybe the magic is --data_type half. Can you kindly try it ?

Thanks,

Shubha

[ed-alertedh]

I've re-run the same conversion with debug logging and --data_type half as suggested and get the same error. Logs and model checkpoint attached - it's just a dummy model based on this architecture but trained on synthetic data. I think it actually ended up with some NaN weights but this shouldn't matter for the conversion process.

eval_graph.zip

From what I have gathered exploring the codebase, --data_type only sets the dtype that all the arrays need to be converted to. Before that happens, the tool appears to load all the constants as numpy arrays in their original data type using a lookup table of supported tensorflow data types. E.g.: https://github.com/opencv/dldt/blob/e206d06f18f9d3c3db29f22c3d496cb3627a16c7/model-optimizer/mo/front/tf/common.py#L30
It looks like this lookup table needs an entry for DT_HALF but after digging a bit more into the internals it looks like tensorflow actually encodes this data type as a list of zero-padded int32 (because the protobuf format doesn't support any kind of 16-bit values, either integer or floating point). So it needs a "reinterpret_cast" with something like this: np.int16(x).view(dtype=np.float16)). The way the code is structured doesn't appear to be quite flexible enough to implement this in the current lookup table without some refactoring, so I assume that is why it was not yet implemented.

I also stumbled on what I believe is a latent bug here: https://github.com/opencv/dldt/blob/e206d06f18f9d3c3db29f22c3d496cb3627a16c7/model-optimizer/mo/front/tf/extractors/utils.py#L83

type_helper[0] is just the numpy dtype for the tensor. Most of the int types are encoded as zero-padded int32, so if that code path attempts to decode something like int16 I don't believe it will do it correctly. See:
https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/core/framework/tensor.proto#L53

I've decided I will just train in FP32 and convert to FP16 for inference as that appears to be the tested pathway. Tensorflow currently has issues around training in FP16 anyway. I probably won't be attempting a PR for this as it looks like a non-trivial refactor, but I'm happy to clarify any of the comments above if somebody else does end up taking a look at this.

[shubha-ramani]

Dearest @ed-alertedh

Thank you for all your careful debugging. I really appreciate it. I will file a bug on your behalf.

Sincerely,

Shubha

[shubha-ramani]

@ed-alertedh
Hi there. So this bug was rejected as "Works as designed". The reason is, we offer the calibration tools (INT8) precision as a way to reduce precision on your models.

Sorry about this. Hope it helps.

Thanks,

Shubha

[ed-alertedh]

That's less than ideal, given that the NCS 2 does not support int8 models (according to none other than yourself!) https://software.intel.com/en-us/forums/computer-vision/topic/807527

Actually, one of the reasons I found the NCS 2 device easier to use compared to Google Coral is that quantization isn't required. For now rounding weights from fp32 to fp16 will be OK for my purposes but it is disappointing that Intel isn't willing to support more advanced methods like mixed-precision training or finetuning post-training to make the most of the Myriad hardware which exclusively supports FP16.