[onert/odc] Draft for odc: hidden switching mechanism. fcircle-qcircle step

[Torrero]

This is draft for odc: hidden switching mechanism. fcircle-qcircle step was implemented.

Conv2D_000.circle is used as model for test (it was added just for this draft verification)
Size of the model: 556B

This draft contains auto compilation (odc: hidden switching mechanism) , which is executed in nnfw_run_with_auto_compilation function.

• nnfw_run_with_auto_compilation function was added.
• odc::QuantizerManager, odc::Quantizer and odc::MinMaxReader were changes for set and minmax threshold and cheсking readiness for quantization. Function for removing minmax file was added. (question about set up the minmax file location)
• OdcInfo class for storaging actual state of the odc was added.

After model quantization, the removing minmax file step was added.
After first loading quantized model the following sequence was implemented:

• save the inputs and outputs buffers of the initial fcricle model
• reload qcircle model to the session
• recover inputs and outputs

after that do inference of the quantized model

For running test of this draft:

BUILD_TYPE=Debug BUILD_JOBS=6 make -f Makefile.template

cd ./Product/out/unittest

./nnfw_api_gtest --gtest_filter=TestOdcAutoCompilation*

Time of the test execution: 11 ms
Peak memory footprint of the test: 4,939 Kb

ONE-DCO-1.0-Signed-off-by: Evgenii Maltsev e.maltsev@samsung.com

[hseok-oh]

I don't check all your implementation yet, but here is my comment

• Please introduce different run API for hidden switching, not in nnfw_run.
• Your QDQBufferAdapter implementation looks almost same with [onert] Type-aware quantized model input/output buffer setting #13287. Please remove functional duplication if it is same.

[Torrero]

Please introduce different run API for hidden switching, not in nnfw_run.

Your QDQBufferAdapter implementation looks almost same with #13287. Please remove functional duplication if it is same.

@hseok-oh Thank you for pointing out your implementation of quantization and dequantization of buffers. I removed this functional duplication and moved hidden_switching_mechanizm from nnfw_run to nnfw_run_odc_hsm

[jyoungyun]

(optional)

I think it would be good to use a different function name to make it easier for the user to understand, but I am not sure about the appropriate function name.. :( It's a long name, but how about nnfw_run_with_auto_compilation? If you have a better idea for the function name, you can use the name. :)

[Torrero]

(optional)

I think it would be good to use a different function name to make it easier for the user to understand, but I am not sure about the appropriate function name.. :( It's a long name, but how about nnfw_run_with_auto_compilation? If you have a better idea for the function name, you can use the name. :)

@jyoungyun It's a good idea, thank you. I renamed the function name to nnfw_run_with_auto_compilation and changed some descriptions.

[Torrero]

@hseok-oh @jyoungyun
I updated this draft, added compilation step.
Now there is an attempt to compile of the quantized model and load compiled model, if it fails, try to load and run the quantized model.

Could you advise me, please, how can I add a test of the compilation part?

[hseok-oh]

IMO, responsibility of saving minmax recording count threshold and checking recording count should have in quantizer, not minmax recorder. So API should query to quantizer that it is ready to quantize, and quantizer should check recording count from file. In this point of view, if quantizer replies it is ready, API will change exec config to stop recording and request quantizer to actually quantize.

Additionally, hdf5 minmax dumper is not used now, and that will be removed.

[Torrero]

@hseok-oh

I moved checking the readiness for quantization and minmax recording threshold to odc::Quantizer. Also function for removing minmax file was added to it (Maybe it is better to move this function to odc::MinMaxReader or we can define another location for it)

[Torrero]

Negative test was added

[Torrero]

@hseok-oh PTAL