Converting an existing model to int8

[kthoru01]

Hello,

I apologize if this is a very novice question, but I'm to this platform. I have an existing .prototxt and .caffemodel files. Is there a way to requantize those without going through the whole training process? If not, is there a way to integrate the .prototxt file into the sampleint8 file? Any assistance would be greatly appreciated. This is for a class project I'm working on.

[ttyio]

Hello @kthoru01 ,
Post-traning-quantization require no extra training/finetuning, but still require you provide the calibration dataset to generate the int8 scale.

this sample demonstrate how to work in INT8 using caffe model, could you take a read? thanks
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleINT8

[kthoru01]

Thank you very much @ttyio ! I did see that example and what it does makes much more sense now. I'm working with the NVDLA platform, running inference on a LeNet model. I was able to get accurate results for the nv_full fp16 configuration, but for nv_full int8 configuration the output is all zeros. I believe this is do to not providing a calibration file to the compiler when creating the nv_full int8 loadable. I need to create an int8 calibration file for LeNet. To do this do I need to run training on the model again? Thank you for your help! I'm completely new to this so I really appreciate it!

[ttyio]

@kthoru01 , could you try generate the calibration file follow https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleINT8#setup-the-calibrator ? thanks!

[kthoru01]

Thank you @ttyio ! I read through it and overall it makes sense. I'll post another issue if I ran into problems. On a different note, I was wondering if you knew anyone that could answer a few questions about the NVDLA source code, specifically how to modify KMD source files to print more information about what the network is doing such as memory access pattern, and how many MAC units it's utilizing from the configuration. Any guidance on this would be incredibly appreciated. Thank you!

[ttyio]

Thanks @kthoru01 , Could ask NVDLA specific question in https://github.com/nvdla, there are NVDLA experts in that group.
I will close this one since this is solved, and welcome to create new one if you hit issue using TRT, thanks!

[kthoru01]

Thank you @ttyio ! I made several posts on the nvdla repository but unfortunately no one has been answering questions in the last year or so. I was hoping you might know someone that could help. I appreciate all of your assistance.

[ttyio]

Hmmm maybe try https://forums.developer.nvidia.com/ ? @kthoru01

[kthoru01]

Hello @ttyio , I looked through the forum you shared with me. First of all, I appreciate you taking the time to share that with me. Unfortunately, almost all the posts are talking about Xavier platform. I'm running my project using the virtual platform. If there is any other resource you can think of that would talk more about modifying NVDLA source code that be amazing. I would greatly appreciate that!

[ttyio]

Sorry @kthoru01 , I do not know other channels for NVDLA ;-(

[kthoru01]

Thank you @ttyio , I really appreciate all of your help! In terms of tensorRT I looked again at the link you shared with me. When I tried to implement the quantization, I realized that to run the example I'll need a GPU because the example does inference. I do not have access to a GPU. Is there a way to get the calibration data without running inference?

[ttyio]

@kthoru01 , GPU is required to calibrate int8 in TRT. If you want to skip the calibration step, maybe you could try the baked resnet50_per_tensor_dynamic_range.txt in https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleINT8API#preparing-sample-data, it is for resnet50, and you can find the file if you download the TRT package from https://developer.nvidia.com/tensorrt