ResNet-18 INT8 Quantized Model: Unnamed Layer

[Hassan313]

@prasshantg Hi Prashant,

I have followed the steps stated in the https://github.com/nvdla/sw/blob/master/LowPrecision.md.

After the model is quantizeed, I get some Unnamed Layers. How can I fix this problem?

Part of the calibration file as below:

TRT-7000-EntropyCalibration2
data: 3caa54fc
(Unnamed Layer* 0) [Convolution]_output: 3d6969df
(Unnamed Layer* 1) [Scale]_output: 3ac9c5e4
(Unnamed Layer* 2) [Scale]_output: 3bb23f5c
conv1: 3bb2132a
pool1: 3bb2132a
(Unnamed Layer* 5) [Convolution]_output: 3b0f3403
(Unnamed Layer* 6) [Scale]_output: 3c19f4a8
res2a_branch1: 3b39bf9e
(Unnamed Layer* 8) [Convolution]_output: 3b64b387
(Unnamed Layer* 9) [Scale]_output: 3c34ac95
(Unnamed Layer* 10) [Scale]_output: 3b309597
res2a_branch2a: 3b21cb45
(Unnamed Layer* 12) [Convolution]_output: 3acc6b2e
(Unnamed Layer* 13) [Scale]_output: 3cb1a784
res2a_branch2b: 3b5eba4f
(Unnamed Layer* 15) [ElementWise]_output: 3ba75a96
res2a: 3b73bec7
(Unnamed Layer* 17) [Convolution]_output: 3b8658ae
(Unnamed Layer* 18) [Scale]_output: 3cce329f
(Unnamed Layer* 19) [Scale]_output: 3b9b34cf
res2b_branch2a: 3b6795f7
(Unnamed Layer* 21) [Convolution]_output: 3b1fb3bc
(Unnamed Layer* 22) [Scale]_output: 3d1fdc85
res2b_branch2b: 3c0d6bd4
(Unnamed Layer* 24) [ElementWise]_output: 3ba9e0a1
res2b: 3ba9e0a1
(Unnamed Layer* 26) [Convolution]_output: 3b16170a
(Unnamed Layer* 27) [Scale]_output: 3cf6451c

Part of the JSON file created from the calibration file:

{
"data": {
"scale": 0.02079247683286667,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 0) [Convolution]_output": {
"scale": 0.05698573216795921,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 1) [Scale]_output": {
"scale": 0.0015394059009850025,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 2) [Scale]_output": {
"scale": 0.0054396819323301315,
"min": 0,
"max": 0,
"offset": 0
},
"conv1": {
"scale": 0.005434413440525532,
"min": 0,
"max": 0,
"offset": 0
},
"pool1": {
"scale": 0.005434413440525532,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 5) [Convolution]_output": {
"scale": 0.0021851069759577513,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 6) [Scale]_output": {
"scale": 0.009396709501743317,
"min": 0,
"max": 0,
"offset": 0
},
"res2a_branch1": {
"scale": 0.002834297250956297,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 8) [Convolution]_output": {
"scale": 0.003489704569801688,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 9) [Scale]_output": {
"scale": 0.011027474887669086,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 10) [Scale]_output": {
"scale": 0.002694463124498725,
"min": 0,
"max": 0,
"offset": 0
},
"res2a_branch2a": {
"scale": 0.0024687808472663164,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 12) [Convolution]_output": {
"scale": 0.0015595906879752874,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 13) [Scale]_output": {
"scale": 0.02168632298707962,
"min": 0,
"max": 0,
"offset": 0
},
"res2a_branch2b": {
"scale": 0.0033985560294240713,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 15) [ElementWise]_output": {
"scale": 0.0051072342321276665,
"min": 0,
"max": 0,
"offset": 0
},
"res2a": {
"scale": 0.0037192569579929113,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 17) [Convolution]_output": {
"scale": 0.00409992691129446,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 18) [Scale]_output": {
"scale": 0.025170622393488884,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 19) [Scale]_output": {
"scale": 0.004736519884318113,
"min": 0,
"max": 0,
"offset": 0
},
"res2b_branch2a": {
"scale": 0.0035337188746780157,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 21) [Convolution]_output": {
"scale": 0.002436860464513302,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 22) [Scale]_output": {
"scale": 0.03902866318821907,
"min": 0,
"max": 0,
"offset": 0
},
"res2b_branch2b": {
"scale": 0.008631665259599686,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 24) [ElementWise]_output": {
"scale": 0.0051842485554516315,
"min": 0,
"max": 0,
"offset": 0
},
"res2b": {
"scale": 0.0051842485554516315,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 26) [Convolution]_output": {
"scale": 0.0022901915945112705,
"min": 0,
"max": 0,
"offset": 0
},
"(Unnamed Layer* 27) [Scale]_output": {
"scale": 0.030062250792980194,
"min": 0,
"max": 0,
"offset": 0
},

Can you kindly help?

Thank you very much.

[prasshantg]

Hi Hassan,

You can ignore those unnamed layers. Just use layers which have names. Those are internal layers used by TRT.

[Hassan313]

@prasshantg Hi Prashant,

Thank you for your response. Below is an image for the Resnet-50 and ResNet-18 calibration JSON files:

The left one is for the ResNet-50 which you have provided and the right one is the ResNet-18, which I have created. It seems that you have not ignored any of those unnamed layers. May I know the reason you have kept those for the ResNet-50, however, you are recommending to ignore those unnamed layer for the ResNet-18?

Thank you very much.

[Hassan313]

Hi Hassan,

You can ignore those unnamed layers. Just use layers which have names. Those are internal layers used by TRT.

I ignored what those layers, and I am trying to compile the ResNet-18 with out the calibration file, and I am getting the below error:

~/nvdla/sw/sw/prebuilt/x86-ubuntu$ ./nvdla_compiler --prototxt ~/CaffeModels/ResNet-18/deploy.prototxt --caffemodel ~/CaffeModels/ResNet-18/resnet-18.caffemodel -o . --profile fast-math --cprecision int8 --configtarget nv_small --calibtable calibtable/resnet18.json --quantizationMode per-filter --batch 1 --informat nhwc
libnvdla<3> mark prob
nvdla_compiler: /odla/vpr/sw/umd/external/include/rapidjson/document.h:1105: rapidjson::GenericValue<Encoding, Allocator>& rapidjson::GenericValue<Encoding, Allocator>::operator[](const rapidjson::GenericValue<Encoding, SourceAllocator>&) [with SourceAllocator = rapidjson::MemoryPoolAllocator<>; Encoding = rapidjson::UTF8<>; Allocator = rapidjson::MemoryPoolAllocator<>]: Assertion `false' failed.
Aborted

Can you kindly help with this issue?

Thank you very much.

[Y-U-KAY]

Hi @Hassan313,
I got the same issue before. I think some layers cannot be ignored.
Some unnamed layers are generated because of the same output name of some layers in resnet18 model. The calibration tool takes the output name as its layer name.
The error shows that the layer name cannot be found in the calibration file.

[Hassan313]

Hi @Hassan313,
I got the same issue before. I think some layers cannot be ignored.
Some unnamed layers are generated because of the same output name of some layers in resnet18 model. The calibration tool takes the output name as its layer name.
The error shows that the layer name cannot be found in the calibration file.

Hi @Y-U-KAY ,

Thank you so much for your reply. Have you done the calibration for any of the neural networks models? Were you successful and if yes, how much the accuracy of the neural network was degraded?

Thank you very much for your reply.

[DCSong]

I meet the same question. Transforming ResNet-50.txt to ResNet-50.json with calib_txt_to_json.py provided by official is not work. The .json will have many Unnamed Layer which is different with the ResNet-50.json provided by official.

[LeiWang1999]

I got same issue , and it can be resolved by my handwrite python3 code :

import json
from collections import OrderedDict
from google.protobuf import text_format
import caffe.proto.caffe_pb2 as caffe_pb2      # 载入caffe.proto编译生成的caffe_pb2文件


caffeprototxt_path = "./deploy.prototxt"
calibletable_json_path = "./resnet18-int8.json"
# load deploy.prototxt
net = caffe_pb2.NetParameter()
text_format.Merge(open(caffeprototxt_path).read(), net)
# load jsonfile
with open(calibletable_json_path, "r") as f:
    calible = json.load(f, object_pairs_hook=OrderedDict)
_scales = []
_mins = []
_maxs = []
_offsets = []
_new = OrderedDict()
items = calible.items()
for key, value in items:
    _scales.append(value['scale'])
    _mins.append(value['min'])
    _maxs.append(value['max'])
    _offsets.append(value['offset'])
for idx, _layer in enumerate(net.layer):
    _tempDict =  OrderedDict({
        "scale": _scales[idx],
        "min": _mins[idx],
        "max": _maxs[idx],
        "offset": _offsets[idx],
    }) 
    _new[_layer.name] =_tempDict
with open('output.json', 'w') as f:
    json.dump(_new, f)

[Hassan313]

Hi Lei,

Thank you for sharing your script here.

What accuracy do you get with the generated calibration file of Resnet-18?

Thank you again for sharing your script here.

Best Regards
Hassan

[LeiWang1999]

Hi @Hassan313

I made an int8 lenet network afew days ago, and it worked on MNIST 👍 , the performance:

input: 0_1.jpg
output:  0 120 0 0 0 0 0 0 0 0
input: 0_2.jpg
output: 0 0 119 0 0 0 1 0 0 0

And i am trying to work on int8 resnet-18 on cifar10 dataset right now，but i got another issue when execute ./nvdla_compiler, the new issue can be found there #211

[Hassan313]

Hi @LeiWang1999 ,

Thank you for letting me know about Lenet-5. Actually, the NVDLA team has provided the calibration table for Lenet-5 and Resnet-50. I have tested those and it is working for me.

Thank you for pointing out the #211 issue you are encountering. I have used the Resnet-18 model for imagenet-data set and the compiler has no problem in compiling for both floating point and int8 precisions. However, the accuracy I got for the int-8 was zero due to the unnmaed layer the TensorRT was creating for the calibration table. Can you kindly check your code on Resnet-18 for imagenet dataset?

Thank you very much.

Best Regards
Hassan

[sandeep1404]

Hi,
I am new to using NVDLA, i have a question why does max and min values of all the layers are zero, if it follows symmetric quantization offset can be zero since its the zero point, but why min and max values of all the layers are zero? can someone clear my doubt.

[long771]

这是来自QQ邮箱的自动回复邮件。 您好，我是龙欣荣，很高兴收到您的邮件，我将尽快处理。