tensorrt5 update to tensorrt7, batch infer exception

[dtiny]

Description

I was used tensorrt5 well before in my program. Recently I update tensorrt5 to tensorrt7 as well in one image infer. While I change one input to batch input, I get a exceptional infer result. although I
change batch from 2 to other num, I only can one correct infer result. Others results always zeros.
I set batch setting in serializion with builder->setMaxBatchSize(mMaxBatchSize) and config->setMaxWorkspaceSize(10 << 20). How can solve this exception?

Environment

C++ interface, caffe model parse
TensorRT Version: tensorrt7
GPU Type: GeForce GTX 1080TI
Nvidia Driver Version: 410.93
CUDA Version: cuda 10.0
CUDNN Version: cudnn 7.6.5
Operating System + Version: 16.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Steps To Reproduce

fatch infer result mOutputIdx[i]:1

0.406353 -0.320382 1.5007 0.457494 0.0969265 1.50484 -0.886063 1.35644 0.245721 -0.121499 0.654094 0.441087 -0.0382616 0.438509 0.366007 0.687773 0.0161026 -0.937535 -0.152266 -0.808002 -1.0469 0.0923725 -0.0164187 -1.25007 -0.481815 -0.582204 0.163657 0.867953 0.869949 -0.551438 -0.458272 0.0660124 -1.14261 1.65907 -0.2369 -0.997435 -0.0952726 0.89377 1.01454 -0.236411 0.553196 0.154934 -0.653818 1.0422 -0.842711 -0.705634 1.79741 0.0375392 -0.844775 0.281589 1.48117 -0.668131 -0.00936072 0.405825 -0.0640269 -0.521449 -0.710661 0.0812242 -1.43366 0.781751 0.307415 -1.96069 0.175185 -0.834108 0.562669 -1.11267 0.225003 0.0494112 -0.113228 -0.737181 0.517156 0.27629 -2.01072 0.548 0.113696 0.20527 -1.06622 0.285509 0.733295 -1.97158 0.463969 -1.01816 -0.13495 -0.599245 -0.0708429 -0.527434 0.241324 -0.237106 -0.585738 -0.927675 -1.83781 -0.0975834 -0.229163 1.10344 0.992267 -0.807182 0.366262 -0.0900305 -0.183515 0.0356093 -0.92432 -2.05609 -0.027872 1.36764 -0.502808 -1.3497 -0.436312 -1.1614 -1.24928 0.295046 2.11463 -0.204547 0.510229 -0.913338 -1.47305 -0.464439 1.41911 -0.322183 0.12843 0.0386554 0.403745 0.274886 -1.51517 -0.425126 -0.144974 -1.19989 -0.492305 0.971405 -0.980981 0.226729 -0.962253 -1.11271 0.0819307 0.180224 0.548219 -1.19641 0.969759 -1.63952 -0.976513 -3.16699 0.0459704 0.815024 0.271491 -1.74563 0.0700583 -0.448591 -0.0798372 0.249779 0.339626 -1.28199 1.61843 -0.308285 1.13836 0.511789 -0.54121 -1.19866 0.129429 -0.173854 -0.358251 -0.55828 0.750858 -1.22212 -0.295723 -0.398226 -1.39757 -0.706725 -0.404669 -0.468153 -0.50701 -1.29585 0.994023 0.0881927 0.860236 -1.8756 -1.63142 -1.72427 0.54279 0.5903 0.409991 0.903088 -0.161422 -0.387014 1.30063 -0.609124 -2.22828 -0.480831 -0.728926 -0.578115 0.646032 0.596666 0.608126 -1.54893 -0.295164 -0.143483 -0.21124 1.0247 -0.376839 -0.694354 -0.590893 -0.837382 -0.444814 0.45696 -1.73967 -1.02547 -0.0128817 -0.613873 1.30854 -1.10418 1.15423 -0.315316 -2.0799 -0.478532 0.742416 0.16364 -0.382626 0.31556 0.17189 -1.33718 0.810278 -0.0816591 1.00955 -0.331771 -1.36839 -0.527535 2.24079 0.205742 0.807573 0.76596 -0.908307 0.387546 0.645354 -0.278659 -0.255232 1.709 -1.56609 0.652879 -0.634007 -0.667714 -0.164865 1.16015 -0.158875 0.20278 0.217261 -1.09209 0.594923 -0.274994 1.25857 -1.2823 0.256867 0.405666 0.860042 -0.231883 -0.616637 0.971031 0.274744 -0.399103 1.43199 0.82662 0.514961 0.842954 0.437559 0.565982 1.1632 0.808834 -1.87737 -0.370885 0.337025 0.831043 0.688389 0.595813 -0.460657 0.272426 -0.758543 -0.905068 0.235525 1.89656 -0.080765 -0.640284 -0.219413 0.140819 0.330152 1.03225 0.451335 1.73993 -0.0900418 0.452018 1.93943 0.821069 0.124759 -0.521162 0.0419835 -1.08423 -0.665381 -0.554943 -1.32223 -0.157837 -0.142827 0.384018 -1.4739 -0.89246 1.26983 -1.2753 -0.218849 -0.0530898 0.160818 0.141393 -0.417113 0.170634 -0.897994 -0.680975 0.279027 0.422929 -1.10184 1.02312 1.24519 -1.03106 0.635554 -0.815788 -1.06496 0.587361 -0.0302848 0.299898 -0.664726 0.523555 0.196908 -0.257223 0.638833 -0.839395 -1.4021 0.634566 -0.30287 0.467005 -0.212698 -0.918735 1.35413 -0.619448 1.67605 0.720621 -0.854841 1.79506 -0.392298 1.11476 1.9123 0.170381 0.364481 0.335852 0.513654 -1.57827 0.772346 -0.12321 -0.401701 0.637391 -0.336177 -0.613069 0.384869 -1.57921 0.557065 -0.557831 0.0483764 0.0879191 0.771807 1.10381 -0.0495069 -1.01708 -0.398865 -1.0098 0.323706 0.564023 -0.507627 0.485689 0.271754 -1.05546 -1.04395 -1.02332 -0.158644 -1.89779 0.706836 0.404797 0.19262 1.45922 1.01591 0.318038 0.0846937 0.423788 1.67103 -0.377224 1.43248 -1.68664 0.0705906 -1.44281 1.61475 -1.77278 0.278337 -0.387826 -1.1676 0.47104 -0.0523239 1.73991 0.612604 -1.35199 1.32312 -0.670404 0.617603 -0.939258 -0.816016 -0.140886 -0.347843 -1.16134 -0.137986 1.10472 -2.10966 -1.57219 0.759368 -0.603187 -0.911311 2.07928 -0.109702 -1.4553 0.148527 0.926771 1.01266 -1.53547 -0.64518 0.545981 -0.848883 -0.0265279 -0.539555 -0.575804 -0.333965 -0.479666 -0.307982 -1.15911 0.547309 0.433744 0.580764 -0.618615 1.15845 0.935896 0.40413 1.23144 -0.650113 0.57803 1.08238 1.14253 1.28084 -0.200051 -1.02007 -2.45203 0.294061 -0.0764502 -0.959355 0.306128 -0.665136 0.536571 -0.443366 1.16706 -0.0970458 0.0426857 1.29729 -0.734713 -0.585136 0.69516 -0.124344 -0.414979 0.986587 -0.0787226 -1.21431 1.71252 0.291973 -0.117967 -0.167183 0.246977 0.334573 -1.06436 -0.935838 -1.05446 -0.100502 -0.358832 1.82056 1.11902 0.798935 1.96519 -0.898557 0.139052 -1.01531 -1.34037 -1.12453 1.7604 0.0943001 0.838453 0.28292 -0.139534 0.165661 1.37864 -0.180487 -0.728953 1.49748 -0.448245 0.497511 1.00661 -0.403384 0.792762 -0.389688 -0.77524 0.551162 -1.72863 -0.0433265 -0.177881 1.2181 1.0489 -1.43918 0.654463 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

[wangyinzhi1989]

Hi, me too... C++ interface, onnx model parse
My solution.
Check 1: see onnx.export dynamic_axes parameter. code eg.
input_names = ["input0"]
output_names = ['loc', 'conf']
# batch is dynamic input tensor
torch_out = torch.onnx._export(net, im_data, args.out_path, export_params=True, verbose=False,
input_names=input_names, output_names=output_names,
dynamic_axes={"input0":[0], "loc":[0], "conf":[0]})

Check 2:Refer to function networkToEngine in samples/common/sampleEngines.cpp. code eg.
// create config
TrtUniquePtr config{ builder.createBuilderConfig() };
auto profile = builder.createOptimizationProfile();
auto input = network.getInput(0);
DataType in_type = input->getType();
TensorFormats in_tf = input->getAllowedFormats();
bool in_st = input->isShapeTensor();
Dims dims = input->getDimensions();
// set dynamic input tensors
dims.d[0] = 1;
profile->setDimensions(input->getName(), OptProfileSelector::kMIN, dims);
dims.d[0] = std::max((params_.max_batch_size_ / 2), 1);
profile->setDimensions(input->getName(), OptProfileSelector::kOPT, dims);
dims.d[0] = std::max(params_.max_batch_size_, 1);
profile->setDimensions(input->getName(), OptProfileSelector::kMAX, dims);
// and optimization profile
config->addOptimizationProfile(profile);
config->setMaxWorkspaceSize(params_.max_workspace_size_);
if (params_.use_fp16_)
config->setFlag(BuilderFlag::kFP16);
cuda_engine_.reset(builder.buildEngineWithConfig(network, *config));
if (!cuda_engine_) {
exit(-1);
}

Check 3:Set the input size for the preprocessor. code eg.
Dims4 inputDims{ batch_size, input_dims_.d[1], input_dims_.d[2], input_dims_.d[3] };
// Set the input size for the preprocessor
cuda_context_->setBindingDimensions(0, inputDims);
// specify full dynamic input
if (!cuda_context_->allInputDimensionsSpecified()) {
return -1;
}
int input_count = std::accumulate(input_dims_.d + 1, input_dims_.d + input_dims_.nbDims, 1, std::multiplies<int64_t>());
NV_CUDA_CHECK(cudaMemcpyAsync(binding_buffers_.at(input_index_), input_blob.data,
batch_size * input_count * sizeof(float), cudaMemcpyHostToDevice, cuda_stream_));
// asynchronously enqueue the inference work
//cuda_context_->enqueue(batch_size, binding_buffers_.data(), cuda_stream_, nullptr);
if (!cuda_context_->enqueueV2(binding_buffers_.data(), cuda_stream_, nullptr)) {
return -1;
}

[rmccorm4]

Hi @dtiny ,

Can you share your inference code and model to reproduce this?

If your model has a fixed batch size of 1 and you parse the model using TensorRT's EXPLICIT_BATCH flag, then this is likely the reason for first sample's output being correct and other sample outputs being all zero.

[rmccorm4]

Hi @wangyinzhi1989 ,

Hi, me too... C++ interface, onnx model parse
My solution.

You mention a solution. Are you saying you have the same problem, or you already fixed it?

[zhongyunuestc]

hi, me too, but i use the tensorrt5, are you fixed it?
my code is following.
void doInference(IExecutionContext& context, const std::map<std::string, nvinfer1::Weights>& inCfg,
std::map<std::string, std::vector>& outCfg, const int batchSize, int verbose = 1)
{

const ICudaEngine& engine = context.getEngine();
const int numBindings = engine.getNbBindings();
if(numBindings != inCfg.size() + outCfg.size()){
    printf("numBindings error...\n");
    return;
}   
printf("batchSize: %d, max_batch %d\n", batchSize, engine.getMaxBatchSize());
std::vector<void*> buffers(numBindings);
allocBindingsFromWeights(engine, buffers, batchSize, inCfg, verbose);
allocBindingsFromVectors(engine, buffers, batchSize, outCfg, verbose);

void** bs = buffers.data();
//cudaProfilerStart();
cudaStream_t stream;
cudaStreamCreate(&stream);
copyToDeviceBindings(engine, buffers, batchSize, inCfg, stream);
context.enqueue(batchSize, bs, stream, nullptr);
//context.execute(batchSize, bs);
copyFromDeviceBindings(engine, buffers, batchSize, outCfg, stream);
cudaStreamSynchronize(stream);
//copyFromDeviceBindings(engine, buffers, batchSize, outCfg, stream);

/*if(cudaDeviceSynchronize()){
    printf("cudaDeviceSynchronize error...\n");
    return;
}*/
//cudaProfilerStop();

for (auto& devptr : buffers)
{   
    if(cudaFree(devptr)){
        printf("free error...\n");
        return;
    }   
}   

}

[zhongyunuestc]

Hi @wangyinzhi1989 ,

Hi, me too... C++ interface, onnx model parse
My solution.

You mention a solution. Are you saying you have the same problem, or you already fixed it?

i meet the same question, but i use the tensortrt5, do you have any suggestion?

[kingyj7]

Hi, I also met this problem with TensorRT7.1.

first sample's output being correct and other sample outputs being all zero.

I solved it just by setting large batch size (e.g 32) for input when exporting onnx model.
While for the rest steps, including build engine and inference are almost the same with the case of batch size = 1 except for providing the input with batch_size = 32 when inference.

[onnx20]

@dtiny @rmccorm4
hi, I have the same problem, and I use the tensorrt7.0, are you fixed it? but tensorrt6.0 is work.
Thanks!

[onnx20]

Hi @dtiny ,

Can you share your inference code and model to reproduce this?

If your model has a fixed batch size of 1 and you parse the model using TensorRT's EXPLICIT_BATCH flag, then this is likely the reason for first sample's output being correct and other sample outputs being all zero.

Hi, I set the parameter builder->setMaxBatchSize(2) and using EXPLICIT_BATCH flag, but the first sample's output being correct and other sample outputs being all zero. Should I need to set the other parameter?

[onnx20]

@rmccorm4

Hi, I set the parameter builder->setMaxBatchSize(2) and using EXPLICIT_BATCH flag, but the first sample's output being correct and other sample outputs being all zero. Should I need to set the other parameter?

[rmccorm4]

Hi @OYRQ ,

I set the parameter builder->setMaxBatchSize(2) and using EXPLICIT_BATCH flag

builder->setMaxBatchSize is only used/valid for implicit batch models. For explicit batch, the engine will use the batch size defined in the ONNX model. If the batch size is fixed in your ONNX model (1,16, 32, etc), then only those results will be correct, and anything else (if you created buffers larger than necessary), will have garbage data. If the batch size is dynamic in your ONNX model (-1, None, etc.), then you will need to create an optimization profile, and set the batch size at runtime using context.setBindingDimensions(...).

More details on this here: https://forums.developer.nvidia.com/t/tensorrt-7-onnx-models-with-variable-batch-size/115302/6?u=nves_r

but the first sample's output being correct and other sample outputs being all zero

This sounds like your ONNX model has a fixed batch size of 1. You can confirm this by viewing your ONNX model in a tool like Netron, and looking at the input shape. You'll need to export the ONNX model with a dynamic batch dimension, and then create an optimization profile if you want to do inference on variable batch sizes.

[onnx20]

Hi @rmccorm4 ,

I set the parameter builder->setMaxBatchSize(2) and using EXPLICIT_BATCH flag

builder->setMaxBatchSize is only used/valid for implicit batch models. For explicit batch, the engine will use the batch size defined in the ONNX model. If the batch size is fixed in your ONNX model (1,16, 32, etc), then only those results will be correct, and anything else (if you created buffers larger than necessary), will have garbage data. If the batch size is dynamic in your ONNX model (-1, None, etc.), then you will need to create an optimization profile, and set the batch size at runtime using context.setBindingDimensions(...).

More details on this here: https://forums.developer.nvidia.com/t/tensorrt-7-onnx-models-with-variable-batch-size/115302/6?u=nves_r

but the first sample's output being correct and other sample outputs being all zero

This sounds like your ONNX model has a fixed batch size of 1. You can confirm this by viewing your ONNX model in a tool like Netron, and looking at the input shape. You'll need to export the ONNX model with a dynamic batch dimension, and then create an optimization profile if you want to do inference on variable batch sizes.

Thank you very much. I use the batch size ONNX model. It's working