trtexec: onnx to tensorRT convertion fails #3894
Assignees
Labels
triaged
Issue has been triaged by maintainers
Comments
|
Hi, if you want to modify your onnx model, onnx-graphsurgeon is probably your best bet. Examples section shows how to use it. Btw, the source for onnx-tensorrt is open here. Given that you're using a slightly older version of TRT, the line numbers (and even filenames) may not exactly match. But, you can see a couple of places where such errors are logged. :) |
|
|
|
Hi, if you're open to sharing the onnx file, please consider doing so. Sorry, I'm not familiar with the term 'opcode'. You could point me to something that'll help me understand. In the meantime, you can also dig deeper using these pointers:
Will await you sharing your observations. |
|
here the requested infos:
|
|
@brb-nv Hi, I meet the similar problem, could you please help me? Thanks a lot! I use trtexec to conduct int8 calibration and quantilize like this: The calibration cache file was generated with polygraphy API. But when I run the code above, it give the following error: What's wrong with it? In fact, I have tried to conduct quantilize directly with polygraphy API, but the engine size is not as small as we expected, only from 156M (FP32 engine) to 95M, what's more, the inference speed have no improvement compared with FP32 engine. But when I tried to generate INT8 engine with trtexec tool (with no calibration), the engine size become 51M, so I want to conduct int8 calibration and quantilize with trtexec tools to check again. So, can I use trtexec tool and the calibration cache file to achieve this goal? If it works, what's wrong with my code? I'm looking froward to your help, many thanks. |
Yes, TRT’s primary means of importing a trained model from a framework is through the ONNX interchange format. Yes, you'll need to remove the unsupported part of the network for the engine to be built. Also, I don't think Unsure about the issue with polygraphy. Please try out our latest release. |
|
@yjiangling kindly open a separate issue. It's quite different from OP's issue. |
|
@geraldstanje What is your |
|
@lix19937 im using the build in function of the huggingface lib which calls torch.onnx.export - see: https://github.com/huggingface/setfit/blob/main/src/setfit/exporters/onnx.py#L183 here it calls the torch.onnx.export: https://github.com/huggingface/setfit/blob/main/src/setfit/exporters/onnx.py#L96-L103 |
The root cause is in Or you modify the forward code(exclude ArrayFeatureExtractor), then re-export onnx. |
installed packages:
|
It's normal, because you do not set those features.
Not match env, use follow cmd polygraphy convert removed.onnx -o model_poly.plan
polygraphy run model_poly.plan --trt
polygraphy run removed.onnx --trt \
--data-loader-script data_loader.py
trex
polygraphy run removed.onnx --trt --onnxrt --fp16 |
|
trtexec:
polygraphy convert:
polygraphy run with tensorrt:
polygraphy run for onnx:
|
|
You should ensure that build and run the engine in the same enviroment. This kind of error is usually due to different tensorrt versions. The tensorrt_version of your polygraphy not match your tensorrt_version of trtexec. If they are the same tensorrt version, the plans are the same. |
|
i run trtexec and polygraphy (installed via pip install polygraphy) in the same docker container - using: |
Ok, thank you. I have open an new issue in here: https://github.com/NVIDIA/TensorRT/issues/3902 |
You can and |
|
Your cuda runtime/drive env not match v1001 requirement, see https://github.com/NVIDIA/TensorRT?tab=readme-ov-file#prerequisites.
TensorRT’s primary means of importing a trained model from a framework is through the ONNX interchange format. TensorRT ships with an ONNX parser library to assist in importing models. Where possible, the parser is backward compatible up to opset 9; the ONNX Model Opset Version Converter can assist in resolving incompatibilities.
Your need know one thing, three tool or process all need libnvinfer.so or libnvinfer.a, if you make sure use the same version so, then pass. |
ok that worked (used docker image: nvcr.io/nvidia/pytorch:24.05-py3)
supported onnx opset for tensorrt 8.6.3 (https://github.com/onnx/onnx-tensorrt/blob/6872a9473391a73b96741711d52b98c2c3e25146/docs/operators.md
polygraphy:
|
This generally means all opset versions [0, 17] of a certain op is supported. For example, if you look at operators.md, BatchNorm has been updated in opsets 15, 14, 9, 7, 6, 1. Given the statement above, ideally, BatchNorm of all those opsets must be supported by TRT.
I'd try to export with opset 17 and keep in mind any gaps in TRT support by looking at the onnx2trt support matrix.
I think trtexec (with --verbose option) shows everything you're looking for. Polygraphy is more suitable for accuracy debugging and not as much for measuring performance.
I can see this with trtexec with --verbose option. |
|
i have a model.plan - i dont know which settings (input shape, output shape, batch_size) i defined for trtexec - how an i figure it out? can i load the model.plan with polygraphy or another tool to get infos/ als is it possible to write the tokenizer in c++ (for huggingface sentence transformer model) for triton inference server? |
|
Use |
|
@geraldstanje ref https://github.com/lix19937/trt-samples-for-hackathon-cn/blob/master/cookbook/01-SimpleDemo/TensorRT8.5/main.cpp use cpp load plan then infer. |
|
@lix19937 whats the diff between the main.cpp and the |
|
The effect is basically the same,
It’s useful for benchmarking networks on random or user-provided input data.
|
|
@lix19937 what is the purpose of workspace? can workspace still be used? i did compare the tensorrt python lib with trtexec:
python version: bash version: tensorrt lib: more infos: more infos: |
|
Let's focus on issues, what is your problem about trtexec convert onnx ? If you use trtexec, add |
|
@yjiangling Is there any code that will generate the calibration.cache file? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
hi, i have an onnx model which i want to convert using trtexec:
it seems the onnx model is currently not supported with those datatypes - how to convert (what tool + settings) the model so tensorRT can use it?
Environment
TensorRT Version: 8.6.1
NVIDIA GPU: Nvidia T4
NVIDIA Driver Version:
CUDA Version: 12.x
CUDNN Version:
Operating System: Ubuntu 20.04
Python Version (if applicable):
Tensorflow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if so, version):
Relevant Files
Model link:
Steps To Reproduce
pip list:
Commands or scripts:
trtexec output:
trtexec output with verbose:
Have you tried the latest release?:
why polygraphy wants to load a cuda 11.x library? i run this inside nvcr.io/nvidia/pytorch:24.03-py3 docker:
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (
polygraphy run <model.onnx> --onnxrt):cc @pranavm-nvidia @sachanub
The text was updated successfully, but these errors were encountered: