TensorRT cannot support nonzero，Is there any other alternative to this function?

[puallee]

the code in Pytorch is ：indices = mask.nonzero(as_tuple=False)[:, 0]，
the error in TensorRT is：
[08/23/2022-10:06:28] [I] [TRT] No importer registered for op: NonZero. Attempting to import as plugin.
[08/23/2022-10:06:28] [I] [TRT] Searching for plugin: NonZero, plugin_version: 1, plugin_namespace:
[08/23/2022-10:06:28] [E] [TRT] ModelImporter.cpp:773: While parsing node number 84 [NonZero -> "onnx::Transpose_634"]:
[08/23/2022-10:06:28] [E] [TRT] ModelImporter.cpp:774: --- Begin node ---
[08/23/2022-10:06:28] [E] [TRT] ModelImporter.cpp:775: input: "onnx::NonZero_629"
output: "onnx::Transpose_634"
name: "NonZero_559"
op_type: "NonZero"

[08/23/2022-10:06:28] [E] [TRT] ModelImporter.cpp:776: --- End node ---
[08/23/2022-10:06:28] [E] [TRT] ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4890 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[08/23/2022-10:06:28] [E] Failed to parse onnx file
[08/23/2022-10:06:28] [I] Finish parsing network model
[08/23/2022-10:06:28] [E] Parsing model failed
[08/23/2022-10:06:28] [E] Failed to create engine from model or file.
[08/23/2022-10:06:28] [E] Engine set up failed

[zerollzeng]

It's in our plan but seems we don't have a target release now. @kevinch-nv for more information.

[puallee]

I use a torch. where(mask!=0), but it still has an error of name: "NonZero_559"

[puallee]

https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md On this website, tensor-RT support torch.where

[yaliqin]

I met similar error when converting ONNX model generated with Huggingface optimum library.
1, The model is bart-lage-mnli
2, I convert it to ONNX using below scripts

import os
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model_checkpoint = "../models/bart-large-mnli"
save_directory = "../models/onnx/"
file_name = "model.onnx"
onnx_path = os.path.join(save_directory, "model.onnx")

# Load a model from transformers and export it through the ONNX format
model = ORTModelForSequenceClassification.from_pretrained(model_checkpoint, from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

# Save the onnx model and tokenizer
model.save_pretrained(save_directory, file_name=file_name)
tokenizer.save_pretrained(save_directory)

3, I convert the ONNX model with "trtexec" command using container with docker image "nvcr.io/nvidia/pytorch:22.08-py3"
Then I met the error message as below

[libprotobuf WARNING /home/jenkins/agent/workspace/OSS/OSS_L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING /home/jenkins/agent/workspace/OSS/OSS_L0_MergeRequest/oss/build/third_party.protobuf/src/third_party.protobuf/src/google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1629756655
[09/12/2022-23:47:50] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:367: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[09/12/2022-23:47:50] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/12/2022-23:47:50] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/12/2022-23:47:50] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/12/2022-23:47:50] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/12/2022-23:47:55] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/12/2022-23:47:55] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
[09/12/2022-23:48:20] [I] [TRT] No importer registered for op: NonZero. Attempting to import as plugin.
[09/12/2022-23:48:20] [I] [TRT] Searching for plugin: NonZero, plugin_version: 1, plugin_namespace: 
[09/12/2022-23:48:20] [E] [TRT] parsers/onnx/ModelImporter.cpp:773: While parsing node number 4422 [NonZero -> "onnx::Transpose_5795"]:
[09/12/2022-23:48:20] [E] [TRT] parsers/onnx/ModelImporter.cpp:774: --- Begin node ---
[09/12/2022-23:48:20] [E] [TRT] parsers/onnx/ModelImporter.cpp:775: input: "onnx::NonZero_5794"
output: "onnx::Transpose_5795"
name: "NonZero_4422"
op_type: "NonZero"

[09/12/2022-23:48:20] [E] [TRT] parsers/onnx/ModelImporter.cpp:776: --- End node ---
[09/12/2022-23:48:20] [E] [TRT] parsers/onnx/ModelImporter.cpp:778: ERROR: parsers/onnx/builtin_op_importers.cpp:4890 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[09/12/2022-23:48:20] [E] Failed to parse onnx file
[09/12/2022-23:48:20] [I] Finish parsing network model
[09/12/2022-23:48:20] [E] Parsing model failed
[09/12/2022-23:48:20] [E] Failed to create engine from model or file.
[09/12/2022-23:48:20] [E] Engine set up failed

Is there any way to work around this?

[puallee]

@kevinch-nv for more information.

[zerollzeng]

https://github.com/onnx/onnx-tensorrt/blob/main/docs/operators.md On this website, tensor-RT support torch.where

it's not torch.where, it's ONNX spec. please check your exported ONNX model.

[puallee]

OK， if I want to use： indices = mask.nonzero(as_tuple=False)[:, 0]， what can I do？

[nvpohanh]

Non-Zero has been supported in TRT 8.5 GA, I think

[ttyio]

Closing since no activity for more than 3 weeks, please reopen if you still have question, thanks!

[Kingxudong]

I didn't use torch.where and nonzero in my code, but I still had the same error!! Why??