How to run FLOAT16 OnnxRuntime models

[zaobao]

It looks like FLOAT16 has not been supported by now

Caused by: java.lang.UnsupportedOperationException: type is not supported: FLOAT16
at ai.djl.onnxruntime.engine.OrtUtils.toDataType(OrtUtils.java:101)
at ai.djl.onnxruntime.engine.OrtNDArray.getDataType(OrtNDArray.java:65)
at ai.djl.onnxruntime.engine.OrtNDArray.toByteBuffer(OrtNDArray.java:121)
at ai.djl.pytorch.engine.PtNDManager.from(PtNDManager.java:55)
at ai.djl.pytorch.engine.PtNDManager.from(PtNDManager.java:31)
at ai.djl.ndarray.NDArrayAdapter.getAlternativeArray(NDArrayAdapter.java:1315)
at ai.djl.ndarray.NDArrayAdapter.split(NDArrayAdapter.java:876)
at ai.djl.ndarray.NDArray.split(NDArray.java:3173)
at ai.djl.translate.StackBatchifier.unbatchify(StackBatchifier.java:118)
at ai.djl.huggingface.translator.CrossEncoderBatchTranslator.processOutput(CrossEncoderBatchTranslator.java:60)
at ai.djl.huggingface.translator.CrossEncoderBatchTranslator.processOutput(CrossEncoderBatchTranslator.java:30)
at ai.djl.inference.Predictor.batchPredict(Predictor.java:173)
... 5 more

[frankfliu]

You can convert model to fp16, but you need to CUDA device. You can use the following command:

djl-convert -o model -f OnnxRuntime -m <MODEL_ID> --optimize O4 --device cuda

see: https://github.com/deepjavalibrary/djl/tree/master/extensions/tokenizers#use-command-line

[zaobao]

You can convert model to fp16, but you need to CUDA device. You can use the following command:

djl-convert -o model -f OnnxRuntime -m <MODEL_ID> --optimize O4 --device cuda

see: https://github.com/deepjavalibrary/djl/tree/master/extensions/tokenizers#use-command-line

I converted the model to fp16 and encountered an exception while loading fp16 model with CrossEncoderBatchTranslator

Caused by: java.lang.UnsupportedOperationException: type is not supported: FLOAT16
at ai.djl.onnxruntime.engine.OrtUtils.toDataType(OrtUtils.java:101)
at ai.djl.onnxruntime.engine.OrtNDArray.getDataType(OrtNDArray.java:65)
at ai.djl.onnxruntime.engine.OrtNDArray.toByteBuffer(OrtNDArray.java:121)
at ai.djl.pytorch.engine.PtNDManager.from(PtNDManager.java:55)
at ai.djl.pytorch.engine.PtNDManager.from(PtNDManager.java:31)
at ai.djl.ndarray.NDArrayAdapter.getAlternativeArray(NDArrayAdapter.java:1315)
at ai.djl.ndarray.NDArrayAdapter.split(NDArrayAdapter.java:876)
at ai.djl.ndarray.NDArray.split(NDArray.java:3173)
at ai.djl.translate.StackBatchifier.unbatchify(StackBatchifier.java:118)
at ai.djl.huggingface.translator.CrossEncoderBatchTranslator.processOutput(CrossEncoderBatchTranslator.java:60)
at ai.djl.huggingface.translator.CrossEncoderBatchTranslator.processOutput(CrossEncoderBatchTranslator.java:30)
at ai.djl.inference.Predictor.batchPredict(Predictor.java:173)
... 5 more

    public static DataType toDataType(OnnxJavaType javaType) {
        switch (javaType) {
            case FLOAT:
                return DataType.FLOAT32;
            case DOUBLE:
                return DataType.FLOAT64;
            case INT8:
                return DataType.INT8;
            case UINT8:
                return DataType.UINT8;
            case INT32:
                return DataType.INT32;
            case INT64:
                return DataType.INT64;
            case BOOL:
                return DataType.BOOLEAN;
            case UNKNOWN:
                return DataType.UNKNOWN;
            case STRING:
                return DataType.STRING;
            default:
                throw new UnsupportedOperationException("type is not supported: " + javaType);
        }
    }

[zaobao]

I added FLOAT16 in OrtUtils.toDataType and disabled CrossEncoderBatchTranslator.sigmoid(PyTorchLibrary doesn't support fp16 sigmoid op with cpu), and the problem was solved

public static DataType toDataType(OnnxJavaType javaType) {
        switch (javaType) {
            case FLOAT:
                return DataType.FLOAT32;
            case FLOAT16:
                return DataType.FLOAT16;
            case DOUBLE:
                return DataType.FLOAT64;
            case INT8:
                return DataType.INT8;
            case UINT8:
                return DataType.UINT8;
            case INT32:
                return DataType.INT32;
            case INT64:
                return DataType.INT64;
            case BOOL:
                return DataType.BOOLEAN;
            case UNKNOWN:
                return DataType.UNKNOWN;
            case STRING:
                return DataType.STRING;
            default:
                throw new UnsupportedOperationException("type is not supported: " + javaType);
        }
    }