Use Encrypted ONNX model in Triton Server

[4BD0U]

I have an encrypted ONNX model using AES and I would like to use it with a Dockerized Triton server. Is it possible to load an encrypted ONNX model into Triton, and if so, how can I do it? Can I use a custom Python backend for decryption?

[dyastremsky]

If OnnxRuntime can't run the encrypted model, the default ORT backend can't either. You'd need to create a custom backend.

CC: @Tabrizian in case I'm missing anything Python backend-specific. There might be a way to do custom logic for decryption.

[Tabrizian]

@4BD0U You could also use the custom repository agent to decrypt your model before loading it in Triton: https://github.com/triton-inference-server/server/blob/da2cabd2a44ffc51af1107e5fec4f9ff8c8795de/docs/customization_guide/repository_agents.md#repository-agent