Repository navigation
[Question] FHE inference over a single image time #658
Description
Activity
Hi,
One possible way to reduce latency is to use a bigger machine, since Concrete makes very good use of parallelism: we obtain around 40 seconds inference time on that model using the
hpc7ainstances on AWS.A second possible approach is, depending on the use-case, to optimize your model by making it smaller or pruning it. You might want to look into structured pruning. See this section of the documentation about such techniques - some are already used in the CIFAR example that you link.
Reacted by MelancianiHi andrei, thanks for your quick reply. I am using a high end workstation with 64 GB RAM and 24 core processor.
I will take a look at the pruning.
Thanks
Hi, I found this community very helpful. I have a question related to inference time.
I am running following use case example.
https://github.com/zama-ai/concrete-ml/blob/main/use_case_examples/cifar/cifar_brevitas_training/evaluate_one_example_fhe.py
The issue is that FHE inference over a single image took more than 28 min.
Is there a way to optimize it or reduce the inference time.