Does the tvm based CPU get accelerated?

[lucasjinreal]

Hi, just wonder tvm based deployed model can get accelerated or not?
Compare with vanilla pytorch CPU or onnxruntime or OpenVINO?

[zhiqwang]

Hi @jinfagang ,

Thanks for your attention here!

At that time, my first plan is to let each side could compile and infer successfully, and because of time limited, I have not carefully compared the time of each side between libtorch, onnxruntime and tvm. (I have the plan to support openvino, also for the time limited, I haven't done this now).

Now, I'm writing and refactoring the trainer, after this is done, I will add the documents of time consuming and comparison.

BTW, we are welcome for contributions in any ways.

[zhiqwang]

And I've uploaded the notebooks of tvm compiling and inference here, within my current experience, the procedure of compiling will consume much more time than libtorch or onnxruntime. the procedure of inference seems normal (Edit: more experiments needed here)

This repo is my first glance on tvm, and I will do more experiments on it.

[lucasjinreal]

@zhiqwang That's weried, tvm should be faster if compares on same CPU device? at least should faster than onnxruntime if chosen CPU as provider.

[zhiqwang]

tvm should be faster if compares on same CPU device? at least should faster than onnxruntime if chosen CPU as provider.

Hi @jinfagang , I agree with you on this point, and this is my goal. The current realization on the tvm backend is only the initial attempt, let us do more efforts to achieve this goal!

[zhiqwang]

Hi @jinfagang

I've added a rough comparison of inference time consumed on Jupyter notebook (iPython).

• On the ONNXRuntime backend,

  CPU times: user 2.04 s, sys: 0 ns, total: 2.04 s
  Wall time: 55.8 ms

• On the TorchScript backend,

  CPU times: user 2.03 s, sys: 32 ms, total: 2.06 s
  Wall time: 60.5 ms

• On the PyTorch backend,

  CPU times: user 3.87 s, sys: 60 ms, total: 3.93 s
  Wall time: 116 ms

• On the TVM backend,

  CPU times: user 528 ms, sys: 364 ms, total: 892 ms
  Wall time: 22.3 ms

You could check the latest updated notebook for more details.

BTW, the displayed time of the onnxruntime notebook is on GPU, I just test it locally.

Although this comparison is a bit rough, we could come to this conclusion that tvm will increase inference speed on CPU device.

So I'll close this issue. If you have more concerns please let me know.