Why use ggml when you have libtorch and flashlight in C++?

[deepwilson]

Why use ggml when you have libtorch and flashlight in C++? I don't understand inspiration behind so many frameworks trying to achieve the same goal?

[cmp-nct]

Torch is an abomination, it helps getting started, can do everything but it's a monster of a code project. It gives you a ton of options while also taking a ton of liberties.
Understanding the entirety of libtorch &co is going to take you a serious amount of time, more than you might be willing to invest.

ggml is leightweight, adaptable and can easily be tuned to your needs. It took me less than a year half-hearted approach to understand every bit of the entire codebase. I would likely have quit in frustration trying the same on torch.

It is hard to tell how ggml will continue to develop but it might end up as a professional replacement option

[deepwilson]

What about flashlight? Have you tried it?

[cmp-nct]

I never looked in detail at it, I just gave it another glance and it looks like a nice machine learning library but llama or falcon are not even mentioned. Also no blib, clip, llava etc.
I've not even found a discussion to run anything modern on it.

[Nick-infinity]

Hello ,@cmp-nct I am trying to understand the opencl implementation of llama cpp. The code seems very complex to me. How can I get started to optimize the opencl implementations?

[FSSRepo]

@Nick-infinity Indeed, the implementation of the OpenCL backend is very much neglected. Direct testing is not possible, and it lacks many kernels. The reason for this neglect is not known.

[Nick-infinity]

Yes @FSSRepo . I have understood the flow and it can be improved quite a lot specially in terms of memory management. I am working on it though