Support for ggerganov/llama.cpp

[ricklamers]

Would be cool if it could support https://github.com/ggerganov/llama.cpp

Is it on your roadmap?

[ricklamers]

Maybe using https://cocktailpeanut.github.io/dalai/#/

[cogentapps]

Integrating with the Dalai API should be doable.

How well does Llama perform in a chat style / conversational format?

[schmidp]

with https://alpaca-ai-custom5.ngrok.io it seems to perform well

[rounakdatta]

Alpaca is explicitly trained for Instruct-style messages. LLAMA by default isn't ready for conversations (yet) without some clever prompt engineering.

[cogentapps]

I'd like to implement this via Dalai's API which now supports Alpaca, but this issue needs resolved first:

cocktailpeanut/dalai#51

It's also a bit slow for longer chats. Ideally it would use llama.cpp's interactive mode instead of resending the whole context with each message, but I don't think this is implemented yet?