[Feature request] Utilizing local LLM model such as LLaMA.cpp or Baichuan

[maltoseslope]

Hi! Thanks for the great work, this is super helpful.

Considering the availability of LLMs on local machines, especially with the ability to quantify LLMs (such as LLaMA.cpp https://github.com/ggerganov/llama.cpp), LLMs can be relatively small and work on local computers and it would be good if this plugin can also support that, so we can run the translation on local machine without using these online translation services. Thanks for your work again!

[bookfere]

Thank you for your suggestion. I apologize for my lack of experience in the area of large language models. Could you please clarify if it's possible to utilize a web API from the project you mentioned?

[anartigone]

There are many of LLMs having the potential of integrating with ETCP to do EN-CHN translation, such as ChatGLM, Baichuan, InternLM, XVERSE and Bloom.
I have been attempting to experiment on this for a long time but currently too busy to start one more side project.
My theory is, ChatGLM may be the easiest one to start with. Because there are many similar applications out there.
According to chatglm.cpp, it provides an OpenAI compatible API. So theoretically we can simply use ChatGLM as GPT3.5 based on existing custom engine template.
But I haven't got a chance to try that. My further plan is to train a LoRA, which is based on my glossary, so that I can make a dedicated translation model with my own flavor in it.

[bookfere]

LLaMA.cpp provides an HTTP API server that gives users the ability to interact with it. You can use the custom engine feature to invoke its endpoints. Here is a recipe you can refer to:

{
    "name": "LLaMA.cpp",
    "languages": {
        "source": {
            "German": "de"
        },
        "target": {
            "English": "en"
        }
    },
    "request": {
        "url": "http://127.0.0.1:8080/completion",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json"
        },
        "data": {
            "prompt": "Translate the content from <slang> to <tlang>: <text>",
            "n_predict": 128
        },
    "response": "response['content']"
}

Please refer to its documentation for more information.