Proposal for Integration of Gemma Model into Jan Project

[jerry-f]

Dear Jan Project Team,

I hope this message finds you well. I am reaching out to propose the integration of the Gemma model into the Jan project. As Gemma has been gaining traction for its exceptional capabilities in areas such as natural language processing and image generation, I believe that incorporating it could significantly enhance the Jan project, especially in terms of processing efficiency and user engagement.

[chenshaoju]

2024-02-22T02:53:20.198Z [NITRO]::Error: llama_model_load: error loading model: unknown model architecture: 'gemma'
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'E:\AI\jan\datafolder\models\gemma-7b.Q4\gemma-7b.Q4_K_M.gguf'

2024-02-22T02:53:20.206Z [NITRO]::Debug: Load model success with response {}
2024-02-22T02:53:20.207Z [NITRO]::Debug: {"timestamp":1708570400,"level":"ERROR","function":"load_model","line":560,"message":"unable to load model","model":"E:\\AI\\jan\\datafolder\\models\\gemma-7b.Q4\\gemma-7b.Q4_K_M.gguf"}
20240222 02:53:20.198000 UTC 8276 ERROR Error loading the model - llamaCPP.cc:565

[jerry-f]

I have tried to add the model manually, but it has not been able to start,
error "Apologies, something's amiss!

Jan's in beta. Find troubleshooting guides here or reach out to us on Discord for assistance.”

[Van-QA]

Gemma will be supported after this PR is merged janhq/cortex.cpp#446

[lededev]

@Van-QA I think v0.4.7 is already merged your PR, are there any example of config file to add gemma-7b and gemma-7b-it?

[Van-QA]

hi @lededev, we tested the gemma using Jan nightly build ✅ https://discord.com/channels/1107178041848909847/1209906514069028885
Please follow this guideline to try it out on our latest nightly build 🙏
https://jan.ai/guides/using-models/import-manually/

[chenshaoju]

Version 0.4.7-293 with the Gemma model is working now.

Thanks.✋

[lededev]

Jan v0.4.7-293 also work for me.
@chenshaoju 能把你model.json贴一下吗？我用的是gemma-7b-it.gguf 34.2GB，那个模型太大了，用RTX 4090只有0.62t/s 。

[chenshaoju]

@lededev 我用的是一个第三方版本，仅供参考： https://huggingface.co/mlabonne/gemma-7b-it-GGUF 。

以下是 model.json ：

{
  "object": "model",
  "version": 1,
  "format": "gguf",
  "sources": [
    {
      "url": "gemma-7b-it.Q4_1.gguf",
      "filename": "gemma-7b-it.Q4_1.gguf"
    }
  ],
  "id": "gemma-7b-it.Q4_1",
  "name": "gemma-7b-it.Q4_1",
  "created": 1709277804515,
  "description": "",
  "settings": {
    "ctx_len": 4096,
    "embedding": false,
    "prompt_template": "{system_message}\n### Instruction: {prompt}\n### Response:",
    "llama_model_path": "gemma-7b-it.Q4_1.gguf"
  },
  "parameters": {
    "temperature": 0.7,
    "top_p": 0.95,
    "stream": true,
    "max_tokens": 2048,
    "stop": [
      "<endofstring>"
    ],
    "frequency_penalty": 0,
    "presence_penalty": 0
  },
  "metadata": {
    "size": 5496286176,
    "author": "User",
    "tags": []
  },
  "engine": "nitro"
}

[tsjslgy]

So when can use gemma, 4.8 ?