Llama.cpp not working and got errors (doc is correct???)

[kun432]

Describe the bug

I want to use Code Llama via llama.cpp's HTTP server. Setting up accoringly to the doc, but got 2 errors:

Traceback (most recent call last):

  File "continuedev/src/continuedev/core/autopilot.py", line 368, in _run_singular_step
    observation = await step(self.continue_sdk)

  File "continuedev/src/continuedev/core/main.py", line 359, in __call__
    return await self.run(sdk)

  File "continuedev/src/continuedev/plugins/steps/chat.py", line 98, in run
    async for chunk in generator:

  File "continuedev/src/continuedev/libs/llm/llamacpp.py", line 110, in _stream_chat
    args["max_tokens"],

KeyError: 'max_tokens'

Traceback (most recent call last):

  File "continuedev/src/continuedev/libs/util/create_async_task.py", line 21, in callback
    future.result()

  File "asyncio/futures.py", line 201, in result

  File "asyncio/tasks.py", line 232, in __step

  File "continuedev/src/continuedev/core/autopilot.py", line 510, in create_title
    title = await self.continue_sdk.models.medium._complete(

  File "continuedev/src/continuedev/libs/llm/llamacpp.py", line 173, in _complete
    json={"prompt": prompt, **self._transform_args(args)},

AttributeError: 'LlamaCpp' object has no attribute '_transform_args'

To Reproduce
Steps to reproduce the behavior:

1. Run llama.cpp server on remote Linux server.

$ ./server -c 16384 --host 0.0.0.0 -t 8 --mlock -m ./models/codellama-34b-instruct.Q4_K_M.gguf -b 512 -ngl 51

2. edit config.py. Only the following points changed from the default:

(snip)
from continuedev.src.continuedev.libs.llm.llamacpp import LlamaCpp     # doc said "GGML" but If so, got NameError 

config = ContinueConfig(
    ...
    models=Models(
        default=LlamaCpp(
            max_context_length=16384,
            server_url="http://192.168.1.1:8080"
        )
    ),
(snip)

3. Run VSCode and ask Continue something like this:

Generate code in python to calculate the Fibonacci sequence below 100.

4. Got errors.

Environment

• Operating System: Mac and Linux
• Python Version: 3.11.4 on Mac
• Continue Version: v0.0.358

Logs

See above.

Additional context

I also read #410 and tried some changes but no lucks. I just want to know the correct way to set config for Llama.cpp.

[sestinj]

Thanks for sharing these, it's nothing related to the config, these were bugs, and I've been able to fix them already. I will upload a new version very soon and let you know when it's ready

[sestinj]

@kun432 The newest version is ready, if you update the extension to v0.0.359 this problem should no longer occur

[kun432]

@sestinj Thank you! I updated the latest version and errors gone. but no output...

but I added llama_cpp_args to my config and it works!

    models=Models(
        default=LlamaCpp(
            max_context_length=16384,
            server_url="http://192.168.1.1:8080",
            llama_cpp_args={'stop': ['[INST]']}
        )
    ),

is this the expecetd behavior? If so, doc should be updated.

[sestinj]

Looks like this was because of a mistake I made in the default llama_cpp_args, and so setting them overrode the mistake. So no, you should not have to do this. I'll make an update with the fix. Thanks for catching this!

[sestinj]

New version is ready, should be able to leave out llama_cpp_args now

[kun432]

Great, works in v0.0.364! will explore other features in Continue!

Thanks!

[sestinj]

awesome! let me know what other questions you run into. Feel free to use Discord if it's ever more convenient: https://discord.gg/NWtdYexhMs