JSONDecodeError

[corv89]

Describe the bug
I've setup ggml in config.py like so,

        default=GGML(
            max_context_length=2048,
            server_url="http://192.168.1.19:8000"
        )

To Reproduce
Steps to reproduce the behavior:

1. Write a prompt and click on continue
2. See error

Expected behavior
Expected a response (API is confirmed to be working)

Environment

• GGUF model file
• Operating System: Linux and MacOS
• Python Version: recent
• Continue Version: latest

Console logs

Traceback (most recent call last):

  File "continuedev/src/continuedev/core/autopilot.py", line 367, in _run_singular_step
    observation = await step(self.continue_sdk)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "continuedev/src/continuedev/core/main.py", line 359, in __call__
    return await self.run(sdk)
           ^^^^^^^^^^^^^^^^^^^

  File "continuedev/src/continuedev/plugins/steps/chat.py", line 85, in run
    async for chunk in generator:

  File "continuedev/src/continuedev/libs/llm/ggml.py", line 112, in stream_chat
    async for chunk in generator():

  File "continuedev/src/continuedev/libs/llm/ggml.py", line 103, in generator
    yield json.loads(chunk[6:])["choices"][0][
          ^^^^^^^^^^^^^^^^^^^^^

  File "json/__init__.py", line 346, in loads

  File "json/decoder.py", line 337, in decode

  File "json/decoder.py", line 355, in raw_decode

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[sestinj]

Are you connecting to a llama-cpp-python server that you are running? If you could share some details about how this is set up, I'll try to reproduce.

[redtier100x]

I am also having this problem.

• I have llama.cpp running in docker locally.
• I'm using VSCodium.
• I downloaded the latest VSIX file and installed that way.

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[corv89]

Are you connecting to a llama-cpp-python server that you are running? If you could share some details about how this is set up, I'll try to reproduce.

Yes, in llama.cpp I'm running ./server -m ../phind-codellama-34b-v1.Q4_K_S.gguf -c 2048 --port 8000 --host 0.0.0.0 -ngl 40

I've tested this on the web app and with curl, /completions works.

My config.py looks like this:

from continuedev.src.continuedev.libs.llm.ggml import GGML

config = ContinueConfig(
    models=Models(
        default=GGML(
            max_context_length=2048,
            server_url="http://192.168.1.19:8000"
        )
    ),

Could it be that llama.cpp made some changes recently?

[corv89]

Okay I see this is my mistake, I'm using the server inside llama.cpp when you meant a third-party project called llama-cpp-python!

[sestinj]

@corv89 yes, that was the intention. I'm working right now though on support for the direct llama.cpp server like you were running. This has several benefits, including tighter control of parameters like grammar

@redtier100x I will share again here once I push the update tomorrow, assuming you'd still prefer to use llama.cpp directly

[corv89]

I would much prefer to run llama.cpp since that works well out of the box (even for AMD) whereas I have trouble compiling llama_cpp_server.

Thanks, looking forward to it!

[fuyunzhishang]

I also have the problem, when i run llma on toghther ai ,

[corv89]

That looks like a different problem. Maybe create a new issue to track the bug?

[corv89]

@sestinj I found a way to solve this

llama.cpp includes a proxy for OpenAI compatible completions:
python ./examples/server/api_like_OAI.py

Using this in conjunction with ./server -m model-name.gguf works!

Simply point Continue at the python API and the python API at the llama C++ server

[sestinj]

@fuyunzhishang I just solved this problem, it should work now! Thanks for reporting. Let me know if it doesn't

[sestinj]

@corv89 nice find! LlamaCpp class is also ready if you'd like to use that instead. I haven't documented it yet, but you should now see the option in the model drop-down and it will connect to llama.cpp's ./server

(llama.cop in the drop-down, and the class is called LlamaCpp)

[changeling]

@sestinj When will this be available through the VS Code Extension update flow?

[corv89]

@changeling It's already there, just reload.

@sestinj Can't quite figure out how to configure this backend, perhaps I'll have to wait for the docs to set it up directly.

Thanks for bringing this feature out so quickly

[sestinj]

@changeling Version 0.0.344 is the latest version if that's helpful

I'm setting it up by running ./server -m models/codellama-7b.Q4_K_M.gguf -c 2048 from the root of the llama.cpp repo, and then my config file looks like this:

config=ContinueConfig(
    ...
    models=Models(
        default=LlamaCpp(
            prompt_templates={},
            max_context_length=2048,
            server_url="http://localhost:8080",
            llama_cpp_args={"stop": ["[INST]"]},
        )
    )
)