Unable to change server endpoint

[JungeWerther]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Linux Ubuntu 22.04
- Continue: 
- IDE VSCode

Description

Unable to change endpoint using python-cpp-llama server which runs on localhost:8000/v1 not localhost:8000.

If I change manually either in OpenAI class or CPP class in config.py, it doesn't change anything.

Reloading server or IDE doesn't solve.

Verified that it sends requests to localhost:8000/completions.

Expected: request sent to localhost:8000/v1/completions.

Also the error message is wrong (says it sends a request to chat/completions but really sends to /completions)

To reproduce

No response

Log output

Traceback (most recent call last):

  File "server/continuedev/core/autopilot.py", line 448, in _run_singular_step

  File "server/continuedev/core/main.py", line 382, in __call__

  File "server/continuedev/plugins/steps/chat.py", line 110, in run

  File "/var/folders/n3/qjs7qlys1gz1t9359fv0m9f40000gn/T/_MEIjaIOdr/continuedev/libs/llm/base.py", line 479, in stream_chat
    async for chunk in self._stream_complete(prompt=prompt, options=options):

  File "/var/folders/n3/qjs7qlys1gz1t9359fv0m9f40000gn/T/_MEIjaIOdr/continuedev/libs/llm/ggml.py", line 225, in _stream_complete
    async for chunk in self._raw_stream_complete(prompt, options):

  File "/var/folders/n3/qjs7qlys1gz1t9359fv0m9f40000gn/T/_MEIjaIOdr/continuedev/libs/llm/ggml.py", line 107, in _raw_stream_complete
    raise Exception(

Exception: Error calling /chat/completions endpoint: 404

[sestinj]

@JungeWerther Could you share your config.py and I can try to quickly debug this?

[JungeWerther]

Actually now it suddenly started working after reinstalling the extension and waiting a bit. Weirdly, because the first time around it didn't work.

Maybe reloading the window didn't initially update the config because the server wasn't closed on host machine (ssh connection).

Config was

`"""
This is the Continue configuration file.

See https://continue.dev/docs/customization to for documentation of the available options.
"""
from continuedev.libs.llm.ggml import GGML
from continuedev.libs.llm.prompts.chat import llama2_template_messages
from continuedev.libs.llm.prompts.edit import alpaca_edit_prompt
from continuedev.libs.llm.prompts.chat import template_alpaca_messages
from continuedev.libs.llm.llamacpp import LlamaCpp

from continuedev.core.models import Models
from continuedev.core.config import CustomCommand, SlashCommand, ContinueConfig
from continuedev.libs.llm import OpenAIFreeTrial

from continuedev.plugins.context_providers import (
    DiffContextProvider,
    TerminalContextProvider,
    URLContextProvider,
    GitHubIssuesContextProvider
)
from continuedev.plugins.steps import (
    ClearHistoryStep,
    CommentCodeStep,
    EditHighlightedCodeStep,
    GenerateShellCommandStep,
    OpenConfigStep,
)
from continuedev.plugins.steps.share_session import ShareSessionStep

config = ContinueConfig(
    allow_anonymous_telemetry=True,
    models=Models(
		saved=[OpenAIFreeTrial(
			model="gpt-4"
		),
		GGML(
			title="WizardCoder-13b-Python",
			model="wizardcoder:13b-python",
			context_length=4096,
			prompt_templates={'edit': {'model': None, 'temperature': None, 'top_p': None, 'top_k': None, 'presence_penalty': None, 'frequency_penalty': None, 'stop': ['```'], 'max_tokens': 600, 'functions': None, 'prompt': 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction: Rewrite the code to satisfy this request: "{{{user_input}}}"\n\n### Input:\n\n```\n{{{code_to_edit}}}\n```\n\n### Response:\n\n```\n', 'raw': True}},
			template_messages=template_alpaca_messages,
			server_url="http://localhost:8080/v1/"
		),
		GGML(
			title="WizardCoder-13b-Python",
			model="wizardcoder:13b-python",
			prompt_templates={'edit': {'model': None, 'temperature': None, 'top_p': None, 'top_k': None, 'presence_penalty': None, 'frequency_penalty': None, 'stop': ['```'], 'max_tokens': 600, 'functions': None, 'prompt': 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction: Rewrite the code to satisfy this request: "{{{user_input}}}"\n\n### Input:\n\n```\n{{{code_to_edit}}}\n```\n\n### Response:\n\n```\n', 'raw': True}},
			template_messages=template_alpaca_messages
		),
		OpenAIFreeTrial(
			model="gpt-3.5-turbo"
		),
		GGML(
			title="CodeLlama-7b-Instruct",
			model="codellama:7b-instruct"
		)],
		default=GGML(
			title="WizardCoder-13b-Python",
			model="wizardcoder:13b-python",
			prompt_templates={'edit': {'model': None, 'temperature': None, 'top_p': None, 'top_k': None, 'presence_penalty': None, 'frequency_penalty': None, 'stop': ['```'], 'max_tokens': 600, 'functions': None, 'prompt': 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction: Rewrite the code to satisfy this request: "{{{user_input}}}"\n\n### Input:\n\n```\n{{{code_to_edit}}}\n```\n\n### Response:\n\n```\n', 'raw': True}},
			template_messages=template_alpaca_messages,
			api_base="http://localhost:8080/v1"
		)
	),
    system_message="",
    temperature=0.5,
    custom_commands=[
        CustomCommand(
            name="test",
            description="Write unit tests for highlighted code",
            prompt="Write a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
        )
    ],
    slash_commands=[
        SlashCommand(
            name="edit",
            description="Edit highlighted code",
            step=EditHighlightedCodeStep,
        ),
        SlashCommand(
            name="config",
            description="Customize Continue",
            step=OpenConfigStep,
        ),
        SlashCommand(
            name="comment",
            description="Write comments for the highlighted code",
            step=CommentCodeStep,
        ),
        SlashCommand(
            name="clear",
            description="Clear step history",
            step=ClearHistoryStep,
        ),
        SlashCommand(
            name="share",
            description="Download and share this session",
            step=ShareSessionStep,
        ),
        SlashCommand(
            name="cmd",
            description="Generate a shell command",
            step=GenerateShellCommandStep,
        )
    ],
    context_providers=[
        # GitHubIssuesContextProvider(
        #     repo_name="<your github username or organization>/<your repo name>",
        #     auth_token="<your github auth token>"
        # ),
        DiffContextProvider(),
        URLContextProvider(
            preset_urls = [
                # Add any common urls you reference here so they appear in autocomplete
            ]
        ),
        TerminalContextProvider(),
    ],
)`