[Copilot BYOM] Allow 'parameterOverrides' in chatLanguageModels.json for API compatibility

[ggeorgakoudis]

I am using the "Bring Your Own Model" feature with chatLanguageModels.json to connect Copilot to an OpenAI-compatible endpoint backed by AWS Bedrock.

The Problem: VS Code sends both temperature and top_p by default. AWS Bedrock (via LiteLLM/other proxies) throws a 400 Error because it forbids sending both parameters simultaneously for Anthropic models.

The Current Workaround: Users are forced to write and run a local Python proxy server just to strip one parameter from the request body.

The Request: Please add an optional completionOptions or parameterOverrides object to the chatLanguageModels.json schema. This would allow users to:

Set static values for strict backends (e.g., "top_p": null).

Example Configuration:

{
    "vendor": "custom",
    "models": [{
        "id": "some-model",
        "parameterOverrides": {
            "top_p": null,   // Explicitly remove/nullify this
            "temperature": 0.5
        }
    }]
}

[vs-code-engineering]

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[vs-code-engineering]

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[vs-code-engineering]

🙂 This feature request received a sufficient number of community upvotes and we moved it to our backlog. To learn more about how we handle feature requests, please see our documentation.

Happy Coding!

[vmizg]

Are there any updates on surfacing this up in the backlog? We can not use any Anthropic models via third-party LLM proxies because of this.

Current workaround is to run a local proxy that drops the conflicting params before going into the LLM proxy:

llm-proxy.py

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import httpx, json

app = FastAPI()
TARGET = "https://<litellm_url>"

@app.post("/v1/{path:path}")
async def proxy(path: str, request: Request):
    body = await request.json()
    body.pop("top_p", None)

    headers = {
        k: v for k, v in request.headers.items()
        if k.lower() not in ("host", "content-length")
    }

    is_stream = body.get("stream", False)

    if is_stream:
        async def stream_response():
            async with httpx.AsyncClient(timeout=120) as client:
                async with client.stream(
                    "POST",
                    f"{TARGET}/{path}",
                    json=body,
                    headers=headers,
                ) as r:
                    async for chunk in r.aiter_raw():
                        yield chunk

        return StreamingResponse(
            stream_response(),
            media_type="text/event-stream",
        )
    else:
        async with httpx.AsyncClient(timeout=120) as client:
            r = await client.post(
                f"{TARGET}/{path}",
                json=body,
                headers=headers,
            )
        return StreamingResponse(
            iter([r.content]),
            status_code=r.status_code,
            media_type=r.headers.get("content-type", "application/json"),
        )

Then running this as:

pip install fastapi uvicorn httpx
uvicorn llm-proxy:app --port 8765

Then pointing your chatLanguageModels.json models to localhost:8765 instead of <litellm_url>

[stevenlele]

This looks like a duplicate of #321514 and should already be fixed in #323532 but the docs have not been updated yet.