LMStudio file context in prompts gets truncated

[pwilkin]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Linux
- Continue version: VSCode 1.0.2, IntelliJ 1.0.2
- IDE version: 
- Model: granite-3.2-8b-instruct
- config:
  
   {
      "apiBase": "http://127.0.0.1:1234/v1/",
      "model": "granite-3.2-8b-instruct",
      "title": "[LM] granite-3.2-8b-instruct",
      "provider": "lmstudio",
      "contextLength": 32000,
      "completionOptions": {
        "temperature": 0.15,
        "maxTokens": 24000
      }
    }

Description

When I try to attach an XSD schema that has around 15k tokens, the schema file gets truncated, even though with other providers (such as Mistral) it gets passed correctly. This happens even if I increase the context size all the way up to 100k.

To reproduce

1. Install LM Studio.
2. Add a model that supports big contexts (preferrably up to 128k).
3. Add a configuration that allows a big context length.
4. Attach a big file (around 15k tokens).
5. Try to attach the file in a prompt to the model.

Log output

[IngLP]

I have the same problems, long files get truncated!

[tomasz-stefaniak]

@pwilkin could you check your logs and see what context length the model receives? Here, for example, I see that I'm using:

{
  "contextLength": 200000,
  "temperature": 0,
  "model": "claude-3-5-sonnet-latest",
  "maxTokens": 8192
}

I'm assuming that when you inspect the prompt logs the file appears truncated?

[pwilkin]

@tomasz-stefaniak The metadata the model receives is fine, but the data is truncated already in the message. This is part of a log from LMStudio:

  "messages": [
    {
      "role": "system",
      "content": "Always include the language and file name in the i... <Truncated in logs> ...rite code blocks, for example '```python file.py'."
    },
    {
      "role": "user",
      "content": "nabler-condition\">\r\n    \t<complexContent>\r\n\t\t\t<ext... <Truncated in logs> ... \nWhat is the main element defined in this schema?"
    }
  ],
  "model": "granite-3.2-8b-instruct",
  "max_tokens": 20000,
  "temperature": 0.15,
  "stream": true
}

As you see, since the file is an XSD, it's pretty clear it has been cut at the start.

From the LMStudio metadata, we're far from the model boundaries here:

generate: n_ctx = 34816, n_batch = 2048, n_predict = 20000, n_keep = 10983

I'll add: this happens only with LMStudio. The same data passed to Mistral, Gemini or OpenRouter works just fine.

[IngLP]

@pwilkin could you check your logs and see what context length the model receives? Here, for example, I see that I'm using:

{
  "contextLength": 200000,
  "temperature": 0,
  "model": "claude-3-5-sonnet-latest",
  "maxTokens": 8192
}

I'm assuming that when you inspect the prompt logs the file appears truncated?

Yes I can see the files are truncated in prompt.log.
This is true for any llm i tried.