Using Local LLM in XeroFlow

[Mystic419]

I have tried several models and they all seem to have the same problem. When generating a long form story the prompt in the first node overloads my local LLM and the LLM then ignores my prompt input, creating a story based on the examples, not my prompt. I'm afraid to modify the system prompt for fear of breaking it. My work around is to use Gemini to start the story then let my local LLM take over from there. Is it as good as making it with Gemini all the way through, no, but its close. It still makes an interesting story.

[Xerophayze]

So in LM studio you can go to the properties or settings for the LLM models and there is an option where you can turn up the context available to the model, it will actually tell you how much context the model can support. I think you told me you're using mistralai-Mistral-Nemo-Instruct-2407-extensive-BP-abl. And I had it set up on the first node and it seemed to work fine. It was still separating the different sections of the high level outline which I didn't want it to do but I realized the reason it was doing that is because it's instructed to format the story into a first second and third act. So it was kind of assuming it needed three separate paragraphs. I think I fixed this. But I thin

k you need to open up the context window in LM studio

[Mystic419]

My context was at 16K, I have increased it to 20K and that almost maxes my vram out. And that seems to have done it! Using 11.5GB vram with no offloading!