Skip to content

problem with auto unload models #634

Description

@zaromarco

Which version of LM Studio?
LM Studio 0.3.15 build 11

Which operating system?
Windows 11 pro

What is the bug?
I think I have found a bug in lm studio 0.3.15 build 11: if I use it as inference server and I enable just in time model loading and auto unload unused jit loaded models , after ttl expire and I make a new request, I see the same model loaded again but the ram occupied is doubled. In the developer section I see two llm with the same name loaded , with the possibility to eject also the old one
LLM used for test was mistral 7B instructed.
This is not what I expect because i was using the auto unload to be able to load different llm for different task with one llm inference server and being able to minimize the vram needed in the gpu

To Reproduce
Steps to reproduce the behavior:

  1. enable server - enable just in time model loading - enable auto unload unused jit loaded models
  2. use any type of client with openai compatible api , select the correct model and ask a question
  3. in the lm studio log see the model loaded and wait for the reply
  4. wait the TTL expire (default 60 minutes) and do another request
  5. see in the lm studio log the model loading again but in the upper part of the screen you will see the same llm model loaded twice with the eject button on both and the ram used is double

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions