Session transcript not used as conversation context - model has no memory of previous turns

[noorbhatia]

When using LanguageModelSession for multi-turn conversations, the transcript accumulates entries correctly, but the model does not appear to receive the transcript as conversation context. Each response is generated as if it's the first message, with no memory of previous exchanges.

In chat-ui-swift a new session is created for every message. Shouldn't the session use transcript to build working memory for the model ?

[mattt]

Oof, you're totally right. I patched chat-ui-swift to reuse session objects (mattt/chat-ui-swift@db27d3d).

I also see where we're not passing the transcript, and opened #61.

Pointing the app at that branch, we now see the model remembering what we just said.

[noorbhatia]

Thanks @mattt I have been testing with MLX and facing the same.
Tested with mlx-community/Llama-3.2-3B-Instruct-4bit

[airy10]

It's a general issue, not limited to the Apple model

[noorbhatia]

@mattt Should we keep the issue open as it's resolved only for MLX as of now ? I believe it's happening for OpenAI and others too.

[airy10]

Shouldn't it be handled at Session level instead of at Model level ? (basically doing what your chat app did when creating new session for every prompt, but integrating that inside the Session when sending new prompts)

[noorbhatia]

@airy10

Shouldn't it be handled at Session level instead of at Model level ? (basically doing what your chat app did when creating new session for every prompt, but integrating that inside the Session when sending new prompts)

I think the problem with handling it at session level is that each model has different chat formats. So the conversion logic must live in each model because only the model knows its native format

[airy10]

If it can be handled at app level, like the chat app, which is going through the session to reach the models, then it seems it should be possible to handle it in the session (which has all of the transcripts)

[noorbhatia]

The chat app fix just persists session in the memory. The real fixes are in #61 #66