Add intelligent persistent history to tools/query.py

[gvanrossum-ms]

It's often awkward to ask follow-up questions because the query engine doesn't understand references like "it", "she", or "the first point". It also often doesn't have the words generated by the answer generator in its index.

I haven't quite thought through the necessary architecture, but essentially it should take a phrase like "What is wrong with it" and turn it into "What is wrong with the parrot".

This could be done using a single LLM roundtrip (not using Structured RAG), feeding it just recent history (for "it", 3-4 turns might well be enough). Or we might be able to do a search in the history (both questions and answers) using a separate (internally managed) Conversation object.

Experiments are needed.

[bmerkle]

cool idea.
Up to now AFAIK no existing LLM framework provides intelligent historical reference resolution.
LangChain has some basic support to memorize (ConversationBufferMemory, ConversationSummaryMemory) but no semantics.
ChatGTP found some interesting research papers, e.g. https://arxiv.org/html/2505.22907v1

One of the fundamental problems i see, it that we only transfer text/tokens without any reference to a meta-model or knowledge graph so the information what means "it", "she", or "the first point" might have been there, but is lost during transport to other layers.

[bmerkle]

I chatted a bit with GPT and the best approach seems to be a combination with Cognee + Memgraph. (https://github.com/topoteretes/cognee) (https://github.com/memgraph/memgraph)
Not sure you think it is too heavyweight, because you mentioned not to use a (structured RAG etc).

User Query
    ↓
Query Rewriter LLM  ←──────────────┐
    ↓                              │
Resolved Query                     │
    ↓                              │
Downstream LLM, RAG, tools, etc.   │
                                   │
           Cognee: Extract entities/relations
                                   ↓
              Store/update in Memgraph
                                   ↓
    Historical reference resolver pulls:
      • last X turns
      • entities
      • relations
      • mentions
      • graph distances

[gvanrossum-ms]

Those tools look like they have a somewhat complicated learning curve. Probably learning typeagent currently is even more complicated, but at least I already know it. :-)

Resolving "it", "her" etc. is a higher priority than eating our own dogfood. This can be done by saving the last X inputs and outputs (3-5 is probably enough) of the current session (I think this doesn't need to be persisted) and adding them to the prompt for the initial query->json translation. This doesn't add another LLM roundtrip; it adds a bit to the context of each request but that's usually faster. I have done a similar thing in a query processor for a different experiment in the past, and it worked quite well.

[UPDATE:] My ChatGPT thinks this will work. :-)