Reflections on Memory and Forgetting.

[HelioMorri]

This discussion is a response to the question posed by @oldnicke at #2921 .

Humans with hyperthymesia may uncontrollably remember every detail, similar to an ever-expanding RAG knowledge base; those with amnesia may forget very important things, similar to LLMs that have undergone multiple rounds of context compression.

For AI memory, the biggest problem is neither remembering too much nor too little, but rather determining which memories are most relevant to the current conversation.

Example 1: Ten years ago, Don bought a set of gardening tools to maintain his garden. A few months later, they moved downtown, and the tools were stored away. Now, Don has a garden again and wants to find the tools, but has completely forgotten where he put them. See, this all conforms to the laws of human forgetting, and this information has no value for ten years until Don needs the tools, at which point it becomes worth $300. —The value of memory cannot be defined by simple rules.

Example 2: Max collects all his medications in a family medicine cabinet. One day, he passes a pharmacy and suddenly wants to replace expired medications with new ones, but he can't remember the complete list of medications or their expiration dates, even though he carefully checked the expiration dates when he put them in the cabinet. —Incomplete memories cannot fully realize their value.

Example 3: Grandma forgets she took her medication half an hour ago and takes it again. —Incorrect or completely forgotten memories can have more serious consequences.

I believe that only memories that are categorized, hierarchically differentiated, and jointly organized by humans and AI are effective.

Assuming a simplified model, Tony has the following:

• Work: Tony works in operations and development at Google Cloud, which involves troubleshooting many problems, writing scripts, developing tools, and communicating extensively with colleagues.
• Life: Tony lives in a large family and needs to plan his schedule and shopping with his wife.
• Hobbies: Tony is an enthusiast of electronics, computers, and mechanics. He frequently searches and reads extensively, takes notes, blogs, and is very active in various forums and group chats. Tony is currently writing a slice-of-life novel and needs AI to help him generate humor and outline his writing.

1. Classification and Connections

• Tony's plans to buy bread tomorrow and troubleshoot a memory leak seem to fall into the categories of life and work. However, failing to solve the memory leak might prevent him from buying bread before the store closes tomorrow. In fact, Tony's sister will be passing by the bakery tomorrow, but Tony doesn't know this. — Some things may not be related on a specific dimension (being placed in the same category), but they may be related on another simple dimension, or even more complexly.
• Tony might draw inspiration from his life experiences and the people around him when writing his novel, but he almost never lives according to the content of the novel. However, an idea generated while writing the novel might resolve a social friction in his life. —Some connections may be unidirectional, but not absolute; some matters may be separate, but these can be easily confused by AI, which can only access limited information through limited channels (LLM can only understand information through textual context, ultimately confusing fiction with reality).
• Tony stumbled upon a post on an electronics enthusiast forum, which led him to solve a bug that had been troubling him at work. Based on the experience he gained at work, Tony created an open-source tool for users on computer forums. —Different entities in two categories may have a strong correlation in a specific scenario, while in other scenarios they may have a weak or no correlation.
• Of course, there are also various connections based on time, space, matter, experience (thought), and logic, which can be arranged and combined to form even richer results.

2. Layering

• Based on usage frequency: For example, weekly routine shopping is high frequency, once a year is low frequency.
• Based on difficulty: Debugging is high difficulty, finding a recipe is low difficulty.
• Based on time window: A family vacation requires at least two weeks; a weekend is not enough for a vacation.
• Based on spatial relationship: There is obviously a big difference between a gas station 1 kilometer away and a convenience store 100 kilometers away.
• Based on value: The debugging process itself has low value, but the resulting debug report (causes, thought processes, solutions, reflections, conclusions) has high value; the code of a company's business system has low value, but the understanding, thinking, design, experience, and lessons learned from real-world business scenarios during the design process are highly valuable. I believe that a knowledge base that stores only code instead of documentation would be a disaster.
• And many other types of information that can be layered.

3. Human Assistance

• Even in a decades-long marriage, no one can fully understand another person. For LLMs, whose thinking patterns are completely different from humans and who primarily rely on text-based communication, accurately understanding human intentions, especially implicit assumptions, underlying meanings, deep considerations and trade-offs, and significant information hidden within a simple sentence, is even more difficult. Therefore, when the memory compiled purely by AI reaches a certain scale, the errors and deviations can accumulate to an unacceptable level.
• However, AI makes mistakes, and so do humans. Memory can be misguided and incorrectly corrected; AI may be commanded to forget important information, and it may simplify complex information when handling memory conflicts, thus causing even greater problems.

Therefore, an excellent memory system should be extremely flexible, automatically evolving, and even itself a very special kind of dynamic evolutionary network.

I hope this content will provide some inspiration for the Hindsight project team.