Entity Quality & Retrieval

[vinodmut]

Problem

The quality of extracted entities directly impacts the usefulness of the learning system. Currently there are gaps in ensuring we extract genuinely useful guidelines and facts (vs. noise), and there is no offline pipeline for maintaining entity quality over time. Retrieval accuracy and efficiency also need improvement to ensure the right entities are surfaced at the right time.

Goals

Extraction Quality

• Improve the signal-to-noise ratio of extracted entities — ensure we're capturing useful guidelines and facts, not trivial or redundant ones
• Better filtering during the learn phase to avoid low-value entities

Offline Entity Maintenance

• Consolidation — merge entities that express the same underlying knowledge
• Garbage collection — remove stale, outdated, or superseded entities
• Clustering — group related entities to identify themes and redundancies
• Deduplication — detect near-duplicate entities and merge them

Retrieval

• Improve retrieval accuracy — surface the most relevant entities for a given task context
• Improve retrieval efficiency — reduce overhead of entity lookup, especially as the knowledge base grows

Possible Approaches

• Define quality metrics for entities (specificity, actionability, reuse frequency)
• Build an offline batch pipeline that periodically consolidates, deduplicates, and garbage-collects entities
• Use embedding-based clustering to find related/redundant entities
• Evaluate and improve the retrieval mechanism (embedding similarity, keyword matching, hybrid approaches)
• A/B test retrieval strategies against real usage patterns
• Add quality scoring at extraction time to filter low-value entities before they enter the knowledge base

Success Criteria

• Measurable reduction in redundant/low-value entities
• Retrieval returns relevant entities with low latency as the knowledge base scales
• Offline maintenance runs without user intervention and improves entity quality over time

[jayaramkr]

We have existing code (for the react agent) to address entity maintenance and retrieval quality. subtask level extraction can help with quality, but has to be evaluated with bob and claude.

[visahak]

We want to understand and provide some confidence on this issue "ensure we're capturing useful guidelines and facts, not trivial or redundant ones"

[vinodmut]

Spun out dreaming / offline memory consolidation as #256.

I think #181 should stay focused on entity quality and retrieval behavior, while #256 covers the background process that improves memory quality over time by reviewing trajectories, consolidating duplicates, promoting recurring patterns, and surfacing stale or conflicting memories for review.