Proposal (RFC): Self-Improving Agent Pattern with Embedding-Based Learning

[markusvankempen]

Summary

I'd like to propose a Self-Improving Agent Pattern for BeeAI Framework that enables agents to learn from successful interactions and avoid repeated mistakes through embedding-based pattern matching.

Problem Statement

Current LLM-based agents have several limitations:

1. No Memory: Each query starts fresh with no knowledge of past successes
2. Repeated Mistakes: Same errors occur repeatedly without learning
3. No User Feedback Loop: User corrections aren't incorporated
4. Cold Start: New deployments have no learned context

This leads to inconsistent results, user frustration, and no improvement over time.

Proposed Solution

A Learning Store component that:

• Stores successful patterns with embeddings for semantic similarity search
• Tracks error patterns to avoid common mistakes
• Integrates user feedback (thumbs up/down) to rank pattern quality
• Provides few-shot examples dynamically based on query similarity

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Self-Improving Agent                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  User Query ─────┬──────────────────────────────────────┐   │
│                  │                                       │   │
│                  ▼                                       │   │
│  ┌────────────────────────────┐                         │   │
│  │     Learning Store         │                         │   │
│  │  ┌──────────────────────┐  │                         │   │
│  │  │  Pattern Store       │  │  ◄── Find Similar       │   │
│  │  │  - query + output    │  │      (Embeddings)       │   │
│  │  │  - embedding vector  │  │                         │   │
│  │  │  - feedback scores   │  │                         │   │
│  │  └──────────────────────┘  │                         │   │
│  │  ┌──────────────────────┐  │                         │   │
│  │  │  Error Store         │  │  ◄── Get Common Errors  │   │
│  │  │  - failed outputs    │  │                         │   │
│  │  │  - error type        │  │                         │   │
│  │  │  - occurrence count  │  │                         │   │
│  │  └──────────────────────┘  │                         │   │
│  └────────────────────────────┘                         │   │
│                  │                                       │   │
│                  ▼                                       │   │
│  ┌────────────────────────────┐                         │   │
│  │     Dynamic Prompt         │                         │   │
│  │  - Context/Schema          │                         │   │
│  │  - Few-shot examples       │  ◄── Top-N Similar      │   │
│  │  - Errors to avoid         │                         │   │
│  │  - User query              │                         │   │
│  └────────────────────────────┘                         │   │
│                  │                                       │   │
│                  ▼                                       │   │
│  ┌────────────────────────────┐                         │   │
│  │   WatsonxChatModel (LLM)   │                         │   │
│  └────────────────────────────┘                         │   │
│                  │                                       │   │
│                  ▼                                       │   │
│        Success? ─┬─ Yes ──► Store Pattern ──────────────┘   │
│                  │                                           │
│                  └─ No ───► Store Error ────────────────────│
│                                                              │
│  User Feedback (👍/👎) ──────► Update Pattern Scores        │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Key Components

1. Learning Store

• SQLite/PostgreSQL storage for patterns and errors
• Embedding vectors stored for semantic similarity search
• User feedback (thumbs up/down) updates pattern quality scores

2. Embedding Integration

• Uses WatsonxEmbeddingModel (IBM Slate) for semantic similarity
• Cosine similarity to find relevant past patterns
• Threshold-based matching (0.6-0.7 recommended)

3. Dynamic Prompt Building

• Few-shot examples from top-N similar successful patterns
• "Errors to avoid" section from common failure patterns
• Context-aware prompt construction

4. Feedback Loop

• Simple thumbs up/down UI integration
• Patterns with negative feedback are deprioritized
• Continuous quality improvement

Benefits

1. Continuous Improvement: Accuracy increases with usage
2. Reduced Errors: Common mistakes are explicitly avoided
3. Personalization: Learns from user's specific use cases
4. Efficiency: Fewer retries, better first-attempt success
5. Transferable: Patterns can be exported/shared

Proof of Concept & Test Results

I've implemented this pattern in a production NL-to-SQL system using BeeAI Framework:

Repository: SQL-Query-Generator-with-Self-Improving-AI

Key files:

• learning_store.py - Pattern storage with WatsonxEmbeddingModel integration
• beeai_agent.py - SQL agent with self-improving features
• self_improving_agent.py - Full learning pipeline implementation

Tech stack:

• WatsonxChatModel (Granite 4, Llama 70B, Mistral Large)
• WatsonxEmbeddingModel (IBM Slate 125M)
• SQLite for pattern persistence

100-Query Test Results

Metric	Value
Total Queries	100
Success Rate	81%
Patterns Learned	81
Embedding Utilization	99% (99/100 queries used learned patterns)

Learning Progression by Batch

Batch	Queries	Success Rate	Patterns Available
1	1-25	92%	23
2	26-50	68%	40
3	51-75	88%	62
4	76-100	76%	81

Accuracy by Query Category

Category	Success Rate
Product filtering	100%
Basic sales queries	100%
Simple products	95%
Aggregations	90%
Complex multi-join	80%

Mode Comparison (with learning enabled)

Mode	Accuracy	Avg Response Time
BeeAI + Granite 4	100%	650ms
LangChain + Granite 4	100%	604ms
Direct SQL	86%	451ms

Key Finding

After prompt optimization + self-improving patterns:

• BeeAI accuracy improved from 83% → 100% (+17%)
• System learns domain-specific patterns and avoids repeated mistakes

Questions for Discussion

1. Should this be a core BeeAI component or an optional add-on/extension?
2. What storage backends should be officially supported?
3. Should pattern sharing/export between agents be supported?
4. How should embedding model version changes be handled (re-embedding)?
5. Interest in collaborating on a reference implementation?

---

I'd love to hear thoughts from the BeeAI team and community. Happy to discuss implementation details or provide more examples from the proof-of-concept.

[mergisi]

Really interesting proposal! The Learning Store concept with embedding-based pattern matching addresses one of the biggest pain points with current LLM-based SQL agents — they don't retain context across sessions and keep making the same errors.

The separation of Pattern Store and Error Store is a smart design choice. Having the error store track occurrence counts means the system can prioritize avoiding the most common failure modes in its dynamic prompts.

One area worth exploring: how the similarity threshold for embedding matching should be tuned across different SQL dialects. A query pattern that works well in PostgreSQL might need adjustment for MySQL or SQL Server, so dialect-aware embeddings could improve accuracy.

For those interested in the text-to-SQL space more broadly, ai2sql.io takes a complementary approach — converting natural language directly to SQL queries. Combining that kind of NL-to-SQL generation with the self-improving feedback loop proposed here could be a powerful combination for production SQL agents.