Feature: AI Memory system for queryable documentation

[Yogesh1290]

Description

Add AI Memory system that transforms scraped documentation into queryable JSON tree structure.

Problem

Scraped docs are 50K-200K tokens. Sending entire docs to LLMs is expensive (~$0.03/query) and slow.

Solution

Transform docs into hierarchical JSON with indexes and embeddings. Query locally to find relevant sections (3-5K tokens), send only those to LLM. 95%+ token savings (~$0.001/query).

Implementation

• Hierarchical tree structure (organized by headings)
• Multiple query types: path, keyword, semantic, hybrid
• Local embeddings (no API costs)
• Token counting
• JSON storage

New CLI flags

\\�ash
webpull --ai-memory # Generate AI memory JSON
webpull --ai-memory-out

# Custom output directory
\\

Use cases

• Documentation chatbots
• CLI tools with doc search
• Offline documentation query
• RAG learning/prototyping

Files

• \src/ai-memory/\ - Core system (17 files)
• \examples/chatbot.ts\ - Example implementation
• \AI-MEMORY-README.md\ - Documentation

Testing

Tested with Bun.sh docs (428 nodes, 82K tokens) and Next.js docs (263 nodes, 47K tokens).

Dependencies

• @xenova/transformers\ (v2.17.2) - Local embeddings
• @google/generative-ai\ (v0.21.0) - Example only (optional)

Breaking changes

None. Purely additive feature.

[m13v]

i hit this wall hand-rolling a docs RAG for a weekend project. the heading-tree assumption breaks the moment you point it at sloppy markdown where the H2 doesn't actually describe the section's topic. local embeddings catch lexical-ish matches but semantic recall on top-k stays mediocre without a rerank step. tested it on bun.sh and next.js docs first, then a smaller library with messier structure, and the 95% token-savings claim collapsed pretty fast. cheapest fix is a tiny cross-encoder rerank on the retrieved chunks before sending to the LLM, otherwise you're shipping confidently-wrong context.

[Yogesh1290]

@m13v Yes, you are right. Basic heading chunking struggles with messy markdown. And local embeddings aren't enough for good recall without increasing top-K, which ruins the token savings.

I mentioned 'no re-ranking' and 'advanced chunking' under the limitations and planned improvements in the AI-MEMORY-README for this exact reason. Adding a cross-encoder for a rerank step is the best way to fix the accuracy.

I will look into adding that next. Thanks for the feedback.