A Node.js-based diagnostic tool that reverse-engineers the Retrieval-Augmented Generation (RAG) pipelines used by modern generative answer systems (e.g., Perplexity, ChatGPT & Co.).
This application simulates the exact deterministic retrieval and probabilistic generation phases of an AI answer system (LLM-ChatBot), allowing technical SEOs and content architects to measure and optimize their "Share of Context" (SoC) against live SERP competitors using advanced techniques like HyDE and Parent-Child chunking.
The tool operates on an Express.js backend and executes a 10-stage, enterprise-grade pipeline:
-
Concurrent Data Ingestion: Simultaneously fetches the user’s target URL via ScraperAPI and the live Top 10 organic SERP competitors via the Tavily Search API.
-
Content Extraction & Parsing: Radically strips DOM noise (navigation, footers, cookie banners) and normalizes aggressive whitespace to extract only core textual content into clean Markdown.
-
Smart Text Chunking: Utilizes LangChain's semantic splitters to divide text into
Parentchunks (~800 chars) for LLM context, andChildchunks (~200 chars) for high-precision mathematical embedding, respecting NLP sentence boundaries. -
SERP Intent Classification: Analyzes competitor URLs to classify the underlying search intent (e.g., listicle, vendor page, forum) to establish query consensus.
-
Semantic Vector Generation (HyDE & Bi-Encoder): Generates a Hypothetical Document Embedding (HyDE) representing the "perfect" answer, and vectorizes all Child chunks using
gemini-embedding-001. -
Deterministic Semantic Search (Auto-Merging): Calculates Cosine Similarity between the HyDE vector and Child vectors. The highest-scoring Child chunks pass their scores back to their respective Parent chunks to form an initial Top 20 Evaluation Pool.
-
Algorithmic LLM Re-Ranking (Cross-Encoder): A secondary evaluation model (
gemini-2.5-prooperating attemperature: 0.0) acts as a deterministic Cross-Encoder, scoring the Top 20 chunks based on factual density, directness, and completeness. -
Context Window Assembly: The system aggregates the final Top 5 highest-scoring chunks, establishing the strict factual boundary the AI is allowed to use.
-
Gen. Synthesis & Optimization Re-Write: The Context Window is handed off to a probabilistic LLM to synthesize the final simulated answer. Simultaneously, the system runs Entity Gap and Information Gain analyses to draft a data-dense rewrite for the user.
-
HTML Rendering & Response Delivery: Calculates stochastic citation probabilities (based on 5 concurrent generative simulations) and renders the analytical data into a visual diagnostic interface.
Additional Analytics & Synthesis:
- Entity Gap Analysis: Extracts Named Entities present in competitor chunks but missing from the user's chunk.
- Information Gain Scoring: Evaluates if the user's chunk provides net-new data compared to the SERP consensus.
- Stochastic Simulation: Fires 5 concurrent LLM generation requests (Temperature 0.7) using the final Context Window to calculate the exact Citation Probability, accounting for AI variance.
- AIO Rewrite: Generates a data-dense rewrite of the user's content to compete for the #1 citation slot.
- Node.js (v18.0.0 or higher recommended)
- API Keys for the following services:
- Google Gemini API (For embeddings and LLM reranking/synthesis)
- Tavily API (For SERP retrieval and text scraping)
- ScraperAPI (For bypassing bot protection on the target URL)
1. Clone the repository
Install dependencies
The application requires Express, CORS, Axios, Cheerio, and Dotenv. Aswell as LangChain Text Splitters.
npm install express cors axios cheerio dotenvnpm install @langchain/textsplitters2. Configure Environment Variables Edit .env file in the root directory of the project and add your API keys.
GOOGLE_API_KEY=your_gemini_api_key_here
SCRAPER_API_KEY=your_scraperapi_key_here
TAVILY_API_KEY=your_tavily_api_key_here
PORT=30003. Start the server
node server.js4. Access the Application Open your web browser and navigate to:
http://localhost:3000(Note: For production deployment, it is highly recommended to route the application through a secure reverse proxy such as Cloudflare Tunnels or Nginx/OpenLiteSpeed to manage SSL and port forwarding.)
Generative AI systems are widely viewed as unpredictable "black boxes" due to their probabilistic text generation. However, this view ignores the underlying system architecture. Search engines and answer engines (like Perplexity or ChatGPT) do not synthesize answers purely from their internal training data. They utilize a Retrieval-Augmented Generation (RAG) framework.
The RAG framework splits the answer generation process into two distinct phases: retrieval and synthesis.
While the final text synthesis relies on probabilistic models, the preceding retrieval phase is fundamentally deterministic. It relies on algorithmic scoring, vector mathematics, and structured filtering.
The Hypothesis: Because the retrieval and weighting of data chunks prior to the synthesis phase are deterministic, they are governed by rules. Consequently, this phase could be systematically influenced and optimized for. If we are able to understand the selection criteria of the context window, we should theoretically be able to architect content to mathematically win those slots.
To test this hypothesis against real-world data and explore potential optimization parameters, I built a custom RAG Pipeline Simulator.
The objective is to feed the simulator live SERP data, run it through a standard multi-stage AI retrieval process, and observe the mathematical outcomes. By exposing the mechanics of how data is weighted, we explore whether we could derive working optimization strategies.
To understand the hypothesis, we must define the core technical sequence of a modern RAG pipeline. The simulator is built upon these 10 sequential stages:
The system requires raw data. It simultaneously fetches the user’s target URL (the content to be optimized) and the live Top 10 organic SERP competitors for the given query.
Raw HTML is filled with DOM noise (navigation, footers, scripts). The scraper strips this architecture to extract only the core textual content, converting it into a clean, normalized Markdown format.
Large Language Models (LLMs) suffer from "vector blur" if fed entire web pages. The parsed text is separated into smaller, semantic blocks (Parent/Child chunks) respecting natural language boundaries to maintain mathematical precision.
Before evaluation, the system analyzes the extracted competitor URLs to classify the underlying search intent (e.g., listicle, vendor page, forum). This establishes the baseline consensus of the query.
Text chunks cannot be computed; they must be mapped as coordinates. An embedding model translates every text chunk into a high-dimensional mathematical vector (Semantic Vector Generation).
The system calculates the geometric distance (Cosine Similarity) between the user's prompt and the available chunk vectors. This is a purely mathematical filter that isolates the top-matching chunks to form an initial evaluation pool.
A secondary evaluation model (gemini-2.5-pro operating at temperature: 0.0) acts as a deterministic Cross-Encoder, scoring the Top 20 chunks based on factual density, directness, and completeness.
The system aggregates the highest-scoring chunks from the Re-Ranking phase. These final, highly vetted text blocks are assembled to form the "Context Window"—the strict factual boundary the AI is allowed to use.
A generative LLM reads the Context Window and synthesizes the final answer. Simultaneously, the system compares the user's excluded content against the winning competitor chunks to calculate Information Gain and Entity Gaps, drafting an optimized rewrite.
The raw analytical data, scores, and mathematical probabilities are rendered into a visual interface, allowing the consultant to audit the pipeline's decisions.
| Phase | LLM involved? |
|---|---|
| Ingestion | ❌ sometimes |
| Embedding generation | ❌ (embedding model, not chat LLM) |
| Retrieval | ❌ |
| Ranking | ❌ or small models |
| Context assembly | ❌ |
| Answer synthesis | ✅ YES — ONLY HERE |
To maintain transparency regarding the simulator's current capabilities, it is important to note that the pipeline currently does not utilize Query Fan-Out (Multi-Query Routing).
When processing complex, multi-intent prompts, production-grade AI Answer Systems dynamically split the user's initial prompt into several distinct sub-queries. The system then runs parallel retrieval processes for each sub-query to populate a much broader, multi-faceted context window before synthesis.
Currently, this simulator processes the user prompt as a single, unified vector query (post-HyDE generation). While highly accurate for standard, single-intent queries, testing extremely complex or comparative prompts may yield a narrower context window than a true multi-query system. Implementing an LLM-based query decomposition routing layer is a planned architectural optimization for future releases.
This tool and the resulting data represent a working hypothesis. If the technical assumption holds true, that deterministic retrieval overrides brand authority and probabilistic variance, the output of this simulator should yield highly actionable optimization directives.
Whether these directives consistently lead to increased "Share of Context" in live AI answer systems is the next phase of this ongoing analysis. I am opening this tool and the underlying methodology for industry discussion and empirical validation.
