Wikidata-Enhanced RAG Pipeline Design

[yiftachashkenazi]

Wikidata-Enhanced RAG Pipeline Design

Overview

This diagram shows the enhanced RAG pipeline that injects Wikidata-derived scientific context to improve the quality and factuality of grant application generation.

Pipeline Flow

graph TD
    A[Research Objectives & Tasks] --> B[Claude Enrichment Step]
    
    B --> B1[Generate Instructions]
    B --> B2[Generate Descriptions]
    B --> B3[Generate Guiding Questions]
    B --> B4[Generate Search Queries]
    B --> B5[Generate 5 Core Scientific Terms]
    
    B1 --> C[Enrichment Response DTO]
    B2 --> C
    B3 --> C
    B4 --> C
    B5 --> C
    
    C --> D[Wikidata Expansion Step]
    
    D --> D1[Query Wikidata for each core term]
    D1 --> D2[Find related scientific concepts]
    D2 --> D3[Find established practices]
    D3 --> D4[Find synonyms and variants]
    
    D4 --> E[Scientific Context Formatter]
    E --> E1[Format for LLM consumption]
    E1 --> E2[Add explanation about context vs proposal]
    
    E2 --> F[Enhanced DTO with Scientific Context]
    
    F --> G[RAG Retrieval Step]
    G --> G1[Use enhanced search queries]
    G1 --> G2[Retrieve relevant documents]
    G2 --> G3[Combine with scientific context]
    
    G3 --> H[Gemini Writing Step]
    H --> H1[Receive scientific foundation context]
    H1 --> H2[Understand established practices]
    H2 --> H3[Generate proposal that builds upon foundation]
    H3 --> H4[Show innovation beyond current knowledge]
    
    H4 --> I[Final Grant Application Text]
    
    %% Styling
    classDef enrichmentStep fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    classDef wikidataStep fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
    classDef retrievalStep fill:#e8f5e8,stroke:#1b5e20,stroke-width:2px
    classDef writingStep fill:#fff3e0,stroke:#e65100,stroke-width:2px
    classDef outputStep fill:#fce4ec,stroke:#880e4f,stroke-width:2px
    
    class B,B1,B2,B3,B4,B5 enrichmentStep
    class D,D1,D2,D3,D4,E,E1,E2 wikidataStep
    class G,G1,G2,G3 retrievalStep
    class H,H1,H2,H3,H4 writingStep
    class I outputStep

Loading

Detailed Component Flow

graph LR
    subgraph "Phase 1: Enhanced Enrichment"
        A1[Research Objective] --> A2[Claude LLM]
        A2 --> A3[Instructions]
        A2 --> A4[Description]
        A2 --> A5[Guiding Questions]
        A2 --> A6[Search Queries]
        A2 --> A7[5 Core Scientific Terms]
    end
    
    subgraph "Phase 2: Wikidata Expansion"
        B1[Core Term: 'computer vision'] --> B2[Wikidata Query]
        B2 --> B3[Related Concepts]
        B3 --> B4[Established Practices]
        B4 --> B5[Scientific Context]
    end
    
    subgraph "Phase 3: Context Injection"
        C1[Scientific Context] --> C2[Context Formatter]
        C2 --> C3[LLM-Ready Format]
        C3 --> C4[Explanation: 'This is foundation, not proposal']
    end
    
    subgraph "Phase 4: Enhanced Retrieval"
        D1[Enhanced Search Queries] --> D2[RAG Retrieval]
        D2 --> D3[Document Chunks]
        D3 --> D4[Combined Context]
    end
    
    subgraph "Phase 5: Informed Writing"
        E1[Scientific Foundation] --> E2[Gemini LLM]
        E2 --> E3[Understanding of Current State]
        E3 --> E4[Proposal that Builds Upon Foundation]
        E4 --> E5[Innovation Beyond Current Knowledge]
    end
    
    A7 --> B1
    B5 --> C1
    C4 --> D1
    D4 --> E1
    E5 --> F[High-Quality Grant Application]

Loading

Data Flow Architecture

graph TD
    subgraph "Input Data"
        A[Research Objectives] --> A1[Title]
        A --> A2[Description]
        A --> A3[Tasks]
    end
    
    subgraph "Enrichment DTO"
        B[EnrichmentDataDTO] --> B1[instructions: str]
        B --> B2[description: str]
        B --> B3[guiding_questions: List[str]]
        B --> B4[search_queries: List[str]]
        B --> B5[scientific_core_terms: List[str]]
    end
    
    subgraph "Wikidata Enhanced DTO"
        C[EnhancedEnrichmentDataDTO] --> C1[All fields from B]
        C --> C2[scientific_context: str]
        C --> C3[wikidata_expansions: Dict[str, List[str]]]
    end
    
    subgraph "Writing Context"
        D[WritingContext] --> D1[Retrieved Documents]
        D --> D2[Scientific Foundation]
        D --> D3[Enrichment Data]
        D --> D4[Search Queries]
    end
    
    A --> B
    B5 --> E[WikidataExpander]
    E --> C2
    C --> D
    D --> F[Final Text Generation]

Loading

Implementation Components

graph TB
    subgraph "Core Components"
        A[enrich_research_objective.py] --> A1[Enhanced Prompt]
        A1 --> A2[Generate 5 Core Terms]
        
        B[wikidata_expander.py] --> B1[SPARQL Queries]
        B1 --> B2[Scientific Concept Mapping]
        
        C[scientific_context.py] --> C1[Context Formatter]
        C1 --> C2[LLM Instruction Generator]
        
        D[handler.py] --> D1[Pipeline Orchestrator]
        D1 --> D2[Wikidata Integration Point]
    end
    
    subgraph "Data Structures"
        E[EnrichmentDataDTO] --> E1[+ scientific_core_terms]
        F[ResearchComponentGenerationDTO] --> F1[+ scientific_context]
        G[WikidataExpansionResult] --> G1[term: str, related_concepts: List[str]]
    end
    
    subgraph "Integration Points"
        H[After Enrichment] --> H1[Before Retrieval]
        I[Before Writing] --> I1[Context Injection]
        J[Prompt Enhancement] --> J1[Scientific Foundation Context]
    end
    
    A2 --> B
    B2 --> C
    C2 --> D2
    E1 --> F1
    G1 --> C1
    H1 --> I1
    I1 --> J1

Loading

Benefits and Outcomes

graph LR
    subgraph "Input Enhancement"
        A[5 Core Scientific Terms] --> B[Wikidata Knowledge Graph]
        B --> C[Expanded Scientific Context]
    end
    
    subgraph "Quality Improvements"
        D[Factual Grounding] --> D1[Verified Scientific Concepts]
        E[Context Awareness] --> E1[Established vs Innovative]
        F[Reduced Hallucinations] --> F1[Clear Background Separation]
        G[Enhanced Retrieval] --> G1[Precise Scientific Terminology]
    end
    
    subgraph "Output Quality"
        H[Better Proposals] --> H1[Properly Acknowledged Foundation]
        I[Innovation Clarity] --> I1[Clear Value Addition]
        J[Scientific Rigor] --> J1[Evidence-Based Approach]
    end
    
    C --> D
    C --> E
    C --> F
    C --> G
    D1 --> H
    E1 --> I
    F1 --> J
    G1 --> H1
    H1 --> I1
    I1 --> J1

Loading

Example Flow

sequenceDiagram
    participant U as User
    participant C as Claude
    participant W as Wikidata
    participant R as RAG
    participant G as Gemini
    participant O as Output
    
    U->>C: Research Objective: "AI Cancer Detection"
    C->>C: Generate 5 Core Terms: ["computer vision", "medical imaging", "machine learning", "diagnostic accuracy", "clinical validation"]
    C->>W: Query Wikidata for each term
    W->>C: Return related concepts and practices
    C->>C: Format scientific context
    C->>R: Enhanced search queries + scientific context
    R->>R: Retrieve relevant documents
    R->>G: Combined context (documents + scientific foundation)
    G->>G: Understand current state of knowledge
    G->>G: Generate proposal that builds upon foundation
    G->>O: High-quality grant application text
    
    Note over G: Gemini now understands:<br/>- What's established in the field<br/>- What practices currently exist<br/>- How to position innovation<br/>- How to build upon foundation

Loading

Key Implementation Points

1. Enhanced Enrichment Prompt: Add instruction to generate exactly 5 core scientific terms
2. Wikidata Integration: SPARQL queries to expand scientific concepts
3. Context Formatter: Clear explanation that scientific context is foundation, not proposal
4. Pipeline Integration: Inject enhanced context before retrieval and writing
5. Writing Enhancement: Modify prompts to use scientific foundation context

This design ensures that the RAG pipeline provides rich, factual scientific context while maintaining clear separation between established knowledge and innovative proposals.

[yiftachashkenazi]

@Goldziher here it is

[Goldziher]

cool. ok so looking at these diagrams - you can see the main issues with the suggested design spatially.

See how the enrichment phase suddenly has 5 steps horizontally:

Whats happening here is a lot of work in a single step, which we also do concurrently to save on time (this is rather slow). We also dont use claude here, we use Gemini (speed and money).

Its important to think on how we can save netwrok trips and optimize for performance.

[yiftachashkenazi]

So just to confirm: you’re suggesting a separate API call using Gemini (possibly version 2.0) that generates the context keywords, and is only connected to the Wiki extraction function. Then both would rejoin the flow at the RAG creation step, right?

[yiftachashkenazi]

@Goldziher waiting for a green light

[Goldziher]

no, what im saying is two things:

1. please fix the broken mermaid diagrams 😁
2. the problem with make wikipedia calls, i i understnad you correct, is that we are making an HTTP request. Each network request we are doing slows us down. This means we need to think of an optimal pattern to REDUCE the number of HTTP requests we make in this step to a minimum. That is, we need to optimize it in advance to use concurrency, or do a single HTTP request.

[yiftachashkenazi]

@Goldziher
Thanks — I’ll fix the Mermaid diagrams.

For now, I’ll focus on:
1. Generating 5 core terms during enrichment.
2. Using batch requests for the Wikidata step (without building a local dataset at this stage).

Let me know if you think there’s a better approach or optimization here.