TopicMarker-RAG

A powerful content generation backend for creating comprehensive lesson plans and educational content. This service uses a hierarchy-based approach combined with real-time web scraping and Google's Gemini LLM to generate accurate, up-to-date MDX content.

Note: Despite the name "RAG", this service does not use traditional Retrieval-Augmented Generation with vector databases. The original plan was to pre-scrape content for all topics, store it in a vector database (Pinecone), and retrieve relevant chunks based on user queries. However, scraping large amounts of data upfront was taking too long and wasn't practical. Instead, we pivoted to a more dynamic approach where content is fetched in real-time from the web only when a user selects a specific topic, providing fresher and more relevant context for LLM generation. The name "RAG" was kept for historical reasons.

How It Works

1. Topic Hierarchy Generation: When a user searches for a topic, the LLM generates a structured hierarchy of main topics and subtopics
2. Content Generation: The user selects a topic and chooses one of three generation modes:
   • Web Crawling Mode: SERP API finds relevant websites → Crawl4AI scrapes their content → Gemini LLM generates MDX using this fresh context
   • URL-Based Mode: User provides specific URLs → Crawl4AI scrapes them → Gemini generates MDX from that content
   • LLM-Only Mode: Gemini generates content using only its training knowledge (no web scraping)
3. Content Refinement: Users can further improve generated content with additional web context or LLM assistance

Features

• Generate structured topic hierarchies for lesson plans
• Create MDX content from web sources with real-time scraping
• Three flexible content generation modes
• Refine content with LLM assistance
• Direct crawling-to-LLM pipeline (no vector database required)
• Multiple refinement options with web crawling integration

Setup

Prerequisites

• Python 3.8+
• pip

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/TopicMarker-RAG.git
   cd TopicMarker-RAG

2. Create and activate a virtual environment:

   For Linux/macOS:

   python -m venv venv
   source venv/bin/activate

   For Windows:

   python -m venv venv
   venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

   The requirements.txt file contains all the necessary dependencies for this project:

   # Core
   fastapi
   uvicorn[standard]
   python-dotenv
   pydantic
   pydantic-settings
   requests
   
   # Content Generation
   google-generativeai
   crawl4ai
   langchain
   langchain-community
   
   # Web Search
   duckduckgo-search
   googlesearch-python
   
   # Legacy (included for compatibility, not actively used)
   pinecone
   langchain-openai
   openai
   

4. Create a .env file in the root directory with the following variables:

   GEMINI_API_KEY=your_gemini_api_key
   
   # Optional - Legacy variables (not required for current implementation)
   # PINECONE_API_KEY=your_pinecone_api_key
   # PINECONE_ENVIRONMENT=your_pinecone_environment
   # PINECONE_INDEX_NAME=your_pinecone_index_name
   

Running the Application

Start the FastAPI server:

uvicorn app.main:app --reload

The API will be available at http://localhost:8000.

API Endpoints

Base Route

• GET / - Welcome message
  • Returns: {"message": "Welcome to the Lesson Plan RAG Backend!"}

Content Generation Routes

Note: These routes are prefixed with /rag for historical reasons, but they use real-time web scraping rather than traditional RAG retrieval.

Topic Generation

• POST /rag/search-topics
  • Input: {"query": "string", "limit": int} (default limit: 2)
  • Returns: A structured list of main topics and subtopics suitable for a lesson plan
  • Example: {"status": "success", "data": {"topics": [...]}}

MDX Generation

• POST /rag/single-topic

  • Input: {"selected_topic": "string", "main_topic": "string", "num_results": int} (default num_results: 2)
  • Returns: Comprehensive MDX content for a single topic
  • Example: {"status": "success", "data": {"mdx_content": "string", "crawled_websites": [...]}}

• POST /rag/single-topic-raw

  • Input: {"selected_topic": "string", "main_topic": "string", "num_results": int} (default num_results: 2)
  • Returns: Raw MDX content as plain text (not JSON)

• POST /rag/generate-mdx-llm-only

  • Input: {"selected_topic": "string", "main_topic": "string"}
  • Returns: MDX content generated using only LLM knowledge (no web crawling)
  • Example: {"status": "success", "data": {"mdx_content": "string"}}

• POST /rag/generate-mdx-llm-only-raw

  • Input: {"selected_topic": "string", "main_topic": "string"}
  • Returns: Raw MDX content generated using only LLM knowledge as plain text (not JSON)

URL-based MDX Generation

• POST /rag/generate-mdx-from-urls

  • Input: {"urls": ["string"], "selected_topic": "string", "main_topic": "string", "topic": "string" (optional), "use_llm_knowledge": bool}
  • URLs: 1 to 5 URLs to crawl
  • selected_topic: The subtopic to focus on
  • main_topic: The main topic that the selected topic belongs to
  • Returns: MDX content generated from multiple URLs
  • Example: {"status": "success", "urls": [...], "selected_topic": "string", "main_topic": "string", "mdx_content": "string"}

• POST /rag/generate-mdx-from-urls-raw

  • Input: {"urls": ["string"], "selected_topic": "string", "main_topic": "string", "topic": "string" (optional), "use_llm_knowledge": bool}
  • Returns: Raw MDX content as plain text (not JSON)

Content Refinement

• POST /rag/refine-with-selection

  • Input: {"mdx": "string", "selected_text": "string", "selected_topic": "string", "main_topic": "string", "question": "string"}
  • Returns: Refined content using the LLM with selected text and topic context
  • Example: {"status": "success", "data": {"answer": "string"}}

• POST /rag/refine-with-selection-raw

  • Input: {"mdx": "string", "selected_text": "string", "selected_topic": "string", "main_topic": "string", "question": "string"}
  • Returns: Raw refined content as plain text (not JSON)

• POST /rag/refine-with-crawling

  • Input: {"mdx": "string", "selected_text": "string", "selected_topic": "string", "main_topic": "string", "question": "string", "num_results": int} (default num_results: 2)
  • Returns: Refined content by first crawling relevant websites and then using the LLM
  • Example: {"status": "success", "data": {"answer": "string", "crawled_websites": [...]}}

• POST /rag/refine-with-crawling-raw

  • Input: {"mdx": "string", "selected_text": "string", "selected_topic": "string", "main_topic": "string", "question": "string", "num_results": int} (default num_results: 2)
  • Returns: Raw refined content as plain text (not JSON)

• POST /rag/refine-with-urls

  • Input: {"mdx": "string", "selected_text": "string", "selected_topic": "string", "main_topic": "string", "question": "string", "urls": ["string"]}
  • Returns: Refined content by crawling specific URLs provided by the user
  • Example: {"status": "success", "data": {"answer": "string", "crawled_websites": [...]}}

• POST /rag/refine-with-urls-raw

  • Input: {"mdx": "string", "selected_text": "string", "selected_topic": "string", "main_topic": "string", "question": "string", "urls": ["string"]}
  • Returns: Raw refined content as plain text (not JSON)

Testing

The project includes a comprehensive test suite organized in the tests directory:

• unit/ - Unit tests for individual components
• api/ - Tests for the API endpoints
• integration/ - Integration tests
• html/ - HTML test files for manual testing

Running Tests

To run all tests:

cd tests
python run_tests.py --all

To run specific test categories:

# Run only unit tests
python run_tests.py --unit

# Run only API tests (requires the API server to be running)
python run_tests.py --api

# Run only integration tests
python run_tests.py --integration

Manual Testing with HTML Test Page

To test the API endpoints through a browser interface:

cd tests
python serve_test_page.py

Then open your browser and navigate to http://localhost:8080/test_api.html

Dependencies

Core

• FastAPI - Web framework
• Uvicorn - ASGI server
• Pydantic - Data validation

Content Generation

• Google Generative AI (Gemini) - LLM for content generation
• crawl4ai - Web scraping and content extraction
• langchain - LLM framework for structured prompts

Web Search

• googlesearch-python - For finding relevant websites (SERP)
• duckduckgo-search - Alternative web search

Legacy (included but not actively used)

• Pinecone - Vector database (originally planned for RAG, not used in current implementation)
• OpenAI - Embeddings (originally planned for vector search)

License

MIT License