Insight Forge

A powerful tool for automatically scraping websites, processing PDFs, and generating FAQs using AI.

Table of Contents

• Overview
• Features
• Prerequisites
• Configuration
• System Requirements
• Usage
  • Running with Docker
  • Interactive Usage
  • Output Files
• Technical Details
  • Scraping Process
  • PDF Processing
  • FAQ Generation
  • Data Persistence
  • Error Handling
• Development
  • Key Components
  • Building from Source
  • Docker Build
• Best Practices
• Troubleshooting
• Maintenance
• Security Considerations
• Tech Stack and Dependencies

Overview

Insight Forge is a Rust-based application that automates the process of scraping web content, processing PDFs, and generating FAQs. It uses headless Chrome for reliable web scraping, OCR capabilities for PDF text extraction, and AI APIs for intelligent FAQ generation.

Features

• 🌐 Automated website scraping with JavaScript rendering support
• 📄 PDF processing with OCR capabilities
• 📝 Conversion of HTML content to Markdown
• ❓ AI-powered FAQ generation using GROQ
• 🔄 Intelligent content processing and cleaning
• 📂 Organized content storage structure
• 🐳 Containerized deployment with Docker

Prerequisites

• Docker and Docker Compose
• Chrome WebDriver (running on port 9515)
• GROQ API key
• Tesseract OCR (for PDF processing)
• poppler-utils (for PDF to image conversion)

Configuration

Copy .env.example to .env and fill in your values:

cp .env.example .env

Required Variables

Variable	Description
API_KEY	Secret key for authenticating API requests to Insight Forge
OPENAI_COMPATIBLE_API_URL	Base URL for your OpenAI-compatible API endpoint (e.g. https://api.openai.com/v1/chat/completions)
OPENAI_COMPATIBLE_API_URL	Router endpoint for parallel/batched LLM requests
GROQ_API_KEY	Groq API key for FAQ generation
DATABASE_URL	PostgreSQL connection string
LANGFUSE_PUBLIC_KEY	Langfuse public key for observability
LANGFUSE_SECRET_KEY	Langfuse secret key for observability
LANGFUSE_BASE_URL	Langfuse instance URL (default: https://cloud.langfuse.com)

Optional Variables

Variable	Default	Description
MODE	cli	Run mode: cli or api
APP_PORT	11111	Server port
IF_UI_BASE_URL	http://localhost:3000	UI base URL for CORS
OPENAI_COMPATIBLE_MODEL	openrouter/qwen/qwen3.6-plus	Primary model for AI tasks
OPENAI_COMPATIBLE_MODEL_FALLBACK	openrouter/qwen/qwen3.5-plus-02-15	Fallback model
OPENAI_COMPATIBLE_STREAM_ENABLED	true	Enable streaming responses
VISION_EXTRACTION_ENABLED	false	Enable vision-based PDF extraction
VISION_PDF_TO_IMG_TTL	1440000	Cache TTL for PDF-to-image conversion (ms)
FAQ_ENABLED	true	Enable FAQ generation
CONTAINER_NAME	insight-forge-server:v0.3.4	Docker container image name

System Requirements

The Docker container is configured with the following resource limits:

• CPU: 0.8 cores
• Memory: 2GB

Usage

Running with Docker

1. Build and start the container:

docker-compose up -d

Interactive Usage

When running the application, you'll be prompted to:

1. Specify the number of FAQs you want per page
2. Enter URLs to scrape or paths to PDF files (one per line)
3. Press Enter twice when done entering URLs/paths

Output Files

For each scraped URL or processed PDF, the system generates files with sanitized filenames (spaces replaced with underscores, punctuation removed) in separate directories:

• raw_html/{sanitized_filename}.md: Complete raw HTML content or PDF text
• raw_text_cleaned/{sanitized_filename}.md: Cleaned HTML content or processed PDF text
• structured_markdown/{sanitized_filename}.md: Converted Markdown content
• faq/faq_{sanitized_filename}.md: Generated FAQs

Note: Filenames are automatically sanitized to remove special characters and replace spaces with underscores for cross-platform compatibility.

Technical Details

Scraping Process

1. Uses Chrome WebDriver with custom user agent
2. Implements scroll-to-bottom functionality
3. Waits for content loading (1.5s delay)
4. Processes and cleans HTML content

PDF Processing

1. Extracts embedded text from PDF using lopdf
2. Performs OCR on PDF content using Tesseract
3. Combines extracted text and OCR results

FAQ Generation

• Utilizes GROQ API for intelligent FAQ generation
• Customizable number of FAQs per page
• Markdown format output

Data Persistence

• Docker volumes for persistent storage:
  • ./server/data/scraped_content:/usr/server/app/server/data/scraped_content
  • ./server/data/faq:/usr/server/app/server/data/faq

Error Handling

• Comprehensive error handling with detailed messages
• Automatic retry mechanism for failed requests
• Graceful handling of existing files

Development

Key Components

• main.rs: Application entry point and orchestration
• scraper.rs: Web scraping functionality
• pdf_handler.rs: PDF processing and OCR
• faq_writer.rs: FAQ file management
• compose.yml: Docker configuration

Building from Source

1. Clone the repository
2. Install Rust and required dependencies
3. Run with cargo:

cargo run

Docker Build

docker build -t insight-forge .

Best Practices

1. Monitor resource usage, especially for large-scale scraping and PDF processing
2. Respect websites' robots.txt and rate limiting
3. Regular backup of the data directories
4. Monitor GROQ API usage and costs

Troubleshooting

1. WebDriver Connection Issues

   • Ensure Chrome WebDriver is running on port 9515
   • Check network connectivity

2. Memory Issues

   • Monitor Docker memory usage
   • Adjust memory limits in compose.yml if needed

3. FAQ Generation Failures

   • Verify GROQ API key validity
   • Check API rate limits
   • Review input content size

4. PDF Processing Issues

   • Ensure Tesseract OCR is properly installed
   • Check PDF file integrity
   • Install required dependencies:

     sudo apt-get update
     sudo apt-get install tesseract-ocr libtesseract-dev libleptonica-dev poppler-utils

5. Dependency Installation

   • If building outside of Docker, ensure all required dependencies are installed:

     sudo apt-get update
     sudo apt-get install tesseract-ocr libtesseract-dev libleptonica-dev poppler-utils

Maintenance

The application is configured to restart automatically (restart: always) and includes resource limits to prevent system overload.

Security Considerations

• Store API keys securely using environment variables
• Regular security updates for dependencies
• Monitor and log access patterns
• Implement rate limiting for production deployments

Tech Stack and Dependencies

1. Rust: The core programming language used for building the application.

   • Provides high performance and memory safety.

2. Selenium WebDriver: Used for web scraping with JavaScript support.

   • Allows interaction with web browsers for dynamic content scraping.

3. lopdf: A Rust library for parsing and manipulating PDF files.

   • Used for extracting embedded text from PDFs.

4. Tesseract: An OCR engine used for extracting text from images in PDFs.

   • Enables text extraction from scanned documents or image-based PDFs.

5. GROQ API: Used for generating FAQs from the extracted content.

   • Provides AI-powered question generation and answering capabilities.

6. anyhow: A Rust library for flexible error handling.

   • Simplifies error management across the application.

7. tokio: An asynchronous runtime for Rust.

   • Enables efficient handling of concurrent operations.

8. reqwest: An HTTP client for Rust.

   • Used for making API requests to GROQ.

9. serde: A framework for serializing and deserializing Rust data structures.

   • Facilitates working with JSON data from APIs.

10. log: A logging facade for Rust.

    • Provides a flexible logging system for the application.

11. dotenvy: A Rust library for loading environment variables from a file with multiline support.

    • Manages configuration and sensitive data.

12. Docker: Used for containerization and deployment.

    • Ensures consistent runtime environment across different systems.

This tech stack combines powerful libraries and tools to create a robust, efficient, and versatile web scraping and FAQ generation system, now with added support for PDF processing and OCR capabilities.