DocPilot - Multimodal Document Processing and Query System

A Flask-based web application that allows users to upload PDF documents, extract and summarize their content (text, images, and tables), and query the processed data using a natural language interface. The application leverages Groq's LLaMA models, HuggingFace embeddings, and FAISS for vector-based retrieval.

---

Table of Contents

1. Features
2. Technologies Used
3. Setup Instructions
4. Application Workflow
5. Usage
6. Limitations

---

Features

• Upload and process PDF files.
• Extract text, images, and tables from documents.
• Summarize extracted content using Groq's LLaMA 3.3 70B model (free).
• Store summarized content in a FAISS vector database.
• Perform semantic queries to retrieve and display relevant information.
• Display relevant images alongside textual answers.

---

Technologies Used

• Flask: For building the web application.
• Groq LLaMA 3.3 70B: For summarizing content and generating answers (free tier).
• HuggingFace Embeddings: For generating vector embeddings (free, no API key needed).
• FAISS: For efficient vector-based document retrieval.
• Unstructured: For PDF content extraction.
• LangChain: To orchestrate LLM workflows.
• Docker: For containerized deployment.

---

Setup Instructions

Prerequisites

• Python 3.11+
• A Groq API key (free at console.groq.com)

Local Installation

1. Clone the repository:

   git clone https://github.com/dhrubak2004/DocPilot.git
   cd DocPilot

2. Install system dependencies (Linux/Mac):

   apt-get install libgl1 libglib2.0-0 poppler-utils tesseract-ocr libmagic1

3. Install Python dependencies:

   pip install -r requirements.txt

4. Set up environment variables: Create a .env file:

   GROQ_API_KEY=your_groq_api_key

5. Run the Flask app:

   python app.py

6. Access the app at http://127.0.0.1:5000

Deployed Version

Live at: https://docpilot-production-4bc3.up.railway.app

---

Application Workflow

1. Upload a PDF Document

• Allowed file types: .pdf
• Text-based PDFs work best (Word converted to PDF)

2. Content Extraction

• Uses unstructured.partition.pdf with fast strategy to extract:
  • Text blocks, titles, narrative text, list items
  • Images (saved as JPG)
  • Tables

3. Content Summarization

• All text is batched and summarized using Groq LLaMA 3.3 70B
• Tables are summarized separately
• Images are stored with a placeholder summary

4. Store in Vector Database

• Summarized content stored in FAISS using HuggingFace all-MiniLM-L6-v2 embeddings

5. Query the System

• User submits a natural language query
• Relevant content retrieved from FAISS
• Detailed response generated by LLaMA 3.3 70B
• Relevant images displayed alongside answers

---

Usage

Uploading a Document

1. Navigate to the upload page
2. Select a PDF file and click "Upload"
3. Wait for "File uploaded and processed successfully. Ready for questions!"

Querying the System

1. Enter a natural language query
2. Click "Submit"
3. View the detailed response and associated images

---

Limitations

• Only .pdf files supported (max 16MB)
• Scanned/image-based PDFs may not extract text properly
• Password protected PDFs not supported
• Vectorstore resets on server restart (re-upload required)
• Groq free tier: 500 requests/day, 6000 tokens/minute

---

Future Improvements

• Support for scanned PDFs via OCR
• Support for .docx and .xlsx formats
• Persistent vector storage across sessions
• Multi-language support
• Enhanced image summarization with vision models

---

License

This project is licensed under the MIT License.