RAG for PDFs and Websites

This project provides a tool to extract and query information from PDFs and websites using Retrieval-Augmented Generation (RAG) techniques. It leverages Streamlit for the user interface, LangChain for text processing, and Hugging Face models for embeddings and language generation.

Features

• Extract text from websites and PDFs
• Split text into manageable chunks
• Generate embeddings using Hugging Face models
• Perform similarity search on the text chunks
• Answer questions based on the extracted text

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/rag-project.git
   cd rag-project

2. Create a virtual environment:

   python -m venv venv
   source venv/bin/activate
   venv\Scripts\activate (for Windows)

3. Install the required dependencies:

   pip install -r requirements.txt

4. Set up your environment variables:
   • Create a .env file in the root directory.
   • Add your Hugging Face API token to the .env file:

     HUGGINGFACEHUB_API_TOKEN=your_huggingface_api_token
     

Usage

For Websites

1. Run the Streamlit app for websites:

   streamlit run rag_web.py

2. Enter the URL of the website you want to query.
3. Ask your question in the provided input box.

For PDFs

1. Run the Streamlit app for PDFs:

   streamlit run rag_pdf.py

2. Upload the PDF file you want to query.
3. Ask your question in the provided input box.