Metadata Generation using LLMs - Solution Methodology

Project Overview

This project aims to streamline metadata generation using Large Language Models (LLMs), specifically leveraging Llama2, Retrieval-Augmented Generation (RAG), and AWS. Metadata provides context, meaning, and structure to data, enhancing workflows and improving information retrieval. This project automates metadata creation, reducing manual efforts while increasing accuracy and efficiency.

Key Features

• Automates metadata generation for project documentation and code files.
• Leverages Llama2 for natural language understanding and FAISS for efficient vector database management.
• Combines RAG techniques with advanced LLMs for context-aware metadata creation.
• Includes a user-friendly interface powered by Streamlit.

---

Tech Stack

• Programming Language: Python 3.10.4
• Key Libraries: langchain, torch, Textract, Tika, HuggingFace Transformers, FAISS
• Model: Llama2 (13 billion parameters, quantized version)
• Cloud Platform: Amazon Web Services (AWS)

---

Setup Instructions

Prerequisites

• Python 3.10.4 installed on your system.
• AWS account for setting up EC2 instances with GPU support.
• Ensure all necessary libraries are installed using the requirements.txt file.

Installation

1. Clone the repository:

   git clone https://github.com/your-repo-name.git
   cd your-repo-name
   

install dependencies

pip install -r requirements.txt