Vitreoretinal Surgery LLM
Surgical trainees rely on information from textbooks and real-time instruction from attending surgeons in the operating room. The authors of the The Duke Manuals of Ophthalmic Surgery aimed to to improve upon this model by creating a textbook series that contains both traditional surgical information as well tips/tricks from attending surgeons for each procedure. The goal of this project was to create a local llamafile-based RAG system with access to the Duke Manual of Vireoretinal Surgery. This will allow ophthalmology residents and vitreoretinal surgery fellows to interact with the information in real time and help them prepare for OR cases and their board exams. If trainees find this system useful, we will expand this effort to cover other medical specialties.
Developing the system architecture involved sourcing chapter PDFs from the publisher, creating text chunks, embedding text chunks, and storing them in a Chroma database. User queries are entered through a simple HTML/Flask interface and embeddings are generated for these queries. The top 3 most similar text passages to the user query are extracted from the Chroma database using semantic search. These text passages are fed into a local llamafile language model (Mistral 7B Instruct) along with the user query and a prompt. The prompt used was as follows:
Answer the question based only on the following context:
You are an experienced vitreoretinal surgeon speaking with trainees. You can answer detailed questions about vitreoretinal surgery concisely and accurately. Keep response length short (within 100 words if possible).
The system generates a response which is fed back to the user through the HTML/Flask interface. A diagram of the system architecture is provided below:
The performance of the Mistral 7B Instruct model was evaluated both with and without RAG augmentation. Responses were compared to an ideal response (sourced from the text) via cosine similarity. Results are provided in the plot below. The RAG system outperformed the LLM-only approach for all questions, but to varying degrees. The LLM-only approach struggled to provide concise answers for more complex questions without RAG augmentation. The code for this comparison is provided in the "notebooks" folder. The average latency for all prompts was approximately 35 seconds with a range of 22 - 55 seconds and standard deviation of 10 seconds. The questions used are provided in the "model_eval_questions.txt" text file.
The Mistral 7B Instruct model was also compared to the TinyLlama-1.1B model. The TinyLlama model had difficulty producing output on occasion and would return responses asking the user to find other sources for the requested information. As such, the Mistral 7B Instruct model was selected for further development tasks.
Download the Mistral 7B Instruct llamafile from the Mozilla Ocho Repository: https://github.com/Mozilla-Ocho/llamafile
Once you have downloaded the Mistral 7B llamafile run the following command in your terminal to grant permission for your computer to run the model (only need to do this once)
chmod +x mistral-7b-instruct-v0.2.Q4_0.llamafile
You must also clone this repository to your local machine. Navigate to the path you'd like to store the files then run the following command:
git clone https://github.com/jsway1/RetinaRAG.git
Activate the model by running the following command (must be done every time you start the application):
./mistral-7b-instruct-v0.2.Q4_0.llamafile
Create a virtual environment by running:
python3 -m venv venv
Activate the virtual environment by running:
source venv/bin/activate
Build the docker image by running:
docker build -t retinarag -f Dockerfile.dockerfile .
After the docker image is created, run the following:
docker run -p 5050:5050 retinarag
The system will prompt you to open a webpage at localhost:5050 where you should see the RetinaLLM frontend. You can now ask questions to the RAG application
Create a virtual environment by running:
python3 -m venv venv
Activate the virtual environment by running
source venv/bin/activate
Install dependencies by running
pip3 install -r requirements.txt
Test database access, embedding generation, LLM querying, and accuracy of responses by running the following command:
pytest
You should receive a green message indicating that 4/4 tests passed
Main file that runs all components of the application
Dockerfile that creates docker image of application and runs app.py
The system was trained on PDFs provided by the authors of the Duke Manual of Vitreoretinal Surgery. This information is protected by copyright, so we were not able to publish these PDFs along with this repository. We have included open source sample PDFs describing common conditions in vitreoretinal surgery to allow prospective users to test the database creation system.
Contains files associated with the chroma database that stores the text chunk embeddings from the textbook chapters used.
Contains Jupyter notebooks used during the app development process
createdb.py - script that contains text chunk generation, text chunk embedding, and chroma database creation code
query_RAG.py - script that contains RAG query function
RetinaLLM.html - HTML template for Flask frontend
test_RAG_application.py - script that tests app database access, embedding creation, LLM query, and response accuracy
Contains 10 test questions and ideal answers used to evaluate performance of Mistral 7B Instruct model both with and without RAG