This project leverages AI chatbots to classify radiology reports using the Arbeitsgemeinschaft Osteosynthesefragen (AO) Fracture and Dislocation Classification Compendium. We compare the performance of generic chatbots, context-aware chatbots, and human readers.
Radiologists adeptly describe fracture morphology. However, translating these into the AO classification is challenging. This project evaluates generic chatbots and context-aware chatbots informed by AO's vector-index. Chatbots identify AO codes faster than humans but have varied accuracy. Context-specific knowledge improves chatbot performance, suggesting that refined context is crucial for maximizing ChatGPT's potential.
- Download the notebook file
Demo of AO Chatbots.ipynb. - Upload the notebook to Google Colab, JupyterLab, or your preferred Jupyter notebook environment.
- Set your OpenAI API key and the folder path containing the AO guidelines in the corresponding cells.
- Execute the cells in the notebook to initialize the index, launch the interface, and interact with the AI chatbots.
The script utilizes various AI models to generate AO codes from radiology reports:
- FracChat (using GPT4): Uses GPT-4 to query the AO guidelines index.
- FracChat (using GPT 3.5-Turbo): Uses GPT-3.5-Turbo to query the AO guidelines index.
- GPT-3.5-Turbo: Provides a response using the standalone GPT-3.5-Turbo model.
- GPT-4: Provides a response using the standalone GPT-4 model.
All model responses are aggregated and showcased in the interface.
- Acquire an OpenAI API key from OpenAI and set the
OPENAI_API_KEYenvironment variable in the script:
os.environ["OPENAI_API_KEY"] = 'sk-ENTER_YOUR_API_CODE'- Specify the directory containing the AO guidelines in PDF format:
FOLDER_PATH = "AO"- Execute the script to initiate the index and unveil the interface.
Link to full publication: Performance of ChatGPT, human radiologists, and context-aware ChatGPT in identifying AO codes from radiology reports