This project provides a comprehensive framework for evaluating and comparing Large Language Models (LLMs). The goal of the system is to help developers, researchers, and organizations better understand the strengths, weaknesses, performance, and practical use cases of different language models through a unified evaluation pipeline.
Currently, the application supports evaluation of the following models:
- ChatGPT (OpenAI API)
- Gemini (Google Gemini API)
- PLLuM (HuggingFace Inference API)
The framework was designed in a modular and extensible way, allowing additional LLMs to be integrated and evaluated with minimal changes to the codebase.
- Interactive chat interface for each model
- Dataset-based automated evaluation
- Manual prompt evaluation
- Multiple evaluation metrics
- Model connection verification
- Evaluation result storage
- Modular API-based architecture
- Support for external and local model providers
The evaluation process is based on datasets containing prompts and expected evaluation criteria. The datasets may either:
- be manually created by the project team,
- or originate from established state-of-the-art benchmarks.
The system evaluates models in several categories, including:
- Reasoning
- Summarization
- Coding
- Polish Language Tasks
- General / Miscellaneous Tasks
The framework supports multiple evaluation strategies:
- Exact Match
- Required Keywords Match
- Optional Keywords Match
- ROUGE-based Similarity Score
For manual evaluation, the user may provide:
- a reference answer,
- required keywords,
- optional keywords,
depending on the selected evaluation method.
When evaluating using a prepared dataset, all required evaluation fields are loaded automatically, eliminating the need for manual input.
The application sends prompts to the selected model, retrieves generated responses, evaluates them using the selected metrics, and stores the results for further analysis and comparison.
The application includes a built-in model verification system.
Before evaluation, users may verify the connection to a selected model.
The verification mechanism confirms:
- successful communication with the model provider,
- the exact model identifier being used,
- provider information,
- response latency,
- and response availability.
This helps ensure that the application is connected to the correct model and that responses are generated by the expected provider.
The project consists of three main layers:
Implemented using Streamlit, the frontend provides:
- interactive chat interfaces,
- dataset upload functionality,
- evaluation configuration,
- visualization of evaluation results,
- and model verification tools.
Implemented using FastAPI, the backend:
- routes requests to the appropriate model,
- performs evaluation,
- stores results,
- and exposes REST API endpoints.
Dedicated model clients are responsible for communication with:
- OpenAI API,
- Google Gemini API,
- HuggingFace Inference API,
- and local models served through LM Studio.
git clone <repository-url>pip install -r requirements.txtFor demonstration and evaluation purposes, API keys required to run the application are already included in the provided .env file.
The application uses:
- OpenAI API for ChatGPT
- Google Gemini API for Gemini
- HuggingFace API for PLLuM
Open the first terminal window and run:
python -m uvicorn src.api:app --reloadOpen the second terminal window and run:
python -m streamlit run src/frontend/app.pyAfter startup, the application will be available at:
http://localhost:8501
The backend API will run at:
http://127.0.0.1:8000
- Internet connection is required for OpenAI, Gemini, and HuggingFace API communication.
- The application may take several seconds to generate responses depending on the selected model and prompt complexity.
- API usage may consume credits associated with the configured provider accounts.