Neural Network Performance Analysis

_{short alias lmurs}

The original version of the NN Stat project was created by Waleed Khalid at the Computer Vision Laboratory, University of Würzburg, Germany.

Overview 📖

Automated conversion of LEMUR data into Excel format with statistical visualizations. It is developed to support the NN Dataset and NNGPT projects.

Create and Activate a Virtual Environment (recommended)

For Linux/Mac:

python3 -m venv .venv
source .venv/bin/activate

For Windows:

python3 -m venv .venv
.venv\Scripts\activate

Environment for NN Stat Contributors

Run the following command to install all the project dependencies:

python -m pip install --upgrade pip
pip install -r requirements.txt

Installation with the LEMUR Dataset

pip install nn-stat[dataset]

Usage

python -m ab.stat.export

Data and statistics are stored in the stat directory in Excel files and PNG/SVG plots.

To use 'ab/stat/nn_analytics.ipynb' install jupyter:

pip install jupyter

and run jupyter notebook:

jupyter notebook --notebook-dir=.

Update of NN Dataset

Install from GitHub to get the most recent code and statistics updates:

rm -rf db
pip uninstall -y nn-dataset
pip install --no-cache-dir git+https://github.com/ABrain-One/nn-dataset

Installing the stable version:

rm -rf db
pip install nn-dataset --upgrade

Docker

All versions of this project are compatible with AI Linux and can be seamlessly executed within the AI Linux Docker container:

docker run -v /a/mm:. abrainone/ai-linux bash -c "PYTHONPATH=/a/mm python -m ab.stat.export"

Some recently added dependencies might be missing in the AI Linux. In this case, you can create a container from the Docker image abrainone/ai-linux, install the missing packages (preferably using pip install <package name>), and then create a new image from the container using docker commit <container name> <new image name>. You can use this new image locally or push it to the registry for deployment on the computer cluster.

Tasks & datasets

Tasks & datasets

Current LEMUR Statistics

Image Classification

Image Captioning, Image Segmentation, Text Generation

Accuracy VS Duration Best Models

Best Per Run Distribution

Best Per Run VS Duration

Model Rank Heatmap

The best models for image classification, image segmentation and text generation tasks across all the datasets.

Model performance and variability across runs

The bars show the average result for each model, while the error bars indicate how much those results vary across different runs when enough data is available.

The plot shows how performance develops over time. Error bars reflect how much the results change across different runs with different settings, and they are only included when enough data is available.

The confidence intervals show how much results vary across different runs. They are not meant to compare models statistically or indicate which model is significantly better.

The idea and leadership of Dr. Ignatov