DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs

To use DeepDiagnosis, you need to add our callback as a subclass in your keras.callbacks.py file.

The core principle of our callback is to get a view of the internal states and statistics of the model during training.

Then you can pass our callback DeepDiagnosis() to the .fit() method of a model as follows:

callback = keras.callbacks.DeepDiagnosis(inputs, outputs, layer_number, batch_size, startTime)
model = keras.models.Sequential()
model.add(keras.layers.Dense(64))
model.add(keras.layers.Activation(activations.relu))
model.compile(keras.optimizers.SGD(), loss='mse')
model.fit(np.arange(100).reshape(5, 20), np.zeros(5), epochs=10, batch_size=1, 
...                     callbacks=[callback], verbose=0)

Prerequisites

The version numbers below were confirmed to work with the current release:

python 3.6.5
Keras  2.2.0
Keras-Applications  1.0.2
Keras-Preprocessing 1.0.1  
numpy 1.19.2
pandas 1.1.5
scikit-learn 0.21.2
scipy 1.6.0
tensorflow 1.14.0

BibTeX Reference

If you find this paper useful in your research, please consider citing:

@inproceedings{wardat2021deepdiagnosis,
author={Mohammad Wardat and Breno Dantas Cruz and Wei Le and Hridesh Rajan},
title={DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs}, 
booktitle = {ICSE'22: The 44th International Conference on Software Engineering},
location = {Pittsburgh, PA, USA},
month = {May 21-May 29, 2022},
year = {2022},
entrysubtype = {conference}
}

This repository contains the reproducibility package of DeepDiagnosis

Extractor Folder:

• Contains the source code to extract (.h5) to source code

AUTOTRAINER Model Code:

• Contains the source code of all AUTOTRAINER Models

DeepDiagnosis Result:

• Contains the results of DeepDiagnosis from AUTOTRAINER dataset

MOTIVATING EXAMPLE:

• Contains the results of the motivating example using AUTOTRAINER

SaturatedActivation:

• Contains the experiments of saturated activation for Sigmoid and Tanh

Difference threshold:

• Contains the result of AUTOTRAINER on normal models with different threshold (accuracy =100%)

Table 6

• Complete result of Table 6

Normal model with 20 % Accuracy

• this model(MNIST_Normal/double_random_99fe3625-3c58-4766-968e-7d40401237fe) is detected by DeepDiagnosis and the accuracy = 20%

DeepLocalize Result

• Contains the results of DeepLocalize from AUTOTRAINER dataset

UMLUAT Result

• Contains the results of UMLUAT from AUTOTRAINER dataset