Steel_Plate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steel Plate Defect Detection\n",
    "\n",
    "__Background__: Steel plate defects are extracted from photos of several faulty steel plates with surface imperfections. Images analysis revealed 27 different features to describe the steel fault. A total of 6 unique types of faults are categorized, with a final category of \"other faults\" for any type of fault that does not fit into the other specific 6 categories.\n",
    "\n",
    "__Features__: There are 27 features that are used to predict the steel faults. These features are extracted from steel plate samples. Computer vision can automatically extract some of this information from images or manually extracted with a user inspecting each plate defect or photo of the steel plate.\n",
    "\n",
    "__Objective__: Develop a neural network in Keras / TensorFlow for a multi-class classification problem. \n",
    "\n",
    "Additional Information from [Kaggle](https://www.kaggle.com/uciml/faulty-steel-plates) and [Machine Learning for Engineers](https://apmonitor.com/pds/index.php/Main/SteelPlateFaults).\n",
    "\n",
    "<img align=left width=400px src='https://apmonitor.com/pds/uploads/Main/steel_plates.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:06.809832Z",
     "start_time": "2021-02-16T18:25:02.153199Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# scikit-learn packages\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split \n",
    "from sklearn.feature_selection import SelectKBest,chi2\n",
    "\n",
    "# keras packages\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data and display a sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:06.857821Z",
     "start_time": "2021-02-16T18:25:06.811823Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('http://apmonitor.com/pds/uploads/Main/steel.txt')\n",
    "data.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Outliers\n",
    "\n",
    "Detect outliers and remove bad data if present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = data.columns[:-7]\n",
    "data[features].plot(kind='box',subplots=True,layout=(5,6),figsize=(12,10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show Balance of Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = data.columns[:-7]\n",
    "labels = data.columns[-7:]\n",
    "X = data[features]\n",
    "y = data[labels]\n",
    "y.idxmax(axis=1).value_counts().plot(kind='bar',figsize=(12,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scale data, select features, and split into train and test dataset\n",
    "\n",
    "Keras also has a validation option in the keras fit method with ''validation_split=0.2''."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:06.873822Z",
     "start_time": "2021-02-16T18:25:06.858823Z"
    }
   },
   "outputs": [],
   "source": [
    "# Scale data\n",
    "s = MinMaxScaler()\n",
    "data_s = s.fit_transform(data)\n",
    "data_s = pd.DataFrame(data_s,columns=data.columns)\n",
    "\n",
    "# Split data into X and y\n",
    "features = data.columns[:-7]\n",
    "labels = data.columns[-7:]\n",
    "X = data_s[features]\n",
    "y = data_s[labels]\n",
    "\n",
    "# Train/test split\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X,y,test_size=0.2,shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select Best Features\n",
    "\n",
    "Select the best features for the classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:07.235776Z",
     "start_time": "2021-02-16T18:25:07.016826Z"
    }
   },
   "outputs": [],
   "source": [
    "bestfeatures = SelectKBest(score_func=chi2, k='all')\n",
    "fit = bestfeatures.fit(X,y)\n",
    "plt.figure(figsize=(12,4))\n",
    "plt.bar(x=X.columns,height=fit.scores_,)\n",
    "plt.xticks(rotation=90);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a one-layer neural network in Keras\n",
    "- 27 inputs -> [8 hidden nodes] -> 7 outputs\n",
    "- Use categorical cross entropy as the loss metric\n",
    "- Use 'softmax' activation in the final layer to return probabilities\n",
    "- Utilize a validation split and plot the loss function to observe any evidence of overfitting\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:48.953260Z",
     "start_time": "2021-02-16T18:25:07.236747Z"
    }
   },
   "outputs": [],
   "source": [
    "# Classification neural network with Keras\n",
    "model = Sequential()\n",
    "model.add(Dense(8, input_dim=Xtrain.shape[1], activation='relu'))\n",
    "model.add(Dense(ytrain.shape[1], activation='softmax'))\n",
    "\n",
    "# Compile model\n",
    "model.compile(loss='categorical_crossentropy', \\\n",
    "              optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "# Train model\n",
    "result = model.fit(Xtrain,ytrain,epochs=1000,\\\n",
    "                   validation_split=0.2,verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the Train and Validation Loss Functions\n",
    "\n",
    "Generate the train and validation loss functions to determine number of epochs and evidence of overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:49.360566Z",
     "start_time": "2021-02-16T18:25:48.956141Z"
    }
   },
   "outputs": [],
   "source": [
    "plt.semilogy(result.history['loss'],label='loss')\n",
    "plt.semilogy(result.history['val_loss'],label='val_loss')\n",
    "plt.legend()\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Epoch')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion:** Look at the plots for loss. Do you see evidence of overfitting? How many epochs are needed for a good fit? Retrain the model with the proper number of epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use test set to analyze model effectiveness\n",
    "\n",
    "Generate predicted labels and probabilities of those labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:49.469782Z",
     "start_time": "2021-02-16T18:25:49.363582Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Make predictions and save in dataframe\n",
    "yp = model.predict(Xtest)\n",
    "yp = pd.DataFrame(yp,columns=ytest.columns)\n",
    "\n",
    "# Extract predicted labels and probabilities\n",
    "predicted_label = yp.idxmax(axis=1)\n",
    "predicted_prob = yp.max(axis=1)\n",
    "actual_label = ytest.idxmax(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display probabilities\n",
    "\n",
    "Display probabilities of faults with the most likely label highlighted and the actual label displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:49.863504Z",
     "start_time": "2021-02-16T18:25:49.470634Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "yp['Actual fault'] = actual_label.values\n",
    "yp.style.highlight_max(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-16T18:25:50.147550Z",
     "start_time": "2021-02-16T18:25:49.864506Z"
    }
   },
   "outputs": [],
   "source": [
    "cm = confusion_matrix(predicted_label,actual_label)\n",
    "sns.heatmap(cm,annot=True)\n",
    "plt.savefig('confusion_matrix.png',dpi=600)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score and Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score1, acc1 = model.evaluate(Xtrain, ytrain)\n",
    "score2, acc2 = model.evaluate(Xtest, ytest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scikit-Learn Classifiers\n",
    "\n",
    "Repeat analysis with 8 supervised learning methods. See [Classification Overview](https://apmonitor.com/pds/index.php/Main/ClassificationOverview) for additional information on supervised learning methods. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neural_network import MLPClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.multioutput import MultiOutputClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ada = AdaBoostClassifier()\n",
    "lr = LogisticRegression()\n",
    "nb = GaussianNB()\n",
    "knn = KNeighborsClassifier()\n",
    "dt = DecisionTreeClassifier()\n",
    "rf = RandomForestClassifier()\n",
    "svc = SVC()\n",
    "mlp = MLPClassifier(max_iter=10000)\n",
    "clist = [ada,lr,nb,knn,dt,rf,svc,mlp]\n",
    "mlist = [None]*8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i,m in enumerate(clist):\n",
    "    mlist[i] = MultiOutputClassifier(m).fit(Xtrain,ytrain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and Test Set Scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = pd.DataFrame()\n",
    "n = ['Keras/TF NN']; s1 = [score1]; s2 = [score2]\n",
    "for i,m in enumerate(mlist):\n",
    "    n.append(str(m.estimator))\n",
    "    s1.append(m.score(Xtrain,ytrain))\n",
    "    s2.append(m.score(Xtest,ytest))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.bar(n,s1,label='Train')\n",
    "plt.bar(n,s2,width=0.5,label='Test')\n",
    "plt.xticks(rotation=90)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}