Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed the imbalanced classification notebook to work for TensorFlow 2.16 #2305

Merged
merged 4 commits into from
Aug 19, 2024
Merged

Fixed the imbalanced classification notebook to work for TensorFlow 2.16 #2305

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e30c5f5
Fixed the imbalanced classification notebook to work for TensorFlow 2.16
priankakariatyml May 2, 2024
5613e4e
Merge branch 'master' into imbalanced_classification_tf_2.16_fixes
priankakariatyml May 30, 2024
c5941e0
Cleared outputs in imbalanced_data.ipy
priankakariatyml May 30, 2024
4b60ead
Applied formatting
priankakariatyml May 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 45 additions & 48 deletions site/en/tutorials/structured_data/imbalanced_data.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,10 +258,10 @@
"train_df, val_df = train_test_split(train_df, test_size=0.2)\n",
"\n",
"# Form np arrays of labels and features.\n",
"train_labels = np.array(train_df.pop('Class'))\n",
"bool_train_labels = train_labels != 0\n",
"val_labels = np.array(val_df.pop('Class'))\n",
"test_labels = np.array(test_df.pop('Class'))\n",
"train_labels = np.array(train_df.pop('Class')).reshape(-1, 1)\n",
"bool_train_labels = train_labels[:, 0] != 0\n",
"val_labels = np.array(val_df.pop('Class')).reshape(-1, 1)\n",
"test_labels = np.array(test_df.pop('Class')).reshape(-1, 1)\n",
"\n",
"train_features = np.array(train_df)\n",
"val_features = np.array(val_df)\n",
Expand Down Expand Up @@ -291,18 +291,17 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "8a_Z_kBmr7Oh"
"id": "ueKV4cmcoRnf"
},
"source": [
"Given the small number of positive labels, this seems about right.\n",
"\n",
"Normalize the input features using the sklearn StandardScaler.\n",
"This will set the mean to 0 and standard deviation to 1.\n",
"\n",
"Note: The `StandardScaler` is only fit using the `train_features` to be sure the model is not peeking at the validation or test sets. "
"Note: The `StandardScaler` is only fit using the `train_features` to be sure the model is not peeking at the validation or test sets."
]
},
{
Expand Down Expand Up @@ -352,7 +351,7 @@
"\n",
"Next compare the distributions of the positive and negative examples over a few features. Good questions to ask yourself at this point are:\n",
"\n",
"* Do these distributions make sense? \n",
"* Do these distributions make sense?\n",
" * Yes. You've normalized the input and these are mostly concentrated in the `+/- 2` range.\n",
"* Can you see the difference between the distributions?\n",
" * Yes the positive examples contain a much higher rate of extreme values."
Expand Down Expand Up @@ -386,7 +385,7 @@
"source": [
"## Define the model and metrics\n",
"\n",
"Define a function that creates a simple neural network with a densly connected hidden layer, a [dropout](https://developers.google.com/machine-learning/glossary/#dropout_regularization) layer to reduce overfitting, and an output sigmoid layer that returns the probability of a transaction being fraudulent: "
"Define a function that creates a simple neural network with a densly connected hidden layer, a [dropout](https://developers.google.com/machine-learning/glossary/#dropout_regularization) layer to reduce overfitting, and an output sigmoid layer that returns the probability of a transaction being fraudulent:"
]
},
{
Expand All @@ -403,7 +402,7 @@
" keras.metrics.TruePositives(name='tp'),\n",
" keras.metrics.FalsePositives(name='fp'),\n",
" keras.metrics.TrueNegatives(name='tn'),\n",
" keras.metrics.FalseNegatives(name='fn'), \n",
" keras.metrics.FalseNegatives(name='fn'),\n",
" keras.metrics.BinaryAccuracy(name='accuracy'),\n",
" keras.metrics.Precision(name='precision'),\n",
" keras.metrics.Recall(name='recall'),\n",
Expand Down Expand Up @@ -432,7 +431,6 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "SU0GX6E6mieP"
Expand All @@ -456,7 +454,7 @@
"In the end, one often wants to predict a class label, 0 or 1, *no fraud* or *fraud*.\n",
"This is called a deterministic classifier.\n",
"To get a label prediction from our probabilistic classifier, one needs to choose a probability threshold $t$.\n",
"The default is to predict label 1 (fraud) if the predicted probability is larger than $t=50\\%$ and all the following metrics implicitly use this default. \n",
"The default is to predict label 1 (fraud) if the predicted probability is larger than $t=50\\%$ and all the following metrics implicitly use this default.\n",
"\n",
"* **False** negatives and **false** positives are samples that were **incorrectly** classified\n",
"* **True** negatives and **true** positives are samples that were **correctly** classified\n",
Expand All @@ -474,7 +472,7 @@
"The following metrics take into account all possible choices of thresholds $t$.\n",
"\n",
"* **AUC** refers to the Area Under the Curve of a Receiver Operating Characteristic curve (ROC-AUC). This metric is equal to the probability that a classifier will rank a random positive sample higher than a random negative sample.\n",
"* **AUPRC** refers to Area Under the Curve of the Precision-Recall Curve. This metric computes precision-recall pairs for different probability thresholds. \n",
"* **AUPRC** refers to Area Under the Curve of the Precision-Recall Curve. This metric computes precision-recall pairs for different probability thresholds.\n",
"\n",
"\n",
"#### Read more:\n",
Expand Down Expand Up @@ -520,8 +518,9 @@
"EPOCHS = 100\n",
"BATCH_SIZE = 2048\n",
"\n",
"early_stopping = tf.keras.callbacks.EarlyStopping(\n",
" monitor='val_prc', \n",
"def early_stopping():\n",
" return tf.keras.callbacks.EarlyStopping(\n",
" monitor='val_prc',\n",
" verbose=1,\n",
" patience=10,\n",
" mode='max',\n",
Expand Down Expand Up @@ -584,7 +583,7 @@
"id": "PdbfWDuVpo6k"
},
"source": [
"With the default bias initialization the loss should be about `math.log(2) = 0.69314` "
"With the default bias initialization the loss should be about `math.log(2) = 0.69314`"
]
},
{
Expand Down Expand Up @@ -630,7 +629,7 @@
"id": "d1juXI9yY1KD"
},
"source": [
"Set that as the initial bias, and the model will give much more reasonable initial guesses. \n",
"Set that as the initial bias, and the model will give much more reasonable initial guesses.\n",
"\n",
"It should be near: `pos/total = 0.0018`"
]
Expand Down Expand Up @@ -700,7 +699,7 @@
},
"outputs": [],
"source": [
"initial_weights = os.path.join(tempfile.mkdtemp(), 'initial_weights')\n",
"initial_weights = os.path.join(tempfile.mkdtemp(), 'initial.weights.h5')\n",
"model.save_weights(initial_weights)"
]
},
Expand All @@ -714,7 +713,7 @@
"\n",
"Before moving on, confirm quick that the careful bias initialization actually helped.\n",
"\n",
"Train the model for 20 epochs, with and without this careful initialization, and compare the losses: "
"Train the model for 20 epochs, with and without this careful initialization, and compare the losses:"
]
},
{
Expand All @@ -733,7 +732,7 @@
" train_labels,\n",
" batch_size=BATCH_SIZE,\n",
" epochs=20,\n",
" validation_data=(val_features, val_labels), \n",
" validation_data=(val_features, val_labels),\n",
" verbose=0)"
]
},
Expand All @@ -752,7 +751,7 @@
" train_labels,\n",
" batch_size=BATCH_SIZE,\n",
" epochs=20,\n",
" validation_data=(val_features, val_labels), \n",
" validation_data=(val_features, val_labels),\n",
" verbose=0)"
]
},
Expand Down Expand Up @@ -794,7 +793,7 @@
"id": "fKMioV0ddG3R"
},
"source": [
"The above figure makes it clear: In terms of validation loss, on this problem, this careful initialization gives a clear advantage. "
"The above figure makes it clear: In terms of validation loss, on this problem, this careful initialization gives a clear advantage."
]
},
{
Expand All @@ -821,7 +820,7 @@
" train_labels,\n",
" batch_size=BATCH_SIZE,\n",
" epochs=EPOCHS,\n",
" callbacks=[early_stopping],\n",
" callbacks=[early_stopping()],\n",
" validation_data=(val_features, val_labels))"
]
},
Expand Down Expand Up @@ -996,10 +995,9 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "P-QpQsip_F2Q"
"id": "kF8k-g9goRni"
},
"source": [
"### Plot the ROC\n",
Expand Down Expand Up @@ -1161,10 +1159,10 @@
" train_labels,\n",
" batch_size=BATCH_SIZE,\n",
" epochs=EPOCHS,\n",
" callbacks=[early_stopping],\n",
" callbacks=[early_stopping()],\n",
" validation_data=(val_features, val_labels),\n",
" # The class weights go here\n",
" class_weight=class_weight) "
" class_weight=class_weight)"
]
},
{
Expand Down Expand Up @@ -1333,7 +1331,7 @@
"source": [
"#### Using NumPy\n",
"\n",
"You can balance the dataset manually by choosing the right number of random \n",
"You can balance the dataset manually by choosing the right number of random\n",
"indices from the positive examples:"
]
},
Expand Down Expand Up @@ -1485,7 +1483,7 @@
},
"outputs": [],
"source": [
"resampled_steps_per_epoch = np.ceil(2.0*neg/BATCH_SIZE)\n",
"resampled_steps_per_epoch = int(np.ceil(2.0*neg/BATCH_SIZE))\n",
"resampled_steps_per_epoch"
]
},
Expand All @@ -1499,7 +1497,7 @@
"\n",
"Now try training the model with the resampled data set instead of using class weights to see how these methods compare.\n",
"\n",
"Note: Because the data was balanced by replicating the positive examples, the total dataset size is larger, and each epoch runs for more training steps. "
"Note: Because the data was balanced by replicating the positive examples, the total dataset size is larger, and each epoch runs for more training steps."
]
},
{
Expand All @@ -1514,17 +1512,17 @@
"resampled_model.load_weights(initial_weights)\n",
"\n",
"# Reset the bias to zero, since this dataset is balanced.\n",
"output_layer = resampled_model.layers[-1] \n",
"output_layer = resampled_model.layers[-1]\n",
"output_layer.bias.assign([0])\n",
"\n",
"val_ds = tf.data.Dataset.from_tensor_slices((val_features, val_labels)).cache()\n",
"val_ds = val_ds.batch(BATCH_SIZE).prefetch(2) \n",
"val_ds = val_ds.batch(BATCH_SIZE).prefetch(2)\n",
"\n",
"resampled_history = resampled_model.fit(\n",
" resampled_ds,\n",
" epochs=EPOCHS,\n",
" steps_per_epoch=resampled_steps_per_epoch,\n",
" callbacks=[early_stopping],\n",
" callbacks=[early_stopping()],\n",
" validation_data=val_ds)"
]
},
Expand All @@ -1536,7 +1534,7 @@
"source": [
"If the training process were considering the whole dataset on each gradient update, this oversampling would be basically identical to the class weighting.\n",
"\n",
"But when training the model batch-wise, as you did here, the oversampled data provides a smoother gradient signal: Instead of each positive example being shown in one batch with a large weight, they're shown in many different batches each time with a small weight. \n",
"But when training the model batch-wise, as you did here, the oversampled data provides a smoother gradient signal: Instead of each positive example being shown in one batch with a large weight, they're shown in many different batches each time with a small weight.\n",
"\n",
"This smoother gradient signal makes it easier to train the model."
]
Expand All @@ -1549,7 +1547,7 @@
"source": [
"### Check training history\n",
"\n",
"Note that the distributions of metrics will be different here, because the training data has a totally different distribution from the validation and test data. "
"Note that the distributions of metrics will be different here, because the training data has a totally different distribution from the validation and test data."
]
},
{
Expand Down Expand Up @@ -1578,7 +1576,7 @@
"id": "KFLxRL8eoDE5"
},
"source": [
"Because training is easier on the balanced data, the above training procedure may overfit quickly. \n",
"Because training is easier on the balanced data, the above training procedure may overfit quickly.\n",
"\n",
"So break up the epochs to give the `tf.keras.callbacks.EarlyStopping` finer control over when to stop training."
]
Expand All @@ -1595,15 +1593,15 @@
"resampled_model.load_weights(initial_weights)\n",
"\n",
"# Reset the bias to zero, since this dataset is balanced.\n",
"output_layer = resampled_model.layers[-1] \n",
"output_layer = resampled_model.layers[-1]\n",
"output_layer.bias.assign([0])\n",
"\n",
"resampled_history = resampled_model.fit(\n",
" resampled_ds,\n",
" # These are not real epochs\n",
" steps_per_epoch=20,\n",
" epochs=10*EPOCHS,\n",
" callbacks=[early_stopping],\n",
" callbacks=[early_stopping()],\n",
" validation_data=(val_ds))"
]
},
Expand Down Expand Up @@ -1696,7 +1694,7 @@
"id": "vayGnv0VOe_v"
},
"source": [
"### Plot the AUPRC\r\n"
"### Plot the AUPRC\n"
]
},
{
Expand All @@ -1707,14 +1705,14 @@
},
"outputs": [],
"source": [
"plot_prc(\"Train Baseline\", train_labels, train_predictions_baseline, color=colors[0])\r\n",
"plot_prc(\"Test Baseline\", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')\r\n",
"\r\n",
"plot_prc(\"Train Weighted\", train_labels, train_predictions_weighted, color=colors[1])\r\n",
"plot_prc(\"Test Weighted\", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')\r\n",
"\r\n",
"plot_prc(\"Train Resampled\", train_labels, train_predictions_resampled, color=colors[2])\r\n",
"plot_prc(\"Test Resampled\", test_labels, test_predictions_resampled, color=colors[2], linestyle='--')\r\n",
"plot_prc(\"Train Baseline\", train_labels, train_predictions_baseline, color=colors[0])\n",
"plot_prc(\"Test Baseline\", test_labels, test_predictions_baseline, color=colors[0], linestyle='--')\n",
"\n",
"plot_prc(\"Train Weighted\", train_labels, train_predictions_weighted, color=colors[1])\n",
"plot_prc(\"Test Weighted\", test_labels, test_predictions_weighted, color=colors[1], linestyle='--')\n",
"\n",
"plot_prc(\"Train Resampled\", train_labels, train_predictions_resampled, color=colors[2])\n",
"plot_prc(\"Test Resampled\", test_labels, test_predictions_resampled, color=colors[2], linestyle='--')\n",
"plt.legend(loc='lower right');"
]
},
Expand All @@ -1732,7 +1730,6 @@
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "imbalanced_data.ipynb",
"toc_visible": true
},
Expand Down