Math125_Project_AndyHe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6deb346b",
   "metadata": {},
   "source": [
    "# Language Identification with Perceptron and Naive Bayes models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a39494e",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4273e374",
   "metadata": {},
   "source": [
    "Welcome to this Jupyter Notebook, which is dedicated to the fascinating field of Natural Language Processing (NLP), specifically focusing on the task of Language Identification. This project is an exploration into the realm of machine learning, where we employ two distinct models - the Perceptron and Naive Bayes - to accurately identify the language of given text samples.\n",
    "\n",
    "Language Identification is a fundamental task in NLP that involves determining the language that a piece of text is written in. This is crucial in various applications such as content categorization, filtering, and as a preprocessing step in complex NLP tasks like translation and sentiment analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0914eade",
   "metadata": {},
   "source": [
    "The language identification dataset is from: https://www.kaggle.com/datasets/zarajamshaid/language-identification-datasst/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa4cecc8",
   "metadata": {},
   "source": [
    "From the original dataset, we split it into train-dev-test sets by the proportion of 70-15-15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7de93abe-8126-4ef4-9494-b71ec1a66be1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import random\n",
    "import math\n",
    "from pathlib import Path\n",
    "from typing import Iterable, Generator, Sequence\n",
    "\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from collections import defaultdict, Counter\n",
    "from operator import itemgetter\n",
    "from sklearn.metrics import accuracy_score, f1_score\n",
    "\n",
    "# Load the dataset\n",
    "dataset_path = Path('dataset') \n",
    "dataset = pd.read_csv(dataset_path / 'lid_dataset.csv')\n",
    "\n",
    "# Splitting the dataset into training (70%) and the remaining (30%)\n",
    "train_set, remaining_set = train_test_split(dataset, test_size=0.3, random_state=42, stratify=dataset['language'])\n",
    "\n",
    "# Split the remaining 30% into validation and test sets (50% each of the remaining, which is 15% each of the total)\n",
    "validation_set, test_set = train_test_split(remaining_set, test_size=0.5, random_state=42, stratify=remaining_set['language'])\n",
    "\n",
    "# Save the sets into separate CSV files\n",
    "train_set.to_csv(os.path.join(dataset_path, 'lid_train.csv'), index=False)\n",
    "validation_set.to_csv(os.path.join(dataset_path, 'lid_dev.csv'), index=False)\n",
    "test_set.to_csv(os.path.join(dataset_path, 'lid_test.csv'), index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c88e4f1",
   "metadata": {},
   "source": [
    "Define necessary helper classes we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c38e49eb-0f62-4c0f-be27-75a3ed11314b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ClassificationInstance:\n",
    "    def __init__(self, label: str, features: Iterable[str]) -> None:\n",
    "        self.label: str = label\n",
    "        self.features: tuple[str, ...] = tuple(features)\n",
    "\n",
    "    def __repr__(self) -> str:\n",
    "        return f\"<ClassificationInstance: {str(self)}>\"\n",
    "\n",
    "    def __str__(self) -> str:\n",
    "        return f\"label={self.label}; features={self.features}\"\n",
    "\n",
    "\n",
    "class LanguageIdentificationInstance:\n",
    "    def __init__(self, text: str, language: str) -> None:\n",
    "        self.language: str = language\n",
    "        self.text: str = text\n",
    "\n",
    "    def __repr__(self) -> str:\n",
    "        return f\"<LanguageIdentificationInstance: {str(self)}>\"\n",
    "\n",
    "    def __str__(self) -> str:\n",
    "        return f\"label={self.language}; text={self.text}\"\n",
    "\n",
    "    @classmethod\n",
    "    def from_line(cls, line: str) -> \"LanguageIdentificationInstance\":\n",
    "        splits = line.rstrip(\"\\n\").split(\",\")\n",
    "        return cls(splits[0], splits[1])\n",
    "\n",
    "\n",
    "def load_lid_instances(path: str) -> Generator[LanguageIdentificationInstance, None, None]:\n",
    "    with open(path, encoding=\"utf8\") as file:\n",
    "        next(file)\n",
    "        for line in file:\n",
    "            yield LanguageIdentificationInstance.from_line(line)\n",
    "\n",
    "\n",
    "def max_item(scores: dict[str, float]) -> tuple[str, float]:\n",
    "    return max(scores.items(), key=itemgetter(1))\n",
    "\n",
    "\n",
    "def items_descending_value(counts: Counter[str]) -> list[str]:\n",
    "    return [key for key, value in sorted(counts.items(), key=_items_sort_key)]\n",
    "\n",
    "\n",
    "def _items_sort_key(item: tuple[str, int]) -> tuple[int, str]:\n",
    "    return -item[1], item[0]\n",
    "\n",
    "\n",
    "class CharBigramFeatureExtractor:\n",
    "    @staticmethod\n",
    "    def extract_features(instance: LanguageIdentificationInstance) -> ClassificationInstance:\n",
    "        features = set()\n",
    "        text = instance.text\n",
    "        for i in range(len(text)-1):\n",
    "            features.add(text[i:i+2])\n",
    "        return ClassificationInstance(instance.language, features)\n",
    "    \n",
    "    \n",
    "class InstanceCounter:\n",
    "    \"\"\"Holds counts of the labels and features seen during training.\n",
    "\n",
    "    See the assignment for an explanation of each method.\"\"\"\n",
    "\n",
    "    def __init__(self) -> None:\n",
    "        self._label_counts = Counter()\n",
    "        self._total_labels = 0\n",
    "        self._feature_label_joint_counts = defaultdict(Counter)\n",
    "        self._label_list = []\n",
    "        self._feature_vocab_size = 0\n",
    "        self._feature_set = set()\n",
    "        self._total_feature_count_for_label = Counter()\n",
    "\n",
    "    def count_instances(self, instances: Iterable[ClassificationInstance]) -> None:\n",
    "        for instance in instances:\n",
    "            label = instance.label\n",
    "            features = instance.features\n",
    "\n",
    "            self._label_counts[label] += 1\n",
    "            self._total_labels += 1\n",
    "            self._feature_label_joint_counts[label].update(features)\n",
    "\n",
    "            self._feature_set.update(features)\n",
    "            self._total_feature_count_for_label[label] += len(features)\n",
    "\n",
    "        self._label_list = list(self._label_counts.keys())\n",
    "        self._feature_vocab_size = len(self._feature_set)\n",
    "\n",
    "    def label_count(self, label: str) -> int:\n",
    "        return self._label_counts[label]\n",
    "\n",
    "    def total_labels(self) -> int:\n",
    "        return self._total_labels\n",
    "\n",
    "    def feature_label_joint_count(self, feature: str, label: str) -> int:\n",
    "        return self._feature_label_joint_counts[label][feature]\n",
    "\n",
    "    def unique_labels(self) -> list[str]:\n",
    "        return self._label_list\n",
    "\n",
    "    def feature_vocab_size(self) -> int:\n",
    "        return self._feature_vocab_size\n",
    "\n",
    "    def feature_set(self) -> set[str]:\n",
    "        return self._feature_set\n",
    "\n",
    "    def total_feature_count_for_label(self, label: str) -> int:\n",
    "        return self._total_feature_count_for_label[label]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8c38129",
   "metadata": {},
   "source": [
    "Prepare out data to be used for both models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b5b860e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Paths\n",
    "TRAIN_PATH = dataset_path / 'lid_train.csv'\n",
    "DEV_PATH = dataset_path / 'lid_dev.csv'\n",
    "TEST_PATH = dataset_path / 'lid_test.csv'\n",
    "\n",
    "# Feature extraction\n",
    "feature_extractor = CharBigramFeatureExtractor()\n",
    "train_instances = [feature_extractor.extract_features(inst) for inst in load_lid_instances(TRAIN_PATH)]\n",
    "dev_instances = [feature_extractor.extract_features(inst) for inst in load_lid_instances(DEV_PATH)]\n",
    "test_instances = [feature_extractor.extract_features(inst) for inst in load_lid_instances(TEST_PATH)]\n",
    "\n",
    "# Count instances\n",
    "instance_counter = InstanceCounter()\n",
    "instance_counter.count_instances(train_instances)\n",
    "labels = instance_counter.unique_labels()\n",
    "\n",
    "# Labels\n",
    "dev_labels = [instance.label for instance in dev_instances]\n",
    "test_labels = [instance.label for instance in test_instances]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30ccc783",
   "metadata": {},
   "source": [
    "## Perceptron & Averaged Perceptron models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1850372e",
   "metadata": {},
   "source": [
    "Define our perceptron and averaged perceptron model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "cade02d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Perceptron:\n",
    "    def __init__(self, labels: list[str]) -> None:\n",
    "        self.labels: list[str] = labels\n",
    "        self.weights: dict[str, defaultdict[str, float]] = {}\n",
    "        for label in labels:\n",
    "            self.weights[label] = defaultdict(float)\n",
    "        self.step: int = 0\n",
    "\n",
    "    def classify(self, features: Iterable[str]) -> str:\n",
    "        scores = {}\n",
    "        for label in self.labels:\n",
    "            scores[label] = 0\n",
    "            for feature in features:\n",
    "                scores[label] += self.weights[label][feature]\n",
    "        return max_item(scores)[0]\n",
    "\n",
    "    def learn(self, instance: ClassificationInstance, lr: float) -> None:\n",
    "        self.step += 1\n",
    "        features = instance.features\n",
    "        label_pred = self.classify(features)\n",
    "        label_true = instance.label\n",
    "\n",
    "        if label_pred != label_true:\n",
    "            for feature in features:\n",
    "                self.pre_update(feature, label_true)\n",
    "                self.pre_update(feature, label_pred)\n",
    "                self.weights[label_true][feature] += lr\n",
    "                self.weights[label_pred][feature] -= lr\n",
    "\n",
    "    def pre_update(self, feature: str, label: str) -> None:\n",
    "        # Do nothing\n",
    "        pass\n",
    "\n",
    "    def finalize_weights(self) -> None:\n",
    "        # Do nothing\n",
    "        pass\n",
    "\n",
    "\n",
    "class AveragedPerceptron(Perceptron):\n",
    "    def __init__(self, labels: list[str]) -> None:\n",
    "        super().__init__(labels)\n",
    "        self.sums: dict[str, defaultdict[str, float]] = {}\n",
    "        self.last_updated: dict[str, defaultdict[str, int]] = {}\n",
    "        for label in labels:\n",
    "            self.sums[label] = defaultdict(float)\n",
    "            self.last_updated[label] = defaultdict(int)\n",
    "\n",
    "    def pre_update(self, feature: str, label: str) -> None:\n",
    "        self.sums[label][feature] += (self.step - self.last_updated[label][feature]) * self.weights[label][feature]\n",
    "        self.last_updated[label][feature] = self.step\n",
    "\n",
    "    def finalize_weights(self) -> None:\n",
    "        self.step += 1\n",
    "        for label in self.labels:\n",
    "            for feature, weight_sum in self.sums[label].items():\n",
    "                self.sums[label][feature] += (self.step-self.last_updated[label][feature])*self.weights[label][feature]\n",
    "                self.weights[label][feature] = self.sums[label][feature] / self.step\n",
    "\n",
    "\n",
    "def train_perceptron(\n",
    "    model: Perceptron, data: list[ClassificationInstance], epochs: int, lr: float\n",
    ") -> None:\n",
    "    for _ in range(epochs):\n",
    "        for instance in data:\n",
    "            model.learn(instance, lr)\n",
    "        random.shuffle(data)\n",
    "    model.finalize_weights()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae82603",
   "metadata": {},
   "source": [
    "Train the perceptron model with different hyperparameters and tune with the dev set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "83920fb2-a79b-476d-9553-37d8a509d6ee",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epochs=1, LR=0.01, Average=False, Accuracy: 0.9678787878787879\n",
      "Epochs=1, LR=0.1, Average=False, Accuracy: 0.9557575757575758\n",
      "Epochs=1, LR=1, Average=False, Accuracy: 0.9675757575757575\n",
      "Epochs=3, LR=0.01, Average=False, Accuracy: 0.9733333333333334\n",
      "Epochs=3, LR=0.1, Average=False, Accuracy: 0.9766666666666667\n",
      "Epochs=3, LR=1, Average=False, Accuracy: 0.9772727272727273\n",
      "Epochs=5, LR=0.01, Average=False, Accuracy: 0.9760606060606061\n",
      "Epochs=5, LR=0.1, Average=False, Accuracy: 0.98\n",
      "Epochs=5, LR=1, Average=False, Accuracy: 0.9775757575757575\n",
      "Epochs=1, LR=0.01, Average=True, Accuracy: 0.9796969696969697\n",
      "Epochs=1, LR=0.1, Average=True, Accuracy: 0.9827272727272728\n",
      "Epochs=1, LR=1, Average=True, Accuracy: 0.9796969696969697\n",
      "Epochs=3, LR=0.01, Average=True, Accuracy: 0.980909090909091\n",
      "Epochs=3, LR=0.1, Average=True, Accuracy: 0.9821212121212122\n",
      "Epochs=3, LR=1, Average=True, Accuracy: 0.9806060606060606\n",
      "Epochs=5, LR=0.01, Average=True, Accuracy: 0.983030303030303\n",
      "Epochs=5, LR=0.1, Average=True, Accuracy: 0.9821212121212122\n",
      "Epochs=5, LR=1, Average=True, Accuracy: 0.9821212121212122\n",
      "Best Hyperparameters for Perceptron:\n",
      "{'n_epochs': 5, 'lr': 0.01, 'average': True}\n",
      "Testing with Best Hyperparameters for Perceptron:\n",
      "Accuracy: 0.9836363636363636\n",
      "Macro F1: 0.9837642209592249\n"
     ]
    }
   ],
   "source": [
    "def perceptron_lid_model():\n",
    "    # Hyperparameters to tune\n",
    "    epoch_choices = [1, 3, 5]\n",
    "    lr_choices = [0.01, 0.1, 1]\n",
    "    average_choices = [False, True]\n",
    "\n",
    "    best_accuracy = 0.0\n",
    "    best_hyperparameters = {}\n",
    "    best_perceptron_model = None\n",
    "\n",
    "    for average in average_choices:\n",
    "        for n_epochs in epoch_choices:\n",
    "            for lr in lr_choices:\n",
    "                # Training\n",
    "                perceptron_model = AveragedPerceptron(labels) if average else Perceptron(labels)\n",
    "                train_perceptron(perceptron_model, train_instances, n_epochs, lr)\n",
    "\n",
    "                # Evaluation on development set\n",
    "                dev_predictions = [perceptron_model.classify(instance.features) for instance in dev_instances]\n",
    "                accuracy = accuracy_score(dev_labels, dev_predictions)\n",
    "\n",
    "                print(f\"Epochs={n_epochs}, LR={lr}, Average={average}, Accuracy: {accuracy}\")\n",
    "\n",
    "                if accuracy > best_accuracy:\n",
    "                    best_accuracy = accuracy\n",
    "                    best_hyperparameters = {\n",
    "                        'n_epochs': n_epochs,\n",
    "                        'lr': lr,\n",
    "                        'average': average,\n",
    "                    }\n",
    "                    best_perceptron_model = perceptron_model\n",
    "\n",
    "    print(\"Best Hyperparameters for Perceptron:\")\n",
    "    print(best_hyperparameters)\n",
    "\n",
    "    # Testing with the best hyperparameters on the test set\n",
    "    test_predictions = [best_perceptron_model.classify(instance.features) for instance in test_instances]\n",
    "    test_accuracy = accuracy_score(test_labels, test_predictions)\n",
    "    test_micro_f1 = f1_score(test_labels, test_predictions, average='micro')\n",
    "    test_macro_f1 = f1_score(test_labels, test_predictions, average='macro')\n",
    "\n",
    "    print(\"Testing with Best Hyperparameters for Perceptron:\")\n",
    "    print(f\"Accuracy: {test_accuracy}\")\n",
    "    print(f\"Macro F1: {test_macro_f1}\")\n",
    "\n",
    "perceptron_lid_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "043c1bd8",
   "metadata": {},
   "source": [
    "Pros:\n",
    "-   Simplicity and Speed: The Perceptron is a simple linear classifier that is generally fast to train, making it suitable for large datasets and real-time prediction.\n",
    "\n",
    "-   Online Learning: It can be easily adapted for online learning, where the model is updated continuously as new data arrives.\n",
    "Robust to Noisy Data: Perceptrons can be quite robust to noise in the input data.\n",
    "\n",
    "-   Scalability: Works well with high-dimensional data and scales nicely with the size of the dataset.\n",
    "\n",
    "Cons:\n",
    "-   Linear Decision Boundary: The Perceptron can only classify data that is linearly separable. It struggles with complex patterns and non-linear relationships.\n",
    "\n",
    "-   Prone to Overfitting: Without proper regularization or early stopping, it can overfit to the training data, especially in the case of noisy or unbalanced datasets.\n",
    "\n",
    "-   Lack of Probabilistic Interpretation: It does not output probabilities for classifications, making it less informative for certain applications where understanding the uncertainty is crucial.\n",
    "\n",
    "-   Convergence Issues: The algorithm may fail to converge if the data is not linearly separable."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e08ae38",
   "metadata": {},
   "source": [
    "Next, we define our Naive Bayes Classifier model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "ca41a8cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "class NaiveBayesClassifier:\n",
    "\n",
    "    def __init__(self, k: float):\n",
    "        self.k: float = k\n",
    "        self.instance_counter: InstanceCounter = InstanceCounter()\n",
    "\n",
    "    def train(self, instances: Iterable[ClassificationInstance]) -> None:\n",
    "        self.instance_counter.count_instances(instances)\n",
    "\n",
    "    def prior_prob(self, label: str) -> float:\n",
    "        return self.instance_counter.label_count(label)/self.instance_counter.total_labels()\n",
    "\n",
    "    def feature_prob(self, feature: str, label) -> float:\n",
    "        count_feature_given_label = self.instance_counter.feature_label_joint_count(feature, label)\n",
    "        big_n = self.instance_counter.total_feature_count_for_label(label)\n",
    "        big_v = self.instance_counter.feature_vocab_size()\n",
    "        return (count_feature_given_label + self.k) / (big_n + big_v * self.k)\n",
    "\n",
    "    def log_posterior_prob(self, features: Sequence[str], label: str) -> float:\n",
    "        log_class_prior_probability = math.log(self.prior_prob(label))\n",
    "        log_posterior_probability = log_class_prior_probability\n",
    "        for feature in features:\n",
    "            if feature in self.instance_counter.feature_set():\n",
    "                likelihood = self.feature_prob(feature, label)\n",
    "                log_posterior_probability += math.log(likelihood)\n",
    "        return log_posterior_probability\n",
    "\n",
    "    def classify(self, features: Sequence[str]) -> str:\n",
    "        probs = []\n",
    "        for label in self.instance_counter.unique_labels():\n",
    "            probs.append((self.log_posterior_prob(features, label), label))\n",
    "        return max(probs)[1]\n",
    "\n",
    "    def test(\n",
    "        self, instances: Iterable[ClassificationInstance]\n",
    "    ) -> tuple[list[str], list[str]]:\n",
    "        predicted_labels = []\n",
    "        true_labels = []\n",
    "        for instance in instances:\n",
    "            true_labels.append(instance.label)\n",
    "            prediction = self.classify(instance.features)\n",
    "            predicted_labels.append(prediction)\n",
    "        return predicted_labels, true_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7d0177d",
   "metadata": {},
   "source": [
    "Train the naive bayes model with different k values and tune with the dev set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "d00ed257-4755-4ea5-95bc-b3a2981248f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "k=0.0001, Accuracy: 0.9763636363636363\n",
      "k=0.001, Accuracy: 0.9769696969696969\n",
      "k=0.01, Accuracy: 0.9766666666666667\n",
      "k=0.1, Accuracy: 0.9763636363636363\n",
      "Best Hyperparameter for k: 0.001\n",
      "Testing with Best Hyperparameters:\n",
      "Accuracy: 0.9781818181818182\n",
      "Macro F1: 0.9785765401597825\n"
     ]
    }
   ],
   "source": [
    "def naive_bayes_lid_model():\n",
    "    \n",
    "    # Hyperparameters to tune\n",
    "    k_choices = [0.0001, 0.001, 0.01, 0.1]\n",
    "\n",
    "    best_accuracy = 0.0\n",
    "    best_k = None\n",
    "    best_nb_model = None\n",
    "\n",
    "    for k in k_choices:\n",
    "        # Training Naive Bayes\n",
    "        nb_model = NaiveBayesClassifier(k)\n",
    "        nb_model.train(train_instances)\n",
    "\n",
    "        # Evaluation on development set\n",
    "        dev_predictions, _ = nb_model.test(dev_instances)\n",
    "        accuracy = accuracy_score(dev_labels, dev_predictions)\n",
    "\n",
    "        print(f\"k={k}, Accuracy: {accuracy}\")\n",
    "\n",
    "        if accuracy > best_accuracy:\n",
    "            best_accuracy = accuracy\n",
    "            best_k = k\n",
    "            best_nb_model = nb_model\n",
    "\n",
    "    print(\"Best Hyperparameter for k:\", best_k)\n",
    "\n",
    "    # Testing with the best hyperparameters on the test set\n",
    "    test_predictions, _ = best_nb_model.test(test_instances)\n",
    "    test_accuracy = accuracy_score(test_labels, test_predictions)\n",
    "    test_macro_f1 = f1_score(test_labels, test_predictions, average='macro')\n",
    "\n",
    "    print(\"Testing with Best Hyperparameters:\")\n",
    "    print(f\"Accuracy: {test_accuracy}\")\n",
    "    print(f\"Macro F1: {test_macro_f1}\")\n",
    "\n",
    "naive_bayes_lid_model()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0153507b",
   "metadata": {},
   "source": [
    "Pros:\n",
    "\n",
    "-   Probabilistic Approach: It provides a probabilistic understanding of classifications, which can be insightful for many applications.\n",
    "\n",
    "-   Good with Small Data: Often performs well even with a relatively small amount of training data, thanks to its probabilistic nature.\n",
    "\n",
    "-   Handling Categorical Data: Naturally handles categorical data and can be adapted for text classification problems.\n",
    "\n",
    "-   Efficiency: Naive Bayes classifiers are computationally efficient, especially in their training phase.\n",
    "\n",
    "Cons:\n",
    "\n",
    "-   Naive Assumption: The assumption of feature independence is often unrealistic in real-world scenarios, leading to potentially suboptimal performance.\n",
    "\n",
    "-   Poor Estimation for Rare Features: It can struggle with features that did not appear in the training set (zero-frequency problem) unless specifically addressed (e.g., with smoothing techniques).\n",
    "\n",
    "-   Sensitive to Imbalanced Data: The performance can be significantly impacted by imbalanced datasets.\n",
    "\n",
    "-   Less Effective for Correlated Features: Due to the independence assumption, it may not perform well when features are correlated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c51551",
   "metadata": {},
   "source": [
    "Summary\n",
    "\n",
    "Perceptron is suitable for problems where a linear decision boundary is sufficient, and the focus is on speed and scalability over interpretability. It's more suitable for online learning scenarios.\n",
    "\n",
    "Naive Bayes excels in probabilistic classification, especially in text-related tasks and scenarios where training data is limited. Its performance can suffer when the assumption of independent features does not hold.\n",
    "\n",
    "In practice, the choice between these models would depend on the specific characteristics of the dataset and the requirements of the task at hand. Often, trying both and comparing their performance on a validation set is the best way to determine which is more suitable for a particular problem."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}