2_Self_Case_Study_2_Data_preprocessing_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re\n",
    "import nltk\n",
    "import spacy\n",
    "import string\n",
    "pd.options.mode.chained_assignment = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "Data = pd.read_csv(r\"C:\\Users\\Lenovo\\Downloads\\Self case study 2\\Data\\jigsaw-unintended-bias-in-toxicity-classification\\train.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comment_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>This is so cool. It's like, 'would you want yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Thank you!! This would make my life a lot less...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>This is such an urgent design problem; kudos t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Is this something I'll be able to install on m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>haha you guys are a bunch of losers.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        comment_text\n",
       "0  This is so cool. It's like, 'would you want yo...\n",
       "1  Thank you!! This would make my life a lot less...\n",
       "2  This is such an urgent design problem; kudos t...\n",
       "3  Is this something I'll be able to install on m...\n",
       "4               haha you guys are a bunch of losers."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Data[['comment_text']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  A. Lets Understand the data first:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <i> 1. randomenly sampling 5 rows of comment_text.<i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 :\n",
      " I grew up in Kaltag, Kwigillingok and Klukwan, and also lived on a homestead on the Glenn Highway for a few years as a young adult. NEVER saw so many moose as I have right in the middle of Anchorage. It's ridiculous. \n",
      "\n",
      "**************************************************************************************************** \n",
      "\n",
      "1 :\n",
      " https://climate.nasa.gov/climate_resources/24/ \n",
      "\n",
      "**************************************************************************************************** \n",
      "\n",
      "2 :\n",
      " Oh, that explains everything, you clearly are much more knowledgeable than me.\n",
      "\n",
      "While the Hyde Amendment actually does not generally allow for the payment for abortions with federal money there are exceptions (for in the cases of life threatening situations, rape and incest) -- contrary to your blanket assertion.  Further, many states use local money to provide funding.  However, my point is the law needs to be changed, as was the call in the Democratic Platform for 2016. \n",
      "\n",
      "**************************************************************************************************** \n",
      "\n",
      "3 :\n",
      " What's your point? \n",
      "\n",
      "**************************************************************************************************** \n",
      "\n",
      "4 :\n",
      " The Liberals want to live high on the hog and expect everyone else to cut their living standards. Blatant hypocrisy at it's finest. I hope Kenney comes out with guns blazing and wipes that smug entitled grin off Trudeau's face. \n",
      "\n",
      "**************************************************************************************************** \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i,j in enumerate(Data['comment_text'].sample(n=5).values):\n",
    "    print(i,':\\n',j,'\\n')\n",
    "    print('*'*100,'\\n')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code> **Observations**: \n",
    "<code>1. contain both lower and upper latter\n",
    "<code>2. contain numbers</code>\n",
    "<code>3. contain extra space</code>\n",
    "<code>4. contain http/https links</code>\n",
    "<code>5. contain Punctuations</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <i> 2. Wheather data contain Non english words (like emojis). <i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda\n",
    "def build_vocab(sentences, verbose =  True):\n",
    "    \"\"\"\n",
    "    :param sentences: list of list of words\n",
    "    :return: dictionary of words and their count\n",
    "    \"\"\"\n",
    "    vocab = {}\n",
    "    for sentence in tqdm(sentences, disable = (not verbose)):\n",
    "        for word in sentence:\n",
    "            try:\n",
    "                vocab[word] += 1\n",
    "            except KeyError:\n",
    "                vocab[word] = 1\n",
    "    return vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "latin_similar = \"’'‘ÆÐƎƏƐƔĲŊŒẞÞǷȜæðǝəɛɣĳŋœĸſßþƿȝĄƁÇĐƊĘĦĮƘŁØƠŞȘŢȚŦŲƯY̨Ƴąɓçđɗęħįƙłøơşșţțŧųưy̨ƴÁÀÂÄǍĂĀÃÅǺĄÆǼǢƁĆĊĈČÇĎḌĐƊÐÉÈĖÊËĚĔĒĘẸƎƏƐĠĜǦĞĢƔáàâäǎăāãåǻąæǽǣɓćċĉčçďḍđɗðéèėêëěĕēęẹǝəɛġĝǧğģɣĤḤĦIÍÌİÎÏǏĬĪĨĮỊĲĴĶƘĹĻŁĽĿʼNŃN̈ŇÑŅŊÓÒÔÖǑŎŌÕŐỌØǾƠŒĥḥħıíìiîïǐĭīĩįịĳĵķƙĸĺļłľŀŉńn̈ňñņŋóòôöǒŏōõőọøǿơœŔŘŖŚŜŠŞȘṢẞŤŢṬŦÞÚÙÛÜǓŬŪŨŰŮŲỤƯẂẀŴẄǷÝỲŶŸȲỸƳŹŻŽẒŕřŗſśŝšşșṣßťţṭŧþúùûüǔŭūũűůųụưẃẁŵẅƿýỳŷÿȳỹƴźżžẓ\"\n",
    "# white_list = string.ascii_letters + string.digits + latin_similar + ' '\n",
    "white_list = string.ascii_letters + latin_similar + ' '\n",
    "white_list += \"'\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [01:43<00:00, 17423.40it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'.,?!-;*\"…:\\n150—()%263#$97&_8/@＼・ω4+🍕=”“[]^–>\\r🐵\\\\°<😑~\\xa0\\ue014•≠\\t™\\uf818\\uf04a\\xadˈʊɒ😢🐶∞§{}·τα❤️☺ɡ\\uf0e0😜😎👊\\u200b\\u200e😁|عدويهصقأناخلىبمغر😍💖¢→̶`💵❥━┣┫Е┗Ｏ►★👎😀😂\\u202a\\u202c🔥😄©―🏻💥ᴍʏʀɪᴇɴᴅᴏᴀᴋʜᴜʟᴛᴄᴘʙғᴊᴡɢ✔®\\x96\\x92●😋👏שלוםבי😱‼£\\x81♥エンジ故障➤´\\u2009🚌ᴵ͞🌟😊😳😧🙀😐😕\\u200f👍😮😃😘¹☕≈÷אעכח♡◐║▬💩′ɔː💯⛽€🚄🏼ஜ۩۞†😖ᴠ🚲‐μ✒➥😟😈═☆ˌ💪🙏🎯◄🌹😇💔½ʻ😡\\x7f👌ἐπὶδηλήσειὲκἀίῃἴρξνʃ🙄✬ＳＵＰＥＲＨＩＴ😠\\ufeff☻±\\u2028😉😤⛺♍🙂µ\\u3000تحكسة👮💙فزط😏º🍾🎉¾😞\\u2008🏾😅😭👻😥😔😓🏽🎆✓◾🍻🍽🎶🌺🤔😪\\x08‑؟🐰🐇🐱🙆．😨⬅🙃💕𝘊𝘦𝘳𝘢𝘵𝘰𝘤𝘺𝘴𝘪𝘧𝘮𝘣💗💚地獄谷℅»ВулканПвоАН🐾🐕❣😆ה⋅🔗¿¬🚽歌舞伎🙈😴🏿🤗🇺🇸♫мυтѕＣＭ⤵🏆🎃β😩█▓▒░\\u200a🌠🐟💫💰💎⇒эпрд\\x95🖐🙅⛲🍰⭐🤐👆›🙌\\u2002💛🙁👀🙊🙉¡₂₃\\u2004❧▰ˢᵒʳʸ▔ᴼᴷᴺʷᵗʰᵉᵘ◞▀\\x13🚬▂▃▄▅▆▇↙🤓\\ue602😵άοόςέγὸ̄תמדףנרךצט😒͝″☹➡«🆕👅👥👄🔄🔤👉👤👶👲🔛🎓φ\\uf0b7⅓„✋：\\uf04c\\x9f\\x10成都¥😣⏺̲̅😌🤑́🌏😯ех😲∙‛Ἰᾶὁ💞🚓◇🔔📚✏🏀👐\\u202d💤🍇\\ue613小土豆🏡▷❔❓⁉❗\\u202f👠¶》कर्मा🇹🇼🌸蔡英文🌞˚🎲レクサス😛˙外国人关系）Ссиб💋💀🎄💜🤢َِʿьыгя✨不是。ɑ\\x80\\x9c\\x9d🗑\\u2005💃📣👿༼つ◕༽😰ḷЗз▱ц￼🤣卖！温哥华议会下降％你失去所有的钱加拿大坏税骗子🐝¯ツ🎅\\x85🍺آإشء−ﬂﬁ🎵🌎͟ἔ油别克🤡🤥😬🤧й\\u2003₁²🚀🤴ʌʲш¼⁴⁄₄⌠чИОРФДЯМю♭ж✘😝🖑ὐύύ特殊作戦群╪щ💨圆明园ק▶ℐ☭✭🏈😺♪🌍⏏ệ🍔🐮🍁☔🍆🍑🌮🌯☠🤦\\u200d♂𝓒𝓲𝓿𝓵안영하세요ЖљКћ🍀😫🤤ῦ我出生在了可以说普通话汉语好极🎼🕺☃🍸🥂🗽🎇🎊🆘☎🤠👩✈🖒✌✰❆☙🚪天一家⚲\\u2006⚭⚆⬭⬯⏖○‣⚓新年∎ℒ▪▙☏⅛✀╌🇫🇷🇩🇪🇮🇬🇧😷🇨🇦ХШ🌐\\x1f杀鸡给猴看ʁ𝗪𝗵𝗲𝗻𝘆𝗼𝘂𝗿𝗮𝗹𝗶𝘇𝗯𝘁𝗰𝘀𝘅𝗽𝘄𝗱📺ｃϖ\\u2000үսａᴦᎥһͺ\\u2007հｓǀ\\u2001ɩ℮ｙｅ൦ｌƽ¸ｗｈ𝐓𝐡𝐞𝐫𝐮𝐝𝐚𝐃𝐜𝐩𝐭𝐢𝐨𝐧Ƅᴨ‚ןᑯ໐ΤᏧ௦Іᴑ܁𝐬𝐰𝐲𝐛𝐦𝐯𝐑𝐙𝐣𝐇𝐂𝐘𝟎ԜТᗞ౦〔Ꭻ𝐳𝐔𝐱𝟔𝟓𝐅🐋∼ﬃ💘💓ё𝘥𝘯𝘶‖💐🌋🌄🌅𝙬𝙖𝙨𝙤𝙣𝙡𝙮𝙘𝙠𝙚𝙙𝙜𝙧𝙥𝙩𝙪𝙗𝙞𝙝𝙛👺🐷ℋℳ𝐀𝐥𝐪❄🚶←𝙢Ἱ🤘ͦ💸☼ج패티Ｗ⋆𝙇ʒᵻ👂👃ɜ🎫\\uf0a7БУі🚢⊂🚂ગુજરાતીῆ🏃、⅔𝓬𝓻𝓴𝓮𝓽𝓼☘¨﴾͡๏̯﴿⚾⚽Φ₽\\ue807𝑻𝒆𝒍𝒕𝒉𝒓𝒖𝒂𝒏𝒅𝒔𝒎𝒗𝒊👽😙\\u200cЛ×‒🎾👹θ￦⎌🏒⛸公寓养宠物吗？🏄🐀🚑🤷（操美𝒑𝒚𝒐𝑴🤙🐒℃欢迎来到阿拉斯ספ𝙫⏩☮🐈𝒌𝙊𝙭𝙆𝙋𝙍𝘼𝙅ﷻ⚠🦄巨收赢得月白鬼愤怒要买额ẽ🚗✊🐳𝟏𝐟𝟖𝟑𝟕𝒄𝟗𝐠𝙄𝙃👇锟斤拷❌⭕▸𝗢𝟳𝟱𝟬⦁マルハニチロ株式社⛷한국어ㄸㅓ니͜ʖ■⇌𝘿𝙔₵𝒩ℯ𝒾𝓁𝒶𝓉𝓇𝓊𝓃𝓈𝓅ℴ𝒻𝒽𝓀𝓌𝒸𝓎𝙏ζ𝙟𝘃𝗺𝟮𝟭𝟯𝟲👋🦊☐☑多伦⚡☄ǫ🐽🎻🎹⛓🏹╭∩╮🍷🦆为和中友谊祝贺与其想象对法如直接问，用自己猜本传教士没积唯认识基督徒曾经让相信耶稣复活死怪他但当们聊些政治题时候例战胜因圣把全堂结婚孩恐惧且栗谓这样还♾🎸🤕🤒⛑🎁批判检讨🏝🦁＞ʕɐ̣Δ₀🙋😶쥐스탱트뤼도석유가격인상이경제황을렵게만들지않록잘관리해야합다캐나에서대마초와화약금의품런성분갈때는반드시허된사용✞🔫👁┈╱╲▏▕┃╰▊▋╯┳┊≥☒凸ὰ↑💲🗯𝙈Ἄ𝒇𝒈𝒘𝒃𝑬𝑶𝕾𝖙𝖗𝖆𝖎𝖌𝖍𝖕𝖊𝖔𝖑𝖉𝖓𝖐𝖜𝖞𝖚𝖇𝕿𝖘𝖄𝖛𝖒𝖋𝖂𝕴𝖟𝖈𝕸👑🚿☝💡知彼百\\uf005𝙀𝒛𝑲𝑳𝑾𝒋𝟒😦𝙒𝘾𝘽🏐ɹ𝘩𝘨ὼṑ✅☛𝑱𝑹𝑫𝑵𝑪🇰🇵👾ᓇᒧᔭᐃᐧᐦᑳᐨᓃᓂᑲᐸᑭᑎᓀᐣ🐄🎈🔨♩🐎🤞☞🐸💟🎰🌝🛳点击查版🍭𝑥𝑦𝑧ＡＮＧＪＢ👣\\uf020っ◔◡🏉↓ф💭🎥♀Ξ🐴👨🤳⬆🦍\\x0b🍩𝑯𝒒😗𝟐🏂👳🍗🕉🐲چی̱ℏ𝑮𝗕𝗴\\x91🍒⠀ꜥˤⲣⲏ╚🐑⏰↺⇤∏鉄リ事件✾◦♬³ї💊「」\\uf203\\uf09a\\uf222\\ue608\\uf202\\uf099\\uf469\\ue607\\uf410\\ue600燻製シの虚偽屁理屈｜／Г𝑩𝑰𝒀𝑺🌤∵∴√𝗳𝗜𝗙𝗦𝗧🍊ὺἈἡχῖΛΩ¤⤏🇳𝒙ψՁմեռայինրւդձ冬至ὀ𝒁🔹🤚🍎𝑷🐂💅𝘬𝘱𝘸𝘷𝘐𝘭𝘓𝘖𝘹𝘲𝘫ک☜Βώ💢▲ΜΟΝΑΕ🇱♲𝝈↴↳💒⊘▫Ȼ‿⬇🚴🖕🖤🥘📍👈➕🚫🎨🌑🐻𝐎𝐍𝐊𝑭🤖🎎✧😼🕷ｇｏｖｒｎｍｔｉｄｕ－２０８ｆｂ＇ｋ𝟰🇴🇭🇻🇲𝗞𝗭𝗘𝗤‰≤👼📉🍟🍦∕🌈🔭《🐊🐍\\uf10aˆ⚜☁ლڡ🐦\\U0001f92f\\U0001f92a🐡💳ἱ🙇𝗸𝗟𝗠𝗷🥜さようなら🔼'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "# https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda\n",
    "jigsaw_chars = build_vocab(list(Data[\"comment_text\"]))\n",
    "jigsaw_symbols = ''.join([c for c in jigsaw_chars if not c in white_list])\n",
    "jigsaw_symbols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code> **Observations**: \n",
    "<code>1. contain emojis\n",
    "<code>2. contain other languages like Chinese </code>\n",
    "<code>3. contain english in different styles</code>\n",
    "<code>4. and some of hard to explain</code>\n",
    "<code>5. contain Punctuations</code>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <i> 3. wheather text data contain meaningless word ,self made shortforms or Not<i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [01:03<00:00, 28304.42it/s]\n"
     ]
    }
   ],
   "source": [
    "length_3 = set()\n",
    "length_3_list = []\n",
    "for i in tqdm(Data['comment_text']):\n",
    "    temp = i.split()\n",
    "\n",
    "    for j in temp:\n",
    "        if len(j)<=3:\n",
    "            length_3.add(j)\n",
    "            length_3_list.append(j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30812\n",
      "36386346\n"
     ]
    }
   ],
   "source": [
    "print(len(length_3))\n",
    "print(len(length_3_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'M-W',\n",
       " 'RWM',\n",
       " 'PFD',\n",
       " 'Tf',\n",
       " '13!',\n",
       " 'WCJ',\n",
       " 'UHF',\n",
       " 'C$.',\n",
       " '2S',\n",
       " 'Lo.',\n",
       " '40%',\n",
       " 'oi',\n",
       " 'Wao',\n",
       " '🔹',\n",
       " 'Mtr',\n",
       " 'ｗｅ',\n",
       " '-15',\n",
       " '\"RG',\n",
       " 'MIA',\n",
       " 'OSC',\n",
       " 'hae',\n",
       " '2-',\n",
       " '#48',\n",
       " 'L4:',\n",
       " 'Ok',\n",
       " '1/3',\n",
       " '<<I',\n",
       " 'p(*',\n",
       " 'K$',\n",
       " 'gho',\n",
       " '‘A',\n",
       " 'G.:',\n",
       " '1)I',\n",
       " '7th',\n",
       " 'COL',\n",
       " 'kag',\n",
       " 'bic',\n",
       " '[&',\n",
       " 'Z.,',\n",
       " \"'50\",\n",
       " 'NTS',\n",
       " 'KWS',\n",
       " 'DG',\n",
       " \"2'd\",\n",
       " 'HNL',\n",
       " 'BP',\n",
       " 'Cyl',\n",
       " 'pg',\n",
       " 'JVs',\n",
       " 'sat',\n",
       " 'OB,',\n",
       " 'sus',\n",
       " '90K',\n",
       " '3].',\n",
       " 'sjo',\n",
       " 'Ke',\n",
       " 'WUS',\n",
       " 'ie:',\n",
       " 'WAY',\n",
       " 'Hus',\n",
       " '~76',\n",
       " 'CAS',\n",
       " 'pad',\n",
       " 'ho,',\n",
       " 'ASD',\n",
       " 'Gpa',\n",
       " '\"a\\'',\n",
       " '𝖋𝖔𝖗',\n",
       " 'ß2',\n",
       " '6-',\n",
       " '#93',\n",
       " 'ADI',\n",
       " '𝒘𝒂𝒔',\n",
       " ':)\"',\n",
       " 'gs',\n",
       " ';(>',\n",
       " '$s?',\n",
       " 'Aiu',\n",
       " \"'2\",\n",
       " 'Gt.',\n",
       " 'WGN',\n",
       " '6lb',\n",
       " 'So?',\n",
       " 'ابن',\n",
       " 'ICU',\n",
       " '31)',\n",
       " '$33',\n",
       " 'NWR',\n",
       " '\"64',\n",
       " 'Imm',\n",
       " '🍊',\n",
       " 'Ft:',\n",
       " '86^',\n",
       " 'ppm',\n",
       " 'op;',\n",
       " '.69',\n",
       " 'ae',\n",
       " '😭😪',\n",
       " '*PS',\n",
       " '\\ufeff',\n",
       " 'Cyr',\n",
       " '🕷.',\n",
       " '1/8',\n",
       " '-9,',\n",
       " '.01',\n",
       " '12?',\n",
       " 'JP:',\n",
       " '27M',\n",
       " 'voi',\n",
       " 'G0',\n",
       " '8\"?',\n",
       " 'Dot',\n",
       " 'BSB',\n",
       " 'BOQ',\n",
       " 'sh',\n",
       " 'u',\n",
       " \"a)'\",\n",
       " '-We',\n",
       " '6%;',\n",
       " 'pm\"',\n",
       " 'Wu.',\n",
       " '\"kū',\n",
       " '▇',\n",
       " 'A',\n",
       " '(29',\n",
       " 'CMC',\n",
       " 'SOS',\n",
       " 'DVM',\n",
       " 'TI',\n",
       " 'AWY',\n",
       " 'M.!',\n",
       " 'VV',\n",
       " 'E,\"',\n",
       " 'ag.',\n",
       " 'ｕｓ！',\n",
       " 'gnu',\n",
       " '15-',\n",
       " 'puh',\n",
       " 'TF',\n",
       " '!1',\n",
       " '3~4',\n",
       " 'x2,',\n",
       " 'VEY',\n",
       " '4++',\n",
       " 'LOC',\n",
       " 'k/',\n",
       " 'com',\n",
       " 'SPD',\n",
       " 'ZM,',\n",
       " \"'60\",\n",
       " 'TST',\n",
       " \"i'v\",\n",
       " '0/',\n",
       " 'OPP',\n",
       " 'JP!',\n",
       " 'J1s',\n",
       " 'Ken',\n",
       " 'BV:',\n",
       " 'NSO',\n",
       " 'Śix',\n",
       " '☻',\n",
       " 'hi.',\n",
       " '3i',\n",
       " '3x4',\n",
       " 'OPE',\n",
       " 'noo',\n",
       " 'a',\n",
       " 'HDs',\n",
       " 'ONo',\n",
       " '11%',\n",
       " 'OYR',\n",
       " 'L-',\n",
       " '{*}',\n",
       " 'gpa',\n",
       " 'cd,',\n",
       " 'SAs',\n",
       " 'lei',\n",
       " 'ILK',\n",
       " 'edu',\n",
       " 'RWR',\n",
       " 'ou!',\n",
       " 'q:',\n",
       " 'was',\n",
       " '599',\n",
       " 'OVH',\n",
       " 'L&T',\n",
       " \"OK'\",\n",
       " 'TV3',\n",
       " '330',\n",
       " 'pue',\n",
       " 'CEP',\n",
       " '1}I',\n",
       " 'Sod',\n",
       " '.36',\n",
       " ':~)',\n",
       " 'P17',\n",
       " 'D.B',\n",
       " 'Op:',\n",
       " 'net',\n",
       " 'CoC',\n",
       " '52K',\n",
       " 'LSD',\n",
       " 'SR,',\n",
       " 'kma',\n",
       " 'thx',\n",
       " 'Cgc',\n",
       " ':0',\n",
       " 'w/s',\n",
       " 'Hp',\n",
       " '598',\n",
       " '5T',\n",
       " '+\"',\n",
       " 'Ya\"',\n",
       " 'x7)',\n",
       " 'JWA',\n",
       " '.HE',\n",
       " 'RBS',\n",
       " 'otj',\n",
       " 'NR.',\n",
       " 'BZ:',\n",
       " '1~2',\n",
       " 'bi',\n",
       " 'jA',\n",
       " 'WP!',\n",
       " 'He/',\n",
       " 'LNL',\n",
       " '(S_',\n",
       " '잘',\n",
       " 'wii',\n",
       " 'gr9',\n",
       " 'SWM',\n",
       " '48%',\n",
       " 'JGD',\n",
       " 'JPQ',\n",
       " '25?',\n",
       " 'yuo',\n",
       " '==I',\n",
       " 'E-1',\n",
       " 'G-4',\n",
       " '302',\n",
       " 'TC)',\n",
       " '229',\n",
       " '5/6',\n",
       " 'QR',\n",
       " 'ESY',\n",
       " 'DU.',\n",
       " 'STA',\n",
       " '\"09',\n",
       " 'д',\n",
       " '!:',\n",
       " 'mo!',\n",
       " 'M-2',\n",
       " '49!',\n",
       " 'oO',\n",
       " 'JlK',\n",
       " '-PR',\n",
       " 'Cnn',\n",
       " 'HCF',\n",
       " '9K+',\n",
       " 'P.M',\n",
       " 'UW',\n",
       " 'ASQ',\n",
       " 'P12',\n",
       " '😉,',\n",
       " 'OH',\n",
       " '3%',\n",
       " '1LT',\n",
       " 'Son',\n",
       " \"O'R\",\n",
       " 'ail',\n",
       " '’t.',\n",
       " '(79',\n",
       " 'U20',\n",
       " 'W-G',\n",
       " 'GA',\n",
       " 'STC',\n",
       " '3M?',\n",
       " 'Jr?',\n",
       " '\"$5',\n",
       " 'vag',\n",
       " 'PSD',\n",
       " 'Dik',\n",
       " '(Gn',\n",
       " 'EMU',\n",
       " 'Ten',\n",
       " 'BWW',\n",
       " '494',\n",
       " '<SO',\n",
       " 'PG,',\n",
       " 'it—',\n",
       " 'APP',\n",
       " '<G>',\n",
       " 'ye.',\n",
       " 'XRE',\n",
       " '8TH',\n",
       " 'VEX',\n",
       " 'GHS',\n",
       " \"40'\",\n",
       " 'wh,',\n",
       " 'ypc',\n",
       " 'k2',\n",
       " \"ya'\",\n",
       " 'LEX',\n",
       " 'hi?',\n",
       " 'Doc',\n",
       " 'OTZ',\n",
       " '@ss',\n",
       " 'R/V',\n",
       " 'Kow',\n",
       " '1-I',\n",
       " '#45',\n",
       " 'mc',\n",
       " 'PBK',\n",
       " 'B.,',\n",
       " 'ESP',\n",
       " 'rvr',\n",
       " 'DO:',\n",
       " 'UNC',\n",
       " '2K+',\n",
       " 'dmz',\n",
       " 'U&C',\n",
       " 'B/',\n",
       " 'Adm',\n",
       " '-tb',\n",
       " 'w/d',\n",
       " 'VCR',\n",
       " '616',\n",
       " 'ri',\n",
       " 'S-5',\n",
       " 'p.4',\n",
       " 'Op?',\n",
       " '\\u202aWe',\n",
       " '997',\n",
       " 'Ha$',\n",
       " 'ATM',\n",
       " '962',\n",
       " 'HCC',\n",
       " '…he',\n",
       " '\"ab',\n",
       " 'AYC',\n",
       " ':(.',\n",
       " '\"mr',\n",
       " 'tif',\n",
       " '(VP',\n",
       " '(25',\n",
       " 'Bt',\n",
       " 'NZ.',\n",
       " 'NCE',\n",
       " '18T',\n",
       " '…….',\n",
       " '19C',\n",
       " 'Sai',\n",
       " ';.)',\n",
       " 'NRW',\n",
       " 'ini',\n",
       " '21m',\n",
       " 'Cus',\n",
       " '𝗱𝗼',\n",
       " '9T',\n",
       " 'CDS',\n",
       " 'jok',\n",
       " '75F',\n",
       " 'X4',\n",
       " 'MG',\n",
       " '70m',\n",
       " 'GPN',\n",
       " 'CWD',\n",
       " 'EK:',\n",
       " '#12',\n",
       " '◞',\n",
       " '☺️💕',\n",
       " 'RUF',\n",
       " 'ys',\n",
       " 'x2-',\n",
       " '&Co',\n",
       " 'dr,',\n",
       " '\"if',\n",
       " 'BR,',\n",
       " '0',\n",
       " 'GTI',\n",
       " 'ya/',\n",
       " 'VC',\n",
       " '.3)',\n",
       " 'CNG',\n",
       " 's*d',\n",
       " '//a',\n",
       " '6bn',\n",
       " 'BBA',\n",
       " 'E.C',\n",
       " '5.5',\n",
       " 'iot',\n",
       " '$6/',\n",
       " 'dir',\n",
       " 'ROL',\n",
       " 'bo.',\n",
       " 'P4C',\n",
       " 'q#1',\n",
       " \"',\",\n",
       " 'KK.',\n",
       " 'DT?',\n",
       " '868',\n",
       " '…)',\n",
       " 'OM',\n",
       " 'wre',\n",
       " 'Max',\n",
       " 'TIN',\n",
       " 'G’s',\n",
       " 'ns.',\n",
       " 'TVs',\n",
       " 'Q1;',\n",
       " 'wt',\n",
       " 'Mz.',\n",
       " 'KS?',\n",
       " '\"80',\n",
       " 'ike',\n",
       " 'cas',\n",
       " '𝙗𝙞𝙜',\n",
       " 'vp,',\n",
       " 'wea',\n",
       " '3;',\n",
       " 'yo.',\n",
       " 'JFJ',\n",
       " '#1a',\n",
       " '@zz',\n",
       " '709',\n",
       " 'Aa',\n",
       " '15%',\n",
       " 'TP',\n",
       " 'TX;',\n",
       " 'tgo',\n",
       " 'MAI',\n",
       " 'Ed.',\n",
       " 'Spr',\n",
       " 'H!?',\n",
       " 'Fl.',\n",
       " 'IRG',\n",
       " '\"Ny',\n",
       " 'pew',\n",
       " 'Isn',\n",
       " '👽😺',\n",
       " 'ppg',\n",
       " '-n-',\n",
       " '604',\n",
       " 'V-1',\n",
       " '$31',\n",
       " 'Chp',\n",
       " 'JCs',\n",
       " '28A',\n",
       " 'TA:',\n",
       " 'sh*',\n",
       " 'B/W',\n",
       " '..i',\n",
       " 'Wè',\n",
       " 'KP',\n",
       " \"'da\",\n",
       " \"On'\",\n",
       " '{In',\n",
       " 'RJ.',\n",
       " ';P',\n",
       " '31a',\n",
       " 'BML',\n",
       " '\"♫♫',\n",
       " 'b/f',\n",
       " 'keg',\n",
       " 'Tok',\n",
       " '(2\"',\n",
       " 'HBS',\n",
       " '20G',\n",
       " '121',\n",
       " 'Yat',\n",
       " ':)',\n",
       " '2a.',\n",
       " 'WPZ',\n",
       " 'oMg',\n",
       " 'h-e',\n",
       " '`by',\n",
       " 'm.',\n",
       " 'and',\n",
       " 'I’',\n",
       " 'owm',\n",
       " 'BAs',\n",
       " 'VVB',\n",
       " 'LPN',\n",
       " 'kiz',\n",
       " 'a30',\n",
       " 'dow',\n",
       " '(JD',\n",
       " '90',\n",
       " '\"p',\n",
       " '\"96',\n",
       " '17t',\n",
       " 'iść',\n",
       " 'QFC',\n",
       " 'Gie',\n",
       " 'gps',\n",
       " 'me😨',\n",
       " 'pa.',\n",
       " 'BMA',\n",
       " 'oe,',\n",
       " '42A',\n",
       " 'F--',\n",
       " '$17',\n",
       " 'Gag',\n",
       " 'IWC',\n",
       " 'CVs',\n",
       " '6G:',\n",
       " 'S/.',\n",
       " 'CU!',\n",
       " '$6+',\n",
       " 'CJR',\n",
       " 'C5',\n",
       " 'PM)',\n",
       " 'elf',\n",
       " 'PFV',\n",
       " 'md;',\n",
       " \"'Il\",\n",
       " '>?',\n",
       " 'TL',\n",
       " 'QIN',\n",
       " 'Icy',\n",
       " 'LUC',\n",
       " '2.:',\n",
       " 'mf.',\n",
       " 'Hit',\n",
       " 'Hyw',\n",
       " 'BCL',\n",
       " 'www',\n",
       " '46m',\n",
       " '…“…',\n",
       " 'NIH',\n",
       " 'lug',\n",
       " 'WHy',\n",
       " '(s',\n",
       " 'HTC',\n",
       " 'Y.',\n",
       " '~2%',\n",
       " 'Des',\n",
       " 'PAL',\n",
       " 'dp!',\n",
       " 'AW,',\n",
       " ':/',\n",
       " 'DE!',\n",
       " '>!\"',\n",
       " 'Yep',\n",
       " '“go',\n",
       " 'By-',\n",
       " 'PND',\n",
       " 'DAY',\n",
       " 'IO',\n",
       " 'DL?',\n",
       " '5M+',\n",
       " 'K)',\n",
       " '498',\n",
       " 'Ms',\n",
       " 'UWB',\n",
       " '?;',\n",
       " '(ps',\n",
       " 'TJ!',\n",
       " 'k80',\n",
       " '727',\n",
       " '85.',\n",
       " 'HI',\n",
       " '???',\n",
       " '>$1',\n",
       " 'GLK',\n",
       " 'two',\n",
       " '4J,',\n",
       " 'SK.',\n",
       " 'fim',\n",
       " 'RI:',\n",
       " \"Si'\",\n",
       " '$s',\n",
       " 'OIW',\n",
       " 'Sí',\n",
       " 'rbi',\n",
       " 'seq',\n",
       " 'PBF',\n",
       " '4j',\n",
       " '/16',\n",
       " 'met',\n",
       " 'SIV',\n",
       " \"'…\",\n",
       " 'Pay',\n",
       " '(96',\n",
       " '46¢',\n",
       " 'Say',\n",
       " '#$@',\n",
       " '🚗',\n",
       " 'YA,',\n",
       " 'Pox',\n",
       " 'or?',\n",
       " 'F!',\n",
       " 'Law',\n",
       " 'ng',\n",
       " 'Rio',\n",
       " 'FBT',\n",
       " \"'03\",\n",
       " 'II”',\n",
       " '5w',\n",
       " 'NW\"',\n",
       " 'nyt',\n",
       " 'Die',\n",
       " 'Ojs',\n",
       " 'dL',\n",
       " 'gow',\n",
       " '◦',\n",
       " '78K',\n",
       " '12',\n",
       " 'NAC',\n",
       " 'tx',\n",
       " '(MJ',\n",
       " '298',\n",
       " 'UPA',\n",
       " 'S&L',\n",
       " '*my',\n",
       " 'NA!',\n",
       " 'GB2',\n",
       " 'CD1',\n",
       " 'Dm',\n",
       " 'VSB',\n",
       " '캐나다',\n",
       " 'DNS',\n",
       " 'usa',\n",
       " '73;',\n",
       " '76.',\n",
       " '\"90',\n",
       " 'Xs',\n",
       " '234',\n",
       " '4?!',\n",
       " 'moy',\n",
       " 'I/3',\n",
       " '\"93',\n",
       " '=16',\n",
       " 'K?\"',\n",
       " '&WW',\n",
       " 'iii',\n",
       " 'loi',\n",
       " 'NOT',\n",
       " 'nuk',\n",
       " '¡!?',\n",
       " 'DOB',\n",
       " 'II/',\n",
       " 'Why',\n",
       " 'Er',\n",
       " 'EDQ',\n",
       " '/15',\n",
       " '9am',\n",
       " 'arb',\n",
       " '-22',\n",
       " 'Vel',\n",
       " '\"DO',\n",
       " '(l',\n",
       " 'HBL',\n",
       " 'Dug',\n",
       " 'Fo',\n",
       " 'YLW',\n",
       " 'f)',\n",
       " 'ig',\n",
       " '21-',\n",
       " 'nun',\n",
       " '=45',\n",
       " 'STD',\n",
       " ';-|',\n",
       " 'N-1',\n",
       " 'eso',\n",
       " 'B62',\n",
       " 'OLP',\n",
       " 'Ww,',\n",
       " '15!',\n",
       " '.as',\n",
       " 'LTA',\n",
       " 'CDB',\n",
       " '06:',\n",
       " 'GM.',\n",
       " 'mbf',\n",
       " '27?',\n",
       " 'PoW',\n",
       " \"'80\",\n",
       " 'Kos',\n",
       " 'im-',\n",
       " ';-s',\n",
       " 'WT_',\n",
       " 'CCV',\n",
       " 'G7,',\n",
       " 'ndp',\n",
       " 'mx.',\n",
       " '✭',\n",
       " \"-'\",\n",
       " 'sw',\n",
       " \"AK'\",\n",
       " '£9k',\n",
       " '8oz',\n",
       " 'عمي',\n",
       " ',,a',\n",
       " 'Tha',\n",
       " 'dap',\n",
       " 'FM?',\n",
       " 'PSW',\n",
       " 'VRX',\n",
       " '\"Ed',\n",
       " 'JRA',\n",
       " 'ᾶ,',\n",
       " 'sh.',\n",
       " 'dod',\n",
       " '75%',\n",
       " 'O\"',\n",
       " 'rug',\n",
       " 'kif',\n",
       " 'cty',\n",
       " 'BBQ',\n",
       " '6+,',\n",
       " 'zac',\n",
       " '\"9\"',\n",
       " 'SA!',\n",
       " 'o9f',\n",
       " '𝙣𝙤𝙩',\n",
       " 'lan',\n",
       " 'Cor',\n",
       " 'bwa',\n",
       " '🇪🇺',\n",
       " 'WNW',\n",
       " 'PFE',\n",
       " 'in2',\n",
       " 'toi',\n",
       " 'NW',\n",
       " 'ha-',\n",
       " 'VdM',\n",
       " 'ro',\n",
       " 'lb',\n",
       " '🔄🔤',\n",
       " 'Kyl',\n",
       " '#1c',\n",
       " '(&',\n",
       " '19.',\n",
       " 'Sol',\n",
       " 'ORG',\n",
       " 'Y2%',\n",
       " '#4?',\n",
       " 'fòr',\n",
       " 'AIH',\n",
       " 'CEO',\n",
       " 'uH',\n",
       " '\\xada\\xad',\n",
       " '65-',\n",
       " 'Rem',\n",
       " 'McG',\n",
       " 'F&L',\n",
       " '-ah',\n",
       " 'tnr',\n",
       " 'FAC',\n",
       " 'TV/',\n",
       " 'BAc',\n",
       " 'nyk',\n",
       " 'IIn',\n",
       " '4\".',\n",
       " 'LE;',\n",
       " '\"BY',\n",
       " 'EMF',\n",
       " '417',\n",
       " 'jj,',\n",
       " 'SKB',\n",
       " 'Ift',\n",
       " 'EEC',\n",
       " '“Et',\n",
       " 'CUE',\n",
       " \"ku'\",\n",
       " 'hr/',\n",
       " 'SoA',\n",
       " 'Nvm',\n",
       " '48)',\n",
       " 'CLX',\n",
       " 'DC.',\n",
       " '[C]',\n",
       " 'bis',\n",
       " 'sno',\n",
       " 'fod',\n",
       " 'ER/',\n",
       " 'KDP',\n",
       " 'giy',\n",
       " 'PC',\n",
       " 'U16',\n",
       " 'ed)',\n",
       " 'upa',\n",
       " '\\'L\"',\n",
       " '👻😉',\n",
       " 'HQS',\n",
       " 'LVN',\n",
       " '6:',\n",
       " 'CG)',\n",
       " 'IMD',\n",
       " 'QO',\n",
       " \"J's\",\n",
       " \"'78\",\n",
       " 'u2',\n",
       " 'PEs',\n",
       " '-,',\n",
       " 'Or!',\n",
       " 'zz',\n",
       " '(VA',\n",
       " 'Ag',\n",
       " '980',\n",
       " 'RIT',\n",
       " '808',\n",
       " '98!',\n",
       " 'peo',\n",
       " \"up'\",\n",
       " '+P;',\n",
       " ':^(',\n",
       " 'ON\"',\n",
       " ';-(',\n",
       " 'Ga',\n",
       " 'HBM',\n",
       " '40)',\n",
       " 'io',\n",
       " 'huh',\n",
       " 'has',\n",
       " 'FIV',\n",
       " 'S26',\n",
       " 'ISU',\n",
       " '3Rs',\n",
       " 'lb+',\n",
       " '8:7',\n",
       " '53s',\n",
       " 'Qud',\n",
       " 'B1,',\n",
       " 'DIs',\n",
       " 'GUT',\n",
       " 'Ruh',\n",
       " '%I',\n",
       " 'JZ!',\n",
       " 'GS,',\n",
       " 'ѕo',\n",
       " '21;',\n",
       " 'c)',\n",
       " '5-9',\n",
       " 'Ab',\n",
       " '\"NW',\n",
       " '669',\n",
       " 'ya!',\n",
       " 'Re;',\n",
       " '#me',\n",
       " '(en',\n",
       " 'Cy,',\n",
       " 'sr.',\n",
       " 'in!',\n",
       " 'jrs',\n",
       " '5.A',\n",
       " 'bs;',\n",
       " ':=)',\n",
       " 'ek',\n",
       " 'MBE',\n",
       " 'NO”',\n",
       " '\"Do',\n",
       " 'BYW',\n",
       " 'RfR',\n",
       " '800',\n",
       " 'mh,',\n",
       " '😀😀',\n",
       " 'IV.',\n",
       " 'MUS',\n",
       " ',!!',\n",
       " '!5%',\n",
       " 'IE-',\n",
       " 'CP',\n",
       " 'M97',\n",
       " '~7',\n",
       " '289',\n",
       " 'PRL',\n",
       " 'em)',\n",
       " '\"Z.',\n",
       " '72M',\n",
       " 'URD',\n",
       " '68%',\n",
       " 'LB,',\n",
       " 'A@#',\n",
       " '(Gm',\n",
       " '/s',\n",
       " 'R:',\n",
       " 'PT?',\n",
       " '\"K\"',\n",
       " '(P8',\n",
       " 'me?',\n",
       " '755',\n",
       " '6-6',\n",
       " 'Dip',\n",
       " '4-F',\n",
       " 'LUF',\n",
       " 'WSL',\n",
       " 'PID',\n",
       " \"'52\",\n",
       " 'HVC',\n",
       " 'af',\n",
       " 'ILA',\n",
       " 'RO-',\n",
       " '(d)',\n",
       " '𝐭𝐡𝐞',\n",
       " 'Erm',\n",
       " \"I'e\",\n",
       " 'Ith',\n",
       " 'BT?',\n",
       " 'Nna',\n",
       " 'P.I',\n",
       " 'wha',\n",
       " 'am',\n",
       " 'R40',\n",
       " 'PIE',\n",
       " ':)-',\n",
       " 'I).',\n",
       " 'WFF',\n",
       " '🌟😊👍',\n",
       " 'S-S',\n",
       " ',I',\n",
       " 'SIS',\n",
       " 'Xie',\n",
       " 'UTX',\n",
       " \"'Ol\",\n",
       " 'Duc',\n",
       " 'FO,',\n",
       " '2cm',\n",
       " \"'An\",\n",
       " 'HK2',\n",
       " 'rip',\n",
       " 'ʻoe',\n",
       " 'OIP',\n",
       " 'AR\"',\n",
       " '\"C\\'',\n",
       " 'FK,',\n",
       " '29)',\n",
       " \"ma'\",\n",
       " 'M7',\n",
       " 'i/',\n",
       " 'Roj',\n",
       " 'seg',\n",
       " 'MJH',\n",
       " 'BCS',\n",
       " 'NWA',\n",
       " 'epd',\n",
       " '♫♪\"',\n",
       " 'Moa',\n",
       " 'mtg',\n",
       " 'WHL',\n",
       " '677',\n",
       " 'UI?',\n",
       " 'ROW',\n",
       " 'tlc',\n",
       " '<To',\n",
       " 'YA!',\n",
       " '𝒍𝒂𝒃',\n",
       " 'UA?',\n",
       " 'I️',\n",
       " 'asa',\n",
       " 'Wn',\n",
       " 'ASS',\n",
       " 'Lah',\n",
       " 'OD.',\n",
       " ':).',\n",
       " 'wer',\n",
       " 'hte',\n",
       " 'y\"',\n",
       " '~An',\n",
       " '#76',\n",
       " '44?',\n",
       " 'I\".',\n",
       " \"Js'\",\n",
       " '‘?',\n",
       " 'FNC',\n",
       " 'to)',\n",
       " '7a!',\n",
       " 'CNs',\n",
       " 'ays',\n",
       " 'U.',\n",
       " '`It',\n",
       " 'un',\n",
       " 'us3',\n",
       " 'Ha)',\n",
       " '517',\n",
       " 'OT:',\n",
       " '9x',\n",
       " 'RJC',\n",
       " 'LP\"',\n",
       " '(86',\n",
       " '7%,',\n",
       " 'Q/',\n",
       " 'EXL',\n",
       " 'CBC',\n",
       " 'H2O',\n",
       " '(C8',\n",
       " '\"54',\n",
       " 'sft',\n",
       " 'eps',\n",
       " 'WUB',\n",
       " '4J:',\n",
       " 'HAV',\n",
       " 'CEE',\n",
       " ',Ur',\n",
       " 'I25',\n",
       " '(H,',\n",
       " 'V8s',\n",
       " 'Tbh',\n",
       " \"f'd\",\n",
       " ...}"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code> **Observations**: \n",
    "<code>1. contain lots of meaningless word which may not found in english dicitionary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <Code> Overall Observation: <Code><br>\n",
    "<code> 1. removing Emojis from data is not good idea. emojis holds lots of sentimental meaning, we have to find the way in which we can convert emojis to its approximate meaning <code>,<br>\n",
    "<code> 2. as we can see there are lots of meaningless data is present , so we have to remove that data (Non english words).<br>\n",
    "<code> 3. we also have to perform dcontraction of word, removing of extra space , removing of Punctuations, http/https links, removing stop words excluding ** NOT **  word.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. contain both lower and upper latter \n",
    "2. contain numbers \n",
    "3. contain extra space \n",
    "4. contain http/https links \n",
    "5. contain Punctuations \n",
    "6. contain emojis \n",
    "7. contain other languages like Chinese \n",
    "8. contain word english in different styles \n",
    "9. contain lots of meaningless word which may not found in english dicitionary\n",
    "8.2 strategy to preprocess text data.\n",
    "1.removing Emojis from data is not good idea. emojis holds lots of sentimental meaning, we have to find the way in which we can convert emojis to its approximate meaning.\n",
    "2. as we can see there are lots of meaningless words are present , so we have to remove that words (Non english words).\n",
    "3. we also have to perform dcontraction of word, removing of extra space , removing of Punctuations, http/https links, removing stop words excluding  NOT   word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## B. Preprocessing toy example with code Explanation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. <i> Converting emojis to meaningful word <i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> thoughts come to mind \n",
    "* problem with this library :https://stackoverflow.com/questions/57744725/how-to-convert-emojis-emoticons-to-their-meanings-in-python ,\n",
    "* e.g : 'I ❤ New York'\n",
    "    * i love New York ----> expected\n",
    "    * i heart New York -----> we got.\n",
    "* as in glove vector similar things have almost same vector representation (not exactly same but near by) so model will take care of that situation. **NO WORRY** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/questions/57580288/how-to-replace-emoji-to-word-in-a-text\n",
    "import emoji\n",
    "def emojis_to_word(text):\n",
    "    text = emoji.demojize(text, delimiters=(\"\", \"\"))\n",
    "    text = text.replace('_',' ')\n",
    "    return text\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'game is on fire'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "''' Toy example'''\n",
    "emojis_to_word('game is on 🔥')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. <i> Converting to Lower case <i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">* Lower casing is a common text preprocessing technique. The idea is to convert the input text into same casing format so that 'text', 'Text' and 'TEXT' are treated the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data['comment_text'] = Data['comment_text'].str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. <i>Decontraction of word<i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#https://www.geeksforgeeks.org/nlp-expand-contractions-in-text-processing/\n",
    "import contractions\n",
    "def decontraction(text):\n",
    "    expanded_words = []    \n",
    "    for word in text.split():\n",
    "      # using contractions.fix to expand the shotened words\n",
    "        expanded_words.append(contractions.fix(word))   \n",
    "\n",
    "    expanded_text = ' '.join(expanded_words)\n",
    "    return expanded_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'You are not see nobody like this before.'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decontraction(\"You ain’t see nobody like this before.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4A. <i>Applying Lemmatization<i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">lemmatization is similar to stemming in reducing inflected words to their word stem but differs in the way that it makes sure the root word (also called as lemma) belongs to the language.\n",
    "* https://www.kaggle.com/sudalairajkumar/getting-started-with-text-preprocessing#Lemmatization\n",
    "* https://www.guru99.com/stemming-lemmatization-python-nltk.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "better : better\n"
     ]
    }
   ],
   "source": [
    "''' toy example'''\n",
    "# import these modules\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "print(\"better :\", lemmatizer.lemmatize(\"better\"))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "def Lemmatization(text):\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    text = lemmatizer.lemmatize(str(text))\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'what a Moe-ron'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lemmatization('what a Moe-ron')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'studies studying cries cry goods sacas cried '"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Lemmatization('studies studying cries cry goods sacas cried ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4B. <i>Applying Stemming <i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "stemmer = PorterStemmer()\n",
    "def stem_words(text):\n",
    "    return \" \".join([stemmer.stem(word) for word in text.split()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'studi studi cri cri good saca cri'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stem_words('studies studying cries  cry goods sacas cried ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'cri ' in embeddings_index:\n",
    "    print(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5A. <i>Removing non english word <i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ### There are two wasy to do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. we can use NLTK english library :\n",
    "    * Problem is that NLTk library dont have enough english vocabulary as well it remove purls word to.\n",
    "    * nltk corpus words is not exhaustive in nature, it does not contain all the different forms of a word, synonyms of a word, etc... :/ it only contains 235886 unique English words\n",
    "    * https://stackoverflow.com/questions/41290028/removing-non-english-words-from-text-using-python\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "original:\n",
      " we plays history day words ansdknak \n",
      "\n",
      "after NLTK :\n",
      " we history day\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "words = set(nltk.corpus.words.words())\n",
    "\n",
    "sent = \"we plays history day words ansdknak\"\n",
    "\n",
    "result = \" \".join(w for w in nltk.wordpunct_tokenize(sent)if w.lower() in words or not w.isalpha())\n",
    "print('\\noriginal:\\n',sent,'\\n\\nafter NLTK :\\n',result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "sent = 'we plays history  降你失去所 有的钱加拿大坏税骗 day  骗good '\n",
    "result = \" \".join(w for w in nltk.wordpunct_tokenize(sent)if w.lower() in words or not w.isalpha())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "original:\n",
      " we plays history  降你失去所 有的钱加拿大坏税骗 day  骗good  \n",
      "\n",
      "after NLTK :\n",
      " we history day\n"
     ]
    }
   ],
   "source": [
    "print('\\noriginal:\\n',sent,'\\n\\nafter NLTK :\\n',result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ** we can see that NLTK is not handling the data well it drops word \"plays\" as well \"words\" **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. We can create our own vocabulary.\n",
    "    \n",
    "    * as we are using glove vector it always recommended that we alwas keep our vocabulary as close as to glove vocabulary.\n",
    "    * https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We will use custom vocab. glove vocab to drop non english word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 400000 word vectors.\n"
     ]
    }
   ],
   "source": [
    "## Glove file\n",
    "from numpy import asarray\n",
    "# load the whole embedding into memory\n",
    "embeddings_index = dict()\n",
    "f = open(r\"C:\\Users\\Lenovo\\Downloads\\glove.6B\\glove.6B.100d.txt\",encoding=\"utf8\")\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "print('Loaded %s word vectors.' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Non_english_word_dropping(text):\n",
    "    final = []\n",
    "    text_temp = text.split()\n",
    "    for i in text_temp:\n",
    "        if i in embeddings_index:\n",
    "            final.append(i)\n",
    "    text = ' '.join(final)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "sent = 'what a Moe-ron'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = Non_english_word_dropping(sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "original:\n",
      " what a Moe-ron \n",
      "\n",
      "after NLTK :\n",
      " what a\n"
     ]
    }
   ],
   "source": [
    "print('\\noriginal:\\n',sent,'\\n\\nafter NLTK :\\n',result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "original:\n",
      " we plays history  降你失去所 有的钱加拿大坏税骗 day  骗good  \n",
      "\n",
      "after NLTK :\n",
      " we plays history day\n"
     ]
    }
   ],
   "source": [
    "sent = 'we plays history  降你失去所 有的钱加拿大坏税骗 day  骗good '\n",
    "result = Non_english_word_dropping(sent)\n",
    "print('\\noriginal:\\n',sent,'\\n\\nafter NLTK :\\n',result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ### It work fine but problem is that it not handling this case: 骗good\n",
    ">so we first have to collect all non english character (other language,symbol) and then we have to remove that"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5B. <i>Removing non english Character/symbols and stop words <i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda\n",
    "def build_vocab(sentences, verbose =  True):\n",
    "    \"\"\"\n",
    "    :param sentences: list of list of words\n",
    "    :return: dictionary of words and their count\n",
    "    \"\"\"\n",
    "    vocab = {}\n",
    "    for sentence in tqdm(sentences, disable = (not verbose)):\n",
    "        for word in sentence:\n",
    "            try:\n",
    "                vocab[word] += 1\n",
    "            except KeyError:\n",
    "                vocab[word] = 1\n",
    "    return vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "latin_similar = \"’'‘ÆÐƎƏƐƔĲŊŒẞÞǷȜæðǝəɛɣĳŋœĸſßþƿȝĄƁÇĐƊĘĦĮƘŁØƠŞȘŢȚŦŲƯY̨Ƴąɓçđɗęħįƙłøơşșţțŧųưy̨ƴÁÀÂÄǍĂĀÃÅǺĄÆǼǢƁĆĊĈČÇĎḌĐƊÐÉÈĖÊËĚĔĒĘẸƎƏƐĠĜǦĞĢƔáàâäǎăāãåǻąæǽǣɓćċĉčçďḍđɗðéèėêëěĕēęẹǝəɛġĝǧğģɣĤḤĦIÍÌİÎÏǏĬĪĨĮỊĲĴĶƘĹĻŁĽĿʼNŃN̈ŇÑŅŊÓÒÔÖǑŎŌÕŐỌØǾƠŒĥḥħıíìiîïǐĭīĩįịĳĵķƙĸĺļłľŀŉńn̈ňñņŋóòôöǒŏōõőọøǿơœŔŘŖŚŜŠŞȘṢẞŤŢṬŦÞÚÙÛÜǓŬŪŨŰŮŲỤƯẂẀŴẄǷÝỲŶŸȲỸƳŹŻŽẒŕřŗſśŝšşșṣßťţṭŧþúùûüǔŭūũűůųụưẃẁŵẅƿýỳŷÿȳỹƴźżžẓ\"\n",
    "# white_list = string.ascii_letters + string.digits + latin_similar + ' '\n",
    "white_list = string.ascii_letters + latin_similar + ' '\n",
    "white_list += \"'\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:44<00:00, 40673.95it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'.,?!-;*\"…:\\n150—()%263#$97&_8/@＼・ω4+🍕=”“[]^–>\\r🐵\\\\°<😑~\\xa0\\ue014•≠\\t™\\uf818\\uf04a\\xadˈʊɒ😢🐶∞§{}·τα❤️☺ɡ\\uf0e0😜😎👊\\u200b\\u200e😁|عدويهصقأناخلىبمغر😍💖¢→̶`💵❥━┣┫Е┗Ｏ►★👎😀😂\\u202a\\u202c🔥😄©―🏻💥ᴍʏʀɪᴇɴᴅᴏᴀᴋʜᴜʟᴛᴄᴘʙғᴊᴡɢ✔®\\x96\\x92●😋👏שלוםבי😱‼£\\x81♥エンジ故障➤´\\u2009🚌ᴵ͞🌟😊😳😧🙀😐😕\\u200f👍😮😃😘¹☕≈÷אעכח♡◐║▬💩′ɔː💯⛽€🚄🏼ஜ۩۞†😖ᴠ🚲‐μ✒➥😟😈═☆ˌ💪🙏🎯◄🌹😇💔½ʻ😡\\x7f👌ἐπὶδηλήσειὲκἀίῃἴρξνʃ🙄✬ＳＵＰＥＲＨＩＴ😠\\ufeff☻±\\u2028😉😤⛺♍🙂µ\\u3000تحكسة👮💙فزط😏º🍾🎉¾😞\\u2008🏾😅😭👻😥😔😓🏽🎆✓◾🍻🍽🎶🌺🤔😪\\x08‑؟🐰🐇🐱🙆．😨⬅🙃💕𝘊𝘦𝘳𝘢𝘵𝘰𝘤𝘺𝘴𝘪𝘧𝘮𝘣💗💚地獄谷℅»ВулканПвоАН🐾🐕❣😆ה⋅🔗¿¬🚽歌舞伎🙈😴🏿🤗🇺🇸♫мυтѕＣＭ⤵🏆🎃β😩█▓▒░\\u200a🌠🐟💫💰💎⇒эпрд\\x95🖐🙅⛲🍰⭐🤐👆›🙌\\u2002💛🙁👀🙊🙉¡₂₃\\u2004❧▰ˢᵒʳʸ▔ᴼᴷᴺʷᵗʰᵉᵘ◞▀\\x13🚬▂▃▄▅▆▇↙🤓\\ue602😵άοόςέγὸ̄תמדףנרךצט😒͝″☹➡«🆕👅👥👄🔄🔤👉👤👶👲🔛🎓φ\\uf0b7⅓„✋：\\uf04c\\x9f\\x10成都¥😣⏺̲̅😌🤑́🌏😯ех😲∙‛Ἰᾶὁ💞🚓◇🔔📚✏🏀👐\\u202d💤🍇\\ue613小土豆🏡▷❔❓⁉❗\\u202f👠¶》कर्मा🇹🇼🌸蔡英文🌞˚🎲レクサス😛˙外国人关系）Ссиб💋💀🎄💜🤢َِʿьыгя✨不是。ɑ\\x80\\x9c\\x9d🗑\\u2005💃📣👿༼つ◕༽😰ḷЗз▱ц￼🤣卖！温哥华议会下降％你失去所有的钱加拿大坏税骗子🐝¯ツ🎅\\x85🍺آإشء−ﬂﬁ🎵🌎͟ἔ油别克🤡🤥😬🤧й\\u2003₁²🚀🤴ʌʲш¼⁴⁄₄⌠чИОРФДЯМю♭ж✘😝🖑ὐύύ特殊作戦群╪щ💨圆明园ק▶ℐ☭✭🏈😺♪🌍⏏ệ🍔🐮🍁☔🍆🍑🌮🌯☠🤦\\u200d♂𝓒𝓲𝓿𝓵안영하세요ЖљКћ🍀😫🤤ῦ我出生在了可以说普通话汉语好极🎼🕺☃🍸🥂🗽🎇🎊🆘☎🤠👩✈🖒✌✰❆☙🚪天一家⚲\\u2006⚭⚆⬭⬯⏖○‣⚓新年∎ℒ▪▙☏⅛✀╌🇫🇷🇩🇪🇮🇬🇧😷🇨🇦ХШ🌐\\x1f杀鸡给猴看ʁ𝗪𝗵𝗲𝗻𝘆𝗼𝘂𝗿𝗮𝗹𝗶𝘇𝗯𝘁𝗰𝘀𝘅𝗽𝘄𝗱📺ｃϖ\\u2000үսａᴦᎥһͺ\\u2007հｓǀ\\u2001ɩ℮ｙｅ൦ｌƽ¸ｗｈ𝐓𝐡𝐞𝐫𝐮𝐝𝐚𝐃𝐜𝐩𝐭𝐢𝐨𝐧Ƅᴨ‚ןᑯ໐ΤᏧ௦Іᴑ܁𝐬𝐰𝐲𝐛𝐦𝐯𝐑𝐙𝐣𝐇𝐂𝐘𝟎ԜТᗞ౦〔Ꭻ𝐳𝐔𝐱𝟔𝟓𝐅🐋∼ﬃ💘💓ё𝘥𝘯𝘶‖💐🌋🌄🌅𝙬𝙖𝙨𝙤𝙣𝙡𝙮𝙘𝙠𝙚𝙙𝙜𝙧𝙥𝙩𝙪𝙗𝙞𝙝𝙛👺🐷ℋℳ𝐀𝐥𝐪❄🚶←𝙢Ἱ🤘ͦ💸☼ج패티Ｗ⋆𝙇ʒᵻ👂👃ɜ🎫\\uf0a7БУі🚢⊂🚂ગુજરાતીῆ🏃、⅔𝓬𝓻𝓴𝓮𝓽𝓼☘¨﴾͡๏̯﴿⚾⚽Φ₽\\ue807𝑻𝒆𝒍𝒕𝒉𝒓𝒖𝒂𝒏𝒅𝒔𝒎𝒗𝒊👽😙\\u200cЛ×‒🎾👹θ￦⎌🏒⛸公寓养宠物吗？🏄🐀🚑🤷（操美𝒑𝒚𝒐𝑴🤙🐒℃欢迎来到阿拉斯ספ𝙫⏩☮🐈𝒌𝙊𝙭𝙆𝙋𝙍𝘼𝙅ﷻ⚠🦄巨收赢得月白鬼愤怒要买额ẽ🚗✊🐳𝟏𝐟𝟖𝟑𝟕𝒄𝟗𝐠𝙄𝙃👇锟斤拷❌⭕▸𝗢𝟳𝟱𝟬⦁マルハニチロ株式社⛷한국어ㄸㅓ니͜ʖ■⇌𝘿𝙔₵𝒩ℯ𝒾𝓁𝒶𝓉𝓇𝓊𝓃𝓈𝓅ℴ𝒻𝒽𝓀𝓌𝒸𝓎𝙏ζ𝙟𝘃𝗺𝟮𝟭𝟯𝟲👋🦊☐☑多伦⚡☄ǫ🐽🎻🎹⛓🏹╭∩╮🍷🦆为和中友谊祝贺与其想象对法如直接问，用自己猜本传教士没积唯认识基督徒曾经让相信耶稣复活死怪他但当们聊些政治题时候例战胜因圣把全堂结婚孩恐惧且栗谓这样还♾🎸🤕🤒⛑🎁批判检讨🏝🦁＞ʕɐ̣Δ₀🙋😶쥐스탱트뤼도석유가격인상이경제황을렵게만들지않록잘관리해야합다캐나에서대마초와화약금의품런성분갈때는반드시허된사용✞🔫👁┈╱╲▏▕┃╰▊▋╯┳┊≥☒凸ὰ↑💲🗯𝙈Ἄ𝒇𝒈𝒘𝒃𝑬𝑶𝕾𝖙𝖗𝖆𝖎𝖌𝖍𝖕𝖊𝖔𝖑𝖉𝖓𝖐𝖜𝖞𝖚𝖇𝕿𝖘𝖄𝖛𝖒𝖋𝖂𝕴𝖟𝖈𝕸👑🚿☝💡知彼百\\uf005𝙀𝒛𝑲𝑳𝑾𝒋𝟒😦𝙒𝘾𝘽🏐ɹ𝘩𝘨ὼṑ✅☛𝑱𝑹𝑫𝑵𝑪🇰🇵👾ᓇᒧᔭᐃᐧᐦᑳᐨᓃᓂᑲᐸᑭᑎᓀᐣ🐄🎈🔨♩🐎🤞☞🐸💟🎰🌝🛳点击查版🍭𝑥𝑦𝑧ＡＮＧＪＢ👣\\uf020っ◔◡🏉↓ф💭🎥♀Ξ🐴👨🤳⬆🦍\\x0b🍩𝑯𝒒😗𝟐🏂👳🍗🕉🐲چی̱ℏ𝑮𝗕𝗴\\x91🍒⠀ꜥˤⲣⲏ╚🐑⏰↺⇤∏鉄リ事件✾◦♬³ї💊「」\\uf203\\uf09a\\uf222\\ue608\\uf202\\uf099\\uf469\\ue607\\uf410\\ue600燻製シの虚偽屁理屈｜／Г𝑩𝑰𝒀𝑺🌤∵∴√𝗳𝗜𝗙𝗦𝗧🍊ὺἈἡχῖΛΩ¤⤏🇳𝒙ψՁմեռայինրւդձ冬至ὀ𝒁🔹🤚🍎𝑷🐂💅𝘬𝘱𝘸𝘷𝘐𝘭𝘓𝘖𝘹𝘲𝘫ک☜Βώ💢▲ΜΟΝΑΕ🇱♲𝝈↴↳💒⊘▫Ȼ‿⬇🚴🖕🖤🥘📍👈➕🚫🎨🌑🐻𝐎𝐍𝐊𝑭🤖🎎✧😼🕷ｇｏｖｒｎｍｔｉｄｕ－２０８ｆｂ＇ｋ𝟰🇴🇭🇻🇲𝗞𝗭𝗘𝗤‰≤👼📉🍟🍦∕🌈🔭《🐊🐍\\uf10aˆ⚜☁ლڡ🐦\\U0001f92f\\U0001f92a🐡💳ἱ🙇𝗸𝗟𝗠𝗷🥜さようなら🔼'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "# https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda\n",
    "jigsaw_chars = build_vocab(list(Data[\"comment_text\"]))\n",
    "jigsaw_symbols = ''.join([c for c in jigsaw_chars if not c in white_list])\n",
    "jigsaw_symbols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"i, me, my, myself, we, our, ours, ourselves, you, you're, you've, you'll, you'd, your, yours, yourself, yourselves, he, him, his, himself, she, she's, her, hers, herself, it, it's, its, itself, they, them, their, theirs, themselves, what, which, who, whom, this, that, that'll, these, those, am, is, are, was, were, be, been, being, have, has, had, having, do, does, did, doing, a, an, the, and, but, if, or, because, as, until, while, of, at, by, for, with, about, against, between, into, through, during, before, after, above, below, to, from, up, down, in, out, on, off, over, under, again, further, then, once, here, there, when, where, why, how, all, any, both, each, few, more, most, other, some, such, no, nor, not, only, own, same, so, than, too, very, s, t, can, will, just, don, don't, should, should've, now, d, ll, m, o, re, ve, y, ain, aren, aren't, couldn, couldn't, didn, didn't, doesn, doesn't, hadn, hadn't, hasn, hasn't, haven, haven't, isn, isn't, ma, mightn, mightn't, mustn, mustn't, needn, needn't, shan, shan't, shouldn, shouldn't, wasn, wasn't, weren, weren't, won, won't, wouldn, wouldn't\""
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\", \".join(stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "179\n",
      "176\n"
     ]
    }
   ],
   "source": [
    "STOPWORDS = set(stopwords.words('english'))\n",
    "print(len(STOPWORDS))\n",
    "STOPWORDS = list(STOPWORDS - {'no','nor','not'})\n",
    "print(len(STOPWORDS))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_stopwords(text):\n",
    "    \"\"\"custom function to remove the stopwords\"\"\"\n",
    "    return \" \".join([word for word in str(text).split() if word not in STOPWORDS])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "sent = 'we plays history or i did we can  day'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'plays history day'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a=remove_stopwords(sent)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_jigsaw_symbols(text):\n",
    "    \"\"\"custom function to remove the punctuation\"\"\"\n",
    "    return text.translate(str.maketrans('', '', jigsaw_symbols))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'we plays   history    day  good '"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = 'we plays😮 😃 😘¹ history  降你失去所 有的钱加拿大坏税骗 day  骗good '\n",
    "a = remove_jigsaw_symbols(sent)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "''' lets write both in one Function'''\n",
    "def remove_jigsaw_symbol_stop_word(text):\n",
    "    ## Dont change order of calling function 1.remove_jigsaw_symbols and 2.remove_stopwords\n",
    "    text = remove_jigsaw_symbols(text)\n",
    "    text = remove_stopwords(text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'plays history not day good'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = 'we plays😮 😃 😘¹ history  降你失去所  a did not ,or  有的钱加拿大坏税骗 day  骗good '\n",
    "a = remove_jigsaw_symbol_stop_word(sent)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. <i>Removing http/https links <i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* https://stackoverflow.com/questions/6532089/regex-to-catch-any-http-address"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'You assume that they would staff it from external states with a influx of 200,000 people.  That is not necessarily true.What if they recruited 50,000 employees locally?  That is more likely.  Some executive types might move here from elsewhere, but most of the hiring would be here - just like with their warehouse. http://www.denverpost.com/2017/01/23/colorado-amazon-fulfillment-center-aurora/ '"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp = 'You assume that they would staff it from external states with a influx of 200,000 people.  That is not necessarily true.What if they recruited 50,000 employees locally?  That is more likely.  Some executive types might move here from elsewhere, but most of the hiring would be here - just like with their warehouse. http://www.denverpost.com/2017/01/23/colorado-amazon-fulfillment-center-aurora/ '\n",
    "temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def removing_links(text):\n",
    "    text = re.sub(\"(http|https)://[\\w\\-]+(\\.[\\w\\-]+)+\\S*\", \"\", text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'You assume that they would staff it from external states with a influx of 200,000 people.  That is not necessarily true.What if they recruited 50,000 employees locally?  That is more likely.  Some executive types might move here from elsewhere, but most of the hiring would be here - just like with their warehouse.  '"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = removing_links(temp)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. <i>removal of Numbers,tabs and newline (\\t,\\n) <i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_num_tab_newline(text):\n",
    "    text = re.sub(r\"[\\d\\t\\n]*\", \"\", text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'shantanu sunil \\n12223 212 ekhande hey \\t  hi\\t  hi2 \\ndbjavjcda'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp='shantanu sunil \\n12223 212 ekhande hey \\t  hi\\t  hi2 \\ndbjavjcda'\n",
    "temp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'shantanu sunil   ekhande hey   hi  hi dbjavjcda'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = remove_num_tab_newline(temp)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. <i>removal Punctuations(\\t,\\n) <i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> we have already removed punctuation,number, when we removed jigsaw symbols. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. <i>removal extra space<i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_extra_speces(text):\n",
    "    text = re.sub(r\"[\\s\\s]+\", \" \", text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp='shantanu sunil  ekhande hey   hi       dbjavjcda'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'shantanu sunil ekhande hey hi dbjavjcda'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "remove_extra_speces(temp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## C. Writing single function for preprocessing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessing_DL(Data,submission_data):\n",
    "    ## Note : dont change preprocessing sequence\n",
    "    ''' Converting Emojis to word'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text\"].progress_apply(lambda text: emojis_to_word(text))\n",
    "    '''Converting in Lower case'''\n",
    "    Data[\"comment_text_pre\"] = Data['comment_text_pre'].str.lower()\n",
    "    '''Decontraction'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: decontraction(text))\n",
    "    '''Lemmatization'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: Lemmatization(text))\n",
    "    '''removing_links'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: removing_links(text))\n",
    "    '''remove_jigsaw_symbol/character_stop_word'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: remove_jigsaw_symbol_stop_word(text))\n",
    "    '''Non_english_word_dropping'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: Non_english_word_dropping(text))\n",
    "    '''remove_num_tab_newline'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: remove_num_tab_newline(text))\n",
    "    '''remove_extra_speces'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: remove_extra_speces(text))\n",
    "    \n",
    "    if submission_data == False:\n",
    "        '''Converting class probability into label'''\n",
    "        Data['class'] = Data['target'].progress_apply(lambda x: \"non-toxic\" if x < 0.5 else \"toxic\")\n",
    "    return Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessing_ML(Data,submission_data):\n",
    "    ## Note : dont change preprocessing sequence\n",
    "    ''' Converting Emojis to word'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text\"].progress_apply(lambda text: emojis_to_word(text))\n",
    "    '''Converting in Lower case'''\n",
    "    Data[\"comment_text_pre\"] = Data['comment_text_pre'].str.lower()\n",
    "    '''Decontraction'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: decontraction(text))\n",
    "    '''removing_links'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: removing_links(text))\n",
    "    '''remove_jigsaw_symbol/character_stop_word'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: remove_jigsaw_symbol_stop_word(text))\n",
    "    '''Non_english_word_dropping'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: Non_english_word_dropping(text))\n",
    "    '''Stemming'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: stem_words(text))\n",
    "    '''remove_num_tab_newline'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: remove_num_tab_newline(text))\n",
    "    '''remove_extra_speces'''\n",
    "    Data[\"comment_text_pre\"] = Data[\"comment_text_pre\"].progress_apply(lambda text: remove_extra_speces(text))\n",
    "    \n",
    "    if submission_data == False:\n",
    "        '''Converting class probability into label'''\n",
    "        Data['class'] = Data['target'].progress_apply(lambda x: \"non-toxic\" if x < 0.5 else \"toxic\")\n",
    "    return Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/questions/45595689/how-to-fix-tqdm-progress-apply-for-pandas-in-jupyter\n",
    "# df['Value'].progress_apply(lambda x: x**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "tqdm.pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing the data for ML models\n",
    "### which contain:\n",
    "    * preprocessing Train.csv\n",
    "    * preprocessing Test.csv (for submission file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## 1. Creating small dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>severe_toxicity</th>\n",
       "      <th>obscene</th>\n",
       "      <th>identity_attack</th>\n",
       "      <th>insult</th>\n",
       "      <th>threat</th>\n",
       "      <th>asian</th>\n",
       "      <th>atheist</th>\n",
       "      <th>...</th>\n",
       "      <th>funny</th>\n",
       "      <th>wow</th>\n",
       "      <th>sad</th>\n",
       "      <th>likes</th>\n",
       "      <th>disagree</th>\n",
       "      <th>sexual_explicit</th>\n",
       "      <th>identity_annotator_count</th>\n",
       "      <th>toxicity_annotator_count</th>\n",
       "      <th>comment_text_pre</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>516089</th>\n",
       "      <td>875306</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Wente has no clue what she is talking about. S...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>wente no clue talking knows democratic party u...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1693158</th>\n",
       "      <td>6197921</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>No, it's obvious she doesn't understand this a...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>no obvious not understand nonnative native lan...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1331033</th>\n",
       "      <td>5741886</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>If NK knows that their next bomb will be their...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>nk knows next bomb last dont think guam key ta...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1075854</th>\n",
       "      <td>5431454</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Point well taken Margaret.  It ends up being a...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>point well taken margaret ends spin political ...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1316108</th>\n",
       "      <td>5723769</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>The past belonged to one gender.  Now the futu...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>past belonged one gender future belongs anothe...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id    target                                       comment_text  \\\n",
       "516089    875306  0.000000  Wente has no clue what she is talking about. S...   \n",
       "1693158  6197921  0.000000  No, it's obvious she doesn't understand this a...   \n",
       "1331033  5741886  0.000000  If NK knows that their next bomb will be their...   \n",
       "1075854  5431454  0.000000  Point well taken Margaret.  It ends up being a...   \n",
       "1316108  5723769  0.166667  The past belonged to one gender.  Now the futu...   \n",
       "\n",
       "         severe_toxicity  obscene  identity_attack    insult  threat  asian  \\\n",
       "516089               0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "1693158              0.0      0.0              0.0  0.000000     0.0    0.0   \n",
       "1331033              0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "1075854              0.0      0.0              0.0  0.000000     0.0    0.0   \n",
       "1316108              0.0      0.0              0.0  0.166667     0.0    0.0   \n",
       "\n",
       "         atheist  ...  funny  wow  sad  likes  disagree  sexual_explicit  \\\n",
       "516089       NaN  ...      0    0    0      0         4         0.000000   \n",
       "1693158      0.0  ...      0    0    0      0         0         0.000000   \n",
       "1331033      NaN  ...      0    0    0      0         0         0.000000   \n",
       "1075854      0.0  ...      0    0    0      9         3         0.000000   \n",
       "1316108      0.0  ...      0    0    0      0         0         0.166667   \n",
       "\n",
       "         identity_annotator_count  toxicity_annotator_count  \\\n",
       "516089                          0                         4   \n",
       "1693158                        10                         4   \n",
       "1331033                         0                         4   \n",
       "1075854                         6                         4   \n",
       "1316108                         4                         6   \n",
       "\n",
       "                                          comment_text_pre      class  \n",
       "516089   wente no clue talking knows democratic party u...  non-toxic  \n",
       "1693158  no obvious not understand nonnative native lan...  non-toxic  \n",
       "1331033  nk knows next bomb last dont think guam key ta...  non-toxic  \n",
       "1075854  point well taken margaret ends spin political ...  non-toxic  \n",
       "1316108  past belonged one gender future belongs anothe...  non-toxic  \n",
       "\n",
       "[5 rows x 47 columns]"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_data = Data.sample(n = 100)\n",
    "sample_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## preprocessing_sample_ML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 8458.65it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 6246.17it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 100414.27it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 4754.91it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 5013.33it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 2179.37it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 33724.40it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 49107.88it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<?, ?it/s]\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_sample_ML= preprocessing_ML(sample_data,submission_data=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comment_text</th>\n",
       "      <th>comment_text_pre</th>\n",
       "      <th>target</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>516089</th>\n",
       "      <td>Wente has no clue what she is talking about. S...</td>\n",
       "      <td>went no clue talk know democrat parti us well ...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1693158</th>\n",
       "      <td>No, it's obvious she doesn't understand this a...</td>\n",
       "      <td>no obviou not understand nonn nativ land tacit...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1331033</th>\n",
       "      <td>If NK knows that their next bomb will be their...</td>\n",
       "      <td>nk know next bomb last dont think guam key tar...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1075854</th>\n",
       "      <td>Point well taken Margaret.  It ends up being a...</td>\n",
       "      <td>point well taken margaret end spin polit desir...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1316108</th>\n",
       "      <td>The past belonged to one gender.  Now the futu...</td>\n",
       "      <td>past belong one gender futur belong anoth conc...</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1273859</th>\n",
       "      <td>Can't wait to see who portrays the Mooch in th...</td>\n",
       "      <td>cannot wait see portray mooch next new segment...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1249009</th>\n",
       "      <td>I doubt that too. He probably sends them ramad...</td>\n",
       "      <td>doubt probabl send ramadan card</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685573</th>\n",
       "      <td>BC Hydro does not have the widespread infrastr...</td>\n",
       "      <td>bc hydro not widespread infrastructur back ga ...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>836980</th>\n",
       "      <td>Time for some good old Alaskan duct tape to fi...</td>\n",
       "      <td>time good old alaskan duct tape fix broken stuff</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>435975</th>\n",
       "      <td>Everyone should try to understand.</td>\n",
       "      <td>everyon tri understand</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              comment_text  \\\n",
       "516089   Wente has no clue what she is talking about. S...   \n",
       "1693158  No, it's obvious she doesn't understand this a...   \n",
       "1331033  If NK knows that their next bomb will be their...   \n",
       "1075854  Point well taken Margaret.  It ends up being a...   \n",
       "1316108  The past belonged to one gender.  Now the futu...   \n",
       "...                                                    ...   \n",
       "1273859  Can't wait to see who portrays the Mooch in th...   \n",
       "1249009  I doubt that too. He probably sends them ramad...   \n",
       "685573   BC Hydro does not have the widespread infrastr...   \n",
       "836980   Time for some good old Alaskan duct tape to fi...   \n",
       "435975                  Everyone should try to understand.   \n",
       "\n",
       "                                          comment_text_pre    target  \\\n",
       "516089   went no clue talk know democrat parti us well ...  0.000000   \n",
       "1693158  no obviou not understand nonn nativ land tacit...  0.000000   \n",
       "1331033  nk know next bomb last dont think guam key tar...  0.000000   \n",
       "1075854  point well taken margaret end spin polit desir...  0.000000   \n",
       "1316108  past belong one gender futur belong anoth conc...  0.166667   \n",
       "...                                                    ...       ...   \n",
       "1273859  cannot wait see portray mooch next new segment...  0.000000   \n",
       "1249009                    doubt probabl send ramadan card  0.000000   \n",
       "685573   bc hydro not widespread infrastructur back ga ...  0.000000   \n",
       "836980    time good old alaskan duct tape fix broken stuff  0.000000   \n",
       "435975                              everyon tri understand  0.000000   \n",
       "\n",
       "             class  \n",
       "516089   non-toxic  \n",
       "1693158  non-toxic  \n",
       "1331033  non-toxic  \n",
       "1075854  non-toxic  \n",
       "1316108  non-toxic  \n",
       "...            ...  \n",
       "1273859  non-toxic  \n",
       "1249009  non-toxic  \n",
       "685573   non-toxic  \n",
       "836980   non-toxic  \n",
       "435975   non-toxic  \n",
       "\n",
       "[100 rows x 4 columns]"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Train_Final_pre_text_data_sample_ML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "comment_text_ = [i for i in Final_pre_text_data_sample_ML['comment_text'].values]\n",
    "comment_text_pre_ = [i for i in Final_pre_text_data_sample_ML['comment_text_pre'].values]\n",
    "target_ = [i for i in Final_pre_text_data_sample_ML['target'].values]\n",
    "class_ = [i for i in Final_pre_text_data_sample_ML['class'].values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "comment_text_ :\n",
      " Wente has no clue what she is talking about. She knows the Democratic Party in the US about as well as my grandmother in Sicily knows how to fix a computer. \n",
      "\n",
      "It is actually astonishing that she can venture such a statement in a newspaper. I would bet $1000 that she would run out of meaningful things to say about the state of the Democratic Party in less than 1 minute.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " went no clue talk know democrat parti us well grandmoth sicili know fix comput actual astonish ventur statement newspap would bet would run meaning thing say state democrat parti less minut\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "******************************************************************************************************************************************************************************************************** \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " No, it's obvious she doesn't understand this at all. By being non-Native and on Native lands, she's tacitly endorsing and supporting colonialism and anti-Native harm. That she seeks to excuse herself from this makes her doubly culpable. She also seems to be engaging in blatantly racist behaviour against yet another group. This can't be tolerated in a modern, progressive society. This is 2017, not 1247.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " no obviou not understand nonn nativ land tacitli endors support coloni harm seek excus make doubli culpabl also seem engag blatantli racist behaviour yet anoth group cannot toler modern progress societi not\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "******************************************************************************************************************************************************************************************************** \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " If NK knows that their next bomb will be their last, I donʻt think Guam is the key target. I would think they would want to get the most bang for the buck. Itʻs going to be US soil.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " nk know next bomb last dont think guam key target would think would want get bang buck go us soil\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "******************************************************************************************************************************************************************************************************** \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " Point well taken Margaret.  It ends up being all about spin and the political desire to always paint men in a negative light.  A more honest examination of the data reveals balance; not unlike the spin surrounding income inequality\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " point well taken margaret end spin polit desir alway paint men neg light honest examin data reveal balanc not unlik spin surround incom inequ\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "******************************************************************************************************************************************************************************************************** \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " The past belonged to one gender.  Now the future belongs to another.  Both concepts are sexist.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " past belong one gender futur belong anoth concept sexist\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.16666666666666666\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "******************************************************************************************************************************************************************************************************** \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print('\\ncomment_text_ :\\n',comment_text_[i])\n",
    "    print('___'*30,'\\n')\n",
    "    \n",
    "    print('\\ncomment_text_pre_ :\\n',comment_text_pre_[i])\n",
    "    print('___'*30,'\\n')\n",
    "    \n",
    "    print('\\ntarget_ :',target_[i])\n",
    "    print('___'*30,'\\n')\n",
    "    \n",
    "    print('\\nclass_ :',class_[i],'\\n\\n')\n",
    "    print('**'*100,'\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## preprocessing_all_Train_data_ML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [02:15<00:00, 13309.42it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████| 1804874/1804874 [03:25<00:00, 8763.46it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:03<00:00, 546116.40it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████| 1804874/1804874 [05:54<00:00, 5092.01it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:13<00:00, 134629.28it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████| 1804874/1804874 [13:09<00:00, 2286.29it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:53<00:00, 33745.32it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:18<00:00, 98179.30it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:05<00:00, 336607.47it/s]\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_ML = preprocessing_ML(Data,submission_data=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1804874 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     object \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(5)\n",
      "memory usage: 647.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Saving File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_Final_pre_text_data_ML.to_csv('Train_Final_pre_text_data_ML_1.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Loding File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_Final_pre_text_data_ML = pd.read_csv(r'Train_Final_pre_text_data_ML_1.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1804874 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     object \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(5)\n",
      "memory usage: 647.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "> ####  Converting comment_text_pre from object data type to string data type "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_Final_pre_text_data_ML['comment_text_pre'] = Train_Final_pre_text_data_ML['comment_text_pre'].astype('string')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1804874 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     string \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(4), string(1)\n",
      "memory usage: 647.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## preprocessing_all_Test_data_ML (For Submission File)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_data =  pd.read_csv(r\"C:\\Users\\Lenovo\\Downloads\\Self case study 2\\Data\\jigsaw-unintended-bias-in-toxicity-classification\\test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 97320/97320 [00:06<00:00, 14723.68it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 97320/97320 [00:10<00:00, 8877.55it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████| 97320/97320 [00:00<00:00, 538738.36it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 97320/97320 [00:18<00:00, 5321.57it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████| 97320/97320 [00:00<00:00, 131650.47it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 97320/97320 [00:44<00:00, 2207.67it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 97320/97320 [00:02<00:00, 35976.01it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 97320/97320 [00:00<00:00, 97501.88it/s]\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_ML = preprocessing_ML(Test_data,submission_data=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 97320 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                97320 non-null  int64 \n",
      " 1   comment_text      97320 non-null  object\n",
      " 2   comment_text_pre  97320 non-null  object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 2.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Saving File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Final_pre_text_data_ML.to_csv('Test_Final_pre_text_data_ML_1.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Loding File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Final_pre_text_data_ML = pd.read_csv(r'Test_Final_pre_text_data_ML_1.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 97320 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                97320 non-null  int64 \n",
      " 1   comment_text      97320 non-null  object\n",
      " 2   comment_text_pre  96980 non-null  object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 2.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "> ####  Converting comment_text_pre from object data type to string data type "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Final_pre_text_data_ML['comment_text_pre'] = Test_Final_pre_text_data_ML['comment_text_pre'].astype('string')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 97320 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                97320 non-null  int64 \n",
      " 1   comment_text      97320 non-null  object\n",
      " 2   comment_text_pre  96980 non-null  string\n",
      "dtypes: int64(1), object(1), string(1)\n",
      "memory usage: 2.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing the data for DL models\n",
    "### which contain:\n",
    "    * preprocessing Train.csv\n",
    "    * preprocessing Test.csv (for submission file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## 1. Creating small dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 9115.28it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 5570.57it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 2687.43it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 100246.27it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 4359.58it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 2639.27it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 20045.42it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 50141.11it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████████| 100/100 [00:00<00:00, 50117.15it/s]\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_sample_DL = preprocessing_DL(sample_data,submission_data=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>severe_toxicity</th>\n",
       "      <th>obscene</th>\n",
       "      <th>identity_attack</th>\n",
       "      <th>insult</th>\n",
       "      <th>threat</th>\n",
       "      <th>asian</th>\n",
       "      <th>atheist</th>\n",
       "      <th>...</th>\n",
       "      <th>funny</th>\n",
       "      <th>wow</th>\n",
       "      <th>sad</th>\n",
       "      <th>likes</th>\n",
       "      <th>disagree</th>\n",
       "      <th>sexual_explicit</th>\n",
       "      <th>identity_annotator_count</th>\n",
       "      <th>toxicity_annotator_count</th>\n",
       "      <th>comment_text_pre</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>516089</th>\n",
       "      <td>875306</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Wente has no clue what she is talking about. S...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>wente no clue talking knows democratic party u...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1693158</th>\n",
       "      <td>6197921</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>No, it's obvious she doesn't understand this a...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>no obvious not understand nonnative native lan...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1331033</th>\n",
       "      <td>5741886</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>If NK knows that their next bomb will be their...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>nk knows next bomb last dont think guam key ta...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1075854</th>\n",
       "      <td>5431454</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Point well taken Margaret.  It ends up being a...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>point well taken margaret ends spin political ...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1316108</th>\n",
       "      <td>5723769</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>The past belonged to one gender.  Now the futu...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>past belonged one gender future belongs anothe...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1273859</th>\n",
       "      <td>5671279</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Can't wait to see who portrays the Mooch in th...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>cannot wait see portrays mooch next new segmen...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1249009</th>\n",
       "      <td>5641120</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>I doubt that too. He probably sends them ramad...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10</td>\n",
       "      <td>6</td>\n",
       "      <td>doubt probably sends ramadan cards</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685573</th>\n",
       "      <td>1080282</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>BC Hydro does not have the widespread infrastr...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>bc hydro not widespread infrastructure back ga...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>836980</th>\n",
       "      <td>5144827</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Time for some good old Alaskan duct tape to fi...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>time good old alaskan duct tape fix broken stuff</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>435975</th>\n",
       "      <td>777379</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Everyone should try to understand.</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>everyone try understand</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id    target                                       comment_text  \\\n",
       "516089    875306  0.000000  Wente has no clue what she is talking about. S...   \n",
       "1693158  6197921  0.000000  No, it's obvious she doesn't understand this a...   \n",
       "1331033  5741886  0.000000  If NK knows that their next bomb will be their...   \n",
       "1075854  5431454  0.000000  Point well taken Margaret.  It ends up being a...   \n",
       "1316108  5723769  0.166667  The past belonged to one gender.  Now the futu...   \n",
       "...          ...       ...                                                ...   \n",
       "1273859  5671279  0.000000  Can't wait to see who portrays the Mooch in th...   \n",
       "1249009  5641120  0.000000  I doubt that too. He probably sends them ramad...   \n",
       "685573   1080282  0.000000  BC Hydro does not have the widespread infrastr...   \n",
       "836980   5144827  0.000000  Time for some good old Alaskan duct tape to fi...   \n",
       "435975    777379  0.000000                 Everyone should try to understand.   \n",
       "\n",
       "         severe_toxicity  obscene  identity_attack    insult  threat  asian  \\\n",
       "516089               0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "1693158              0.0      0.0              0.0  0.000000     0.0    0.0   \n",
       "1331033              0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "1075854              0.0      0.0              0.0  0.000000     0.0    0.0   \n",
       "1316108              0.0      0.0              0.0  0.166667     0.0    0.0   \n",
       "...                  ...      ...              ...       ...     ...    ...   \n",
       "1273859              0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "1249009              0.0      0.0              0.0  0.000000     0.0    0.0   \n",
       "685573               0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "836980               0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "435975               0.0      0.0              0.0  0.000000     0.0    NaN   \n",
       "\n",
       "         atheist  ...  funny  wow  sad  likes  disagree  sexual_explicit  \\\n",
       "516089       NaN  ...      0    0    0      0         4         0.000000   \n",
       "1693158      0.0  ...      0    0    0      0         0         0.000000   \n",
       "1331033      NaN  ...      0    0    0      0         0         0.000000   \n",
       "1075854      0.0  ...      0    0    0      9         3         0.000000   \n",
       "1316108      0.0  ...      0    0    0      0         0         0.166667   \n",
       "...          ...  ...    ...  ...  ...    ...       ...              ...   \n",
       "1273859      NaN  ...      0    0    0      1         0         0.000000   \n",
       "1249009      0.0  ...      0    0    1      0         1         0.000000   \n",
       "685573       NaN  ...      0    0    0      1         0         0.000000   \n",
       "836980       NaN  ...      0    0    0      0         0         0.000000   \n",
       "435975       NaN  ...      0    0    0      0         0         0.000000   \n",
       "\n",
       "         identity_annotator_count  toxicity_annotator_count  \\\n",
       "516089                          0                         4   \n",
       "1693158                        10                         4   \n",
       "1331033                         0                         4   \n",
       "1075854                         6                         4   \n",
       "1316108                         4                         6   \n",
       "...                           ...                       ...   \n",
       "1273859                         0                         4   \n",
       "1249009                        10                         6   \n",
       "685573                          0                         4   \n",
       "836980                          0                         4   \n",
       "435975                          0                         4   \n",
       "\n",
       "                                          comment_text_pre      class  \n",
       "516089   wente no clue talking knows democratic party u...  non-toxic  \n",
       "1693158  no obvious not understand nonnative native lan...  non-toxic  \n",
       "1331033  nk knows next bomb last dont think guam key ta...  non-toxic  \n",
       "1075854  point well taken margaret ends spin political ...  non-toxic  \n",
       "1316108  past belonged one gender future belongs anothe...  non-toxic  \n",
       "...                                                    ...        ...  \n",
       "1273859  cannot wait see portrays mooch next new segmen...  non-toxic  \n",
       "1249009                 doubt probably sends ramadan cards  non-toxic  \n",
       "685573   bc hydro not widespread infrastructure back ga...  non-toxic  \n",
       "836980    time good old alaskan duct tape fix broken stuff  non-toxic  \n",
       "435975                             everyone try understand  non-toxic  \n",
       "\n",
       "[100 rows x 47 columns]"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Train_Final_pre_text_data_sample_DL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "comment_text_ = [i for i in Train_Final_pre_text_data_sample_DL['comment_text'].values]\n",
    "comment_text_pre_ = [i for i in Train_Final_pre_text_data_sample_DL['comment_text_pre'].values]\n",
    "target_ = [i for i in Train_Final_pre_text_data_sample_DL['target'].values]\n",
    "class_ = [i for i in Train_Final_pre_text_data_sample_DL['class'].values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "comment_text_ :\n",
      " Wente has no clue what she is talking about. She knows the Democratic Party in the US about as well as my grandmother in Sicily knows how to fix a computer. \n",
      "\n",
      "It is actually astonishing that she can venture such a statement in a newspaper. I would bet $1000 that she would run out of meaningful things to say about the state of the Democratic Party in less than 1 minute.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " wente no clue talking knows democratic party us well grandmother sicily knows fix computer actually astonishing venture statement newspaper would bet would run meaningful things say state democratic party less minute\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "************************************************************ \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " No, it's obvious she doesn't understand this at all. By being non-Native and on Native lands, she's tacitly endorsing and supporting colonialism and anti-Native harm. That she seeks to excuse herself from this makes her doubly culpable. She also seems to be engaging in blatantly racist behaviour against yet another group. This can't be tolerated in a modern, progressive society. This is 2017, not 1247.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " no obvious not understand nonnative native lands tacitly endorsing supporting colonialism harm seeks excuse makes doubly culpable also seems engaging blatantly racist behaviour yet another group cannot tolerated modern progressive society not\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "************************************************************ \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " If NK knows that their next bomb will be their last, I donʻt think Guam is the key target. I would think they would want to get the most bang for the buck. Itʻs going to be US soil.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " nk knows next bomb last dont think guam key target would think would want get bang buck going us soil\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "************************************************************ \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " Point well taken Margaret.  It ends up being all about spin and the political desire to always paint men in a negative light.  A more honest examination of the data reveals balance; not unlike the spin surrounding income inequality\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " point well taken margaret ends spin political desire always paint men negative light honest examination data reveals balance not unlike spin surrounding income inequality\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.0\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "************************************************************ \n",
      "\n",
      "\n",
      "comment_text_ :\n",
      " The past belonged to one gender.  Now the future belongs to another.  Both concepts are sexist.\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "comment_text_pre_ :\n",
      " past belonged one gender future belongs another concepts sexist\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "target_ : 0.16666666666666666\n",
      "__________________________________________________________________________________________ \n",
      "\n",
      "\n",
      "class_ : non-toxic \n",
      "\n",
      "\n",
      "************************************************************ \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(5):\n",
    "    print('\\ncomment_text_ :\\n',comment_text_[i])\n",
    "    print('___'*30,'\\n')\n",
    "    \n",
    "    print('\\ncomment_text_pre_ :\\n',comment_text_pre_[i])\n",
    "    print('___'*30,'\\n')\n",
    "    \n",
    "    print('\\ntarget_ :',target_[i])\n",
    "    print('___'*30,'\\n')\n",
    "    \n",
    "    print('\\nclass_ :',class_[i],'\\n\\n')\n",
    "    print('**'*30,'\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## preprocessing_all_Train_data_DL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [02:06<00:00, 14264.15it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████| 1804874/1804874 [03:28<00:00, 8640.49it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:07<00:00, 250529.64it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:03<00:00, 566058.34it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████| 1804874/1804874 [05:40<00:00, 5306.02it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:13<00:00, 133236.64it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [01:00<00:00, 29666.06it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████| 1804874/1804874 [00:19<00:00, 93294.73it/s]\n",
      "100%|███████████████████████████████████████████████████████████████████| 1804874/1804874 [00:01<00:00, 1363524.57it/s]\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL = preprocessing_DL(Data,submission_data=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1804874 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     object \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(5)\n",
      "memory usage: 647.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Saving File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_Final_pre_text_data_DL.to_csv('Train_Final_pre_text_data_DL_1.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### Loding file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>severe_toxicity</th>\n",
       "      <th>obscene</th>\n",
       "      <th>identity_attack</th>\n",
       "      <th>insult</th>\n",
       "      <th>threat</th>\n",
       "      <th>asian</th>\n",
       "      <th>atheist</th>\n",
       "      <th>...</th>\n",
       "      <th>funny</th>\n",
       "      <th>wow</th>\n",
       "      <th>sad</th>\n",
       "      <th>likes</th>\n",
       "      <th>disagree</th>\n",
       "      <th>sexual_explicit</th>\n",
       "      <th>identity_annotator_count</th>\n",
       "      <th>toxicity_annotator_count</th>\n",
       "      <th>comment_text_pre</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59848</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>This is so cool. It's like, 'would you want yo...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>cool like want mother read really great idea w...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59849</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Thank you!! This would make my life a lot less...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>thank would make life lot less keep not let an...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>59852</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>This is such an urgent design problem; kudos t...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>urgent design problem kudos taking impressive</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>59855</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Is this something I'll be able to install on m...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>something able install site releasing</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>59856</td>\n",
       "      <td>0.893617</td>\n",
       "      <td>haha you guys are a bunch of losers.</td>\n",
       "      <td>0.021277</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.021277</td>\n",
       "      <td>0.872340</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>47</td>\n",
       "      <td>haha guys bunch losers</td>\n",
       "      <td>toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1804869</th>\n",
       "      <td>6333967</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Maybe the tax on \"things\" would be collected w...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>maybe tax things would collected product impor...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1804870</th>\n",
       "      <td>6333969</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>What do you call people who STILL think the di...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>call people still think divine role creation</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1804871</th>\n",
       "      <td>6333982</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>thank you ,,,right or wrong,,, i am following ...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>thank right wrong following advice</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1804872</th>\n",
       "      <td>6334009</td>\n",
       "      <td>0.621212</td>\n",
       "      <td>Anyone who is quoted as having the following e...</td>\n",
       "      <td>0.030303</td>\n",
       "      <td>0.030303</td>\n",
       "      <td>0.045455</td>\n",
       "      <td>0.621212</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>66</td>\n",
       "      <td>anyone quoted following exchange even apocryph...</td>\n",
       "      <td>toxic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1804873</th>\n",
       "      <td>6334010</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>Students defined as EBD are legally just as di...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>students defined ebd legally disabled eligible...</td>\n",
       "      <td>non-toxic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1804874 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id    target                                       comment_text  \\\n",
       "0          59848  0.000000  This is so cool. It's like, 'would you want yo...   \n",
       "1          59849  0.000000  Thank you!! This would make my life a lot less...   \n",
       "2          59852  0.000000  This is such an urgent design problem; kudos t...   \n",
       "3          59855  0.000000  Is this something I'll be able to install on m...   \n",
       "4          59856  0.893617               haha you guys are a bunch of losers.   \n",
       "...          ...       ...                                                ...   \n",
       "1804869  6333967  0.000000  Maybe the tax on \"things\" would be collected w...   \n",
       "1804870  6333969  0.000000  What do you call people who STILL think the di...   \n",
       "1804871  6333982  0.000000  thank you ,,,right or wrong,,, i am following ...   \n",
       "1804872  6334009  0.621212  Anyone who is quoted as having the following e...   \n",
       "1804873  6334010  0.000000  Students defined as EBD are legally just as di...   \n",
       "\n",
       "         severe_toxicity   obscene  identity_attack    insult  threat  asian  \\\n",
       "0               0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "1               0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "2               0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "3               0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "4               0.021277  0.000000         0.021277  0.872340     0.0    0.0   \n",
       "...                  ...       ...              ...       ...     ...    ...   \n",
       "1804869         0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "1804870         0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "1804871         0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "1804872         0.030303  0.030303         0.045455  0.621212     0.0    NaN   \n",
       "1804873         0.000000  0.000000         0.000000  0.000000     0.0    NaN   \n",
       "\n",
       "         atheist  ...  funny  wow  sad  likes  disagree  sexual_explicit  \\\n",
       "0            NaN  ...      0    0    0      0         0              0.0   \n",
       "1            NaN  ...      0    0    0      0         0              0.0   \n",
       "2            NaN  ...      0    0    0      0         0              0.0   \n",
       "3            NaN  ...      0    0    0      0         0              0.0   \n",
       "4            0.0  ...      0    0    0      1         0              0.0   \n",
       "...          ...  ...    ...  ...  ...    ...       ...              ...   \n",
       "1804869      NaN  ...      0    0    0      0         0              0.0   \n",
       "1804870      NaN  ...      0    0    0      0         0              0.0   \n",
       "1804871      NaN  ...      0    0    0      0         0              0.0   \n",
       "1804872      NaN  ...      0    0    0      0         0              0.0   \n",
       "1804873      NaN  ...      0    0    0      0         0              0.0   \n",
       "\n",
       "         identity_annotator_count  toxicity_annotator_count  \\\n",
       "0                               0                         4   \n",
       "1                               0                         4   \n",
       "2                               0                         4   \n",
       "3                               0                         4   \n",
       "4                               4                        47   \n",
       "...                           ...                       ...   \n",
       "1804869                         0                         4   \n",
       "1804870                         0                         4   \n",
       "1804871                         0                         4   \n",
       "1804872                         0                        66   \n",
       "1804873                         0                         4   \n",
       "\n",
       "                                          comment_text_pre      class  \n",
       "0        cool like want mother read really great idea w...  non-toxic  \n",
       "1        thank would make life lot less keep not let an...  non-toxic  \n",
       "2            urgent design problem kudos taking impressive  non-toxic  \n",
       "3                    something able install site releasing  non-toxic  \n",
       "4                                   haha guys bunch losers      toxic  \n",
       "...                                                    ...        ...  \n",
       "1804869  maybe tax things would collected product impor...  non-toxic  \n",
       "1804870       call people still think divine role creation  non-toxic  \n",
       "1804871                 thank right wrong following advice  non-toxic  \n",
       "1804872  anyone quoted following exchange even apocryph...      toxic  \n",
       "1804873  students defined ebd legally disabled eligible...  non-toxic  \n",
       "\n",
       "[1804874 rows x 47 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL = pd.read_csv(r'Train_Final_pre_text_data_DL_1.csv')\n",
    "Train_Final_pre_text_data_DL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1804874 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     object \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(5)\n",
      "memory usage: 647.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Converting comment_text_pre from object data type to string data type "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "Train_Final_pre_text_data_DL['comment_text_pre'] = Train_Final_pre_text_data_DL['comment_text_pre'].astype('string')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1804874 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     string \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(4), string(1)\n",
      "memory usage: 647.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## preprocessing_all_Test_data_DL (For Submission File)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_data =  pd.read_csv(r\"C:\\Users\\Lenovo\\Downloads\\Self case study 2\\Data\\jigsaw-unintended-bias-in-toxicity-classification\\test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████████████████████████████████| 97320/97320 [00:06<00:00, 14731.40it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 97320/97320 [00:11<00:00, 8766.04it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████| 97320/97320 [00:00<00:00, 231568.73it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████| 97320/97320 [00:00<00:00, 524502.90it/s]\n",
      "100%|██████████████████████████████████████████████████████████████████████████| 97320/97320 [00:18<00:00, 5237.55it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████| 97320/97320 [00:00<00:00, 119366.13it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 97320/97320 [00:03<00:00, 30204.52it/s]\n",
      "100%|█████████████████████████████████████████████████████████████████████████| 97320/97320 [00:01<00:00, 93195.39it/s]\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_DL = preprocessing_DL(Test_data,submission_data=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 97320 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                97320 non-null  int64 \n",
      " 1   comment_text      97320 non-null  object\n",
      " 2   comment_text_pre  97320 non-null  object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 2.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_DL.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Saving File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Final_pre_text_data_DL.to_csv('Test_Final_pre_text_data_DL_1.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ####  Loding File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Final_pre_text_data_DL = pd.read_csv(r'Test_Final_pre_text_data_DL_1.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 97320 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                97320 non-null  int64 \n",
      " 1   comment_text      97320 non-null  object\n",
      " 2   comment_text_pre  96980 non-null  object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 2.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_DL.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "> ####  Converting comment_text_pre from object data type to string data type "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Final_pre_text_data_DL['comment_text_pre'] = Test_Final_pre_text_data_DL['comment_text_pre'].astype('string')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 97320 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                97320 non-null  int64 \n",
      " 1   comment_text      97320 non-null  object\n",
      " 2   comment_text_pre  96980 non-null  string\n",
      "dtypes: int64(1), object(1), string(1)\n",
      "memory usage: 2.2+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_DL.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ### Restart Kernel to clear RAM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ###  All about Data : There are Two data set train.csv and Test.csv\n",
    "* train.csv : we have to break train.csv into 3 part.(do it for all vectorizers)\n",
    "    * Train\n",
    "    * Test\n",
    "    * CV\n",
    "* Test.csv:  Submission File. so after finding good model we have to pass this data to best model. and submmit it on kaggle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6963"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL['comment_text_pre'].isnull().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6965"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Train_Final_pre_text_data_ML['comment_text_pre'].isnull().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Droping NAN values\n",
    "Train_Final_pre_text_data_ML.dropna(subset = [\"comment_text_pre\"],inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1797909 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     object \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(5)\n",
      "memory usage: 658.4+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(Train_Final_pre_text_data_ML['comment_text_pre'].values, Train_Final_pre_text_data_ML['class'].values, test_size=0.15,stratify=Train_Final_pre_text_data_ML['class'].values ,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_cv, y_train, y_cv = train_test_split(x_train,y_train,stratify=y_train, test_size=0.15, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Saving x_train,x_cv,x_test data into pickle file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pickle\n",
    "# with open('data_train_test_cv_csv_all_columns.pkl', 'wb') as file:\n",
    "#     pickle.dump([x_train, x_cv , x_test, y_train, y_cv, y_test], file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('data_train_test_cv_csv.pkl', 'wb') as file:\n",
    "    pickle.dump([x_train, x_cv , x_test, y_train, y_cv, y_test], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Loding data into pickel file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import pickle\n",
    "  \n",
    "# Open the file in binary mode\n",
    "with open('data_train_test_cv_csv.pkl', 'rb') as file:\n",
    "      \n",
    "    # Call load method to deserialze\n",
    "    x_train, x_cv , x_test, y_train, y_cv, y_test = pickle.load(file)\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Distribution of data in different Data sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'non-toxic': 1492439, 'toxic': 130173}"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train\n",
    "unique, counts = np.unique(y_train, return_counts=True)\n",
    "dict(zip(unique, counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'non-toxic': 78550, 'toxic': 6851}"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# CV\n",
    "unique, counts = np.unique(y_cv, return_counts=True)\n",
    "dict(zip(unique, counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'non-toxic': 82684, 'toxic': 7212}"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test\n",
    "unique, counts = np.unique(y_test, return_counts=True)\n",
    "dict(zip(unique, counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Data sets\n",
    "* x_train , y_train\n",
    "* x_cv    , y_cv\n",
    "* x_test  , y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. BOW:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "vectorizer_BOW= CountVectorizer(min_df=10, max_features=10000)\n",
    "vectorizer_BOW.fit(x_train)\n",
    "\n",
    "x_train_BOW = vectorizer_BOW.transform(x_train)\n",
    "x_cv_BOW = vectorizer_BOW.transform(x_cv)\n",
    "x_test_BOW = vectorizer_BOW.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vectorizer_BOW.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train_BOW: (1298988, 10000)\n",
      "x_cv_BOW: (229234, 10000)\n",
      "x_test_BOW: (269687, 10000)\n"
     ]
    }
   ],
   "source": [
    "print('x_train_BOW:',x_train_BOW.shape)\n",
    "print('x_cv_BOW:',x_cv_BOW.shape)\n",
    "print('x_test_BOW:',x_test_BOW.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here we are also applying vectorizer_BOW on Test.csv(submission file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 96980 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                96980 non-null  int64 \n",
      " 1   comment_text      96980 non-null  object\n",
      " 2   comment_text_pre  96980 non-null  string\n",
      "dtypes: int64(1), object(1), string(1)\n",
      "memory usage: 3.0+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_ML = pd.read_csv(r'Test_Final_pre_text_data_ML_1.csv')\n",
    "Test_Final_pre_text_data_ML['comment_text_pre'] = Test_Final_pre_text_data_ML['comment_text_pre'].astype('string')\n",
    "## Droping NAN values\n",
    "Test_Final_pre_text_data_ML.dropna(subset = [\"comment_text_pre\"],inplace=True)\n",
    "Test_Final_pre_text_data_ML.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Final_pre_text_data_ML_BOW = vectorizer_BOW.transform(Test_Final_pre_text_data_ML['comment_text_pre'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test_Final_pre_text_data_ML_BOW: (96980, 10000)\n"
     ]
    }
   ],
   "source": [
    "print('Test_Final_pre_text_data_ML_BOW:',Test_Final_pre_text_data_ML_BOW.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Saving BOW data into pickle file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('data_BOW.pkl', 'wb') as file:\n",
    "    pickle.dump([x_train_BOW , x_cv_BOW , x_test_BOW, y_train, y_cv, y_test,Test_Final_pre_text_data_ML_BOW], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Loding data into pickel file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import pickle\n",
    "  \n",
    "# Open the file in binary mode\n",
    "with open('data_BOW.pkl', 'rb') as file:\n",
    "      \n",
    "    # Call load method to deserialze\n",
    "    x_train_BOW , x_cv_BOW , x_test_BOW, y_train, y_cv, y_test,Test_Final_pre_text_data_ML_BOW = pickle.load(file)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. TF-IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "vectorizer_TFIDF = TfidfVectorizer(min_df=10, max_features=10000)\n",
    "\n",
    "vectorizer_TFIDF.fit(x_train)\n",
    "\n",
    "x_train_TFIDF = vectorizer_TFIDF.transform(x_train)\n",
    "x_cv_TFIDF = vectorizer_TFIDF.transform(x_cv)\n",
    "x_test_TFIDF = vectorizer_TFIDF.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train_TFIDF: (1298988, 10000)\n",
      "x_cv_TFIDF: (229234, 10000)\n",
      "x_test_TFIDF: (269687, 10000)\n"
     ]
    }
   ],
   "source": [
    "print('x_train_TFIDF:',x_train_TFIDF.shape)\n",
    "print('x_cv_TFIDF:',x_cv_TFIDF.shape)\n",
    "print('x_test_TFIDF:',x_test_TFIDF.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here we are also applying vectorizer_TFIDF on Test.csv(submission file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 96980 entries, 0 to 97319\n",
      "Data columns (total 3 columns):\n",
      " #   Column            Non-Null Count  Dtype \n",
      "---  ------            --------------  ----- \n",
      " 0   id                96980 non-null  int64 \n",
      " 1   comment_text      96980 non-null  object\n",
      " 2   comment_text_pre  96980 non-null  string\n",
      "dtypes: int64(1), object(1), string(1)\n",
      "memory usage: 3.0+ MB\n"
     ]
    }
   ],
   "source": [
    "Test_Final_pre_text_data_ML = pd.read_csv(r'Test_Final_pre_text_data_ML_1.csv')\n",
    "Test_Final_pre_text_data_ML['comment_text_pre'] = Test_Final_pre_text_data_ML['comment_text_pre'].astype('string')\n",
    "## Droping NAN values\n",
    "Test_Final_pre_text_data_ML.dropna(subset = [\"comment_text_pre\"],inplace=True)\n",
    "Test_Final_pre_text_data_ML.info()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Saving TFIDF data into pickle file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('data_TFIDF.pkl', 'wb') as file:\n",
    "    pickle.dump([x_train_TFIDF , x_cv_TFIDF , x_test_TFIDF, y_train, y_cv, y_test,Test_Final_pre_text_data_ML], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Loding data from pickel file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "  \n",
    "# Open the file in binary mode\n",
    "with open('data_TFIDF.pkl', 'rb') as file:\n",
    "      \n",
    "    # Call load method to deserialze\n",
    "    x_train_TFIDF , x_cv_TFIDF , x_test_TFIDF, y_train, y_cv, y_test,Test_Final_pre_text_data_ML = pickle.load(file)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Tokenizing data For DL "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6963"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL['comment_text_pre'].isnull().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Droping NAN values\n",
    "Train_Final_pre_text_data_DL.dropna(subset = [\"comment_text_pre\"],inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1797911 entries, 0 to 1804873\n",
      "Data columns (total 47 columns):\n",
      " #   Column                               Dtype  \n",
      "---  ------                               -----  \n",
      " 0   id                                   int64  \n",
      " 1   target                               float64\n",
      " 2   comment_text                         object \n",
      " 3   severe_toxicity                      float64\n",
      " 4   obscene                              float64\n",
      " 5   identity_attack                      float64\n",
      " 6   insult                               float64\n",
      " 7   threat                               float64\n",
      " 8   asian                                float64\n",
      " 9   atheist                              float64\n",
      " 10  bisexual                             float64\n",
      " 11  black                                float64\n",
      " 12  buddhist                             float64\n",
      " 13  christian                            float64\n",
      " 14  female                               float64\n",
      " 15  heterosexual                         float64\n",
      " 16  hindu                                float64\n",
      " 17  homosexual_gay_or_lesbian            float64\n",
      " 18  intellectual_or_learning_disability  float64\n",
      " 19  jewish                               float64\n",
      " 20  latino                               float64\n",
      " 21  male                                 float64\n",
      " 22  muslim                               float64\n",
      " 23  other_disability                     float64\n",
      " 24  other_gender                         float64\n",
      " 25  other_race_or_ethnicity              float64\n",
      " 26  other_religion                       float64\n",
      " 27  other_sexual_orientation             float64\n",
      " 28  physical_disability                  float64\n",
      " 29  psychiatric_or_mental_illness        float64\n",
      " 30  transgender                          float64\n",
      " 31  white                                float64\n",
      " 32  created_date                         object \n",
      " 33  publication_id                       int64  \n",
      " 34  parent_id                            float64\n",
      " 35  article_id                           int64  \n",
      " 36  rating                               object \n",
      " 37  funny                                int64  \n",
      " 38  wow                                  int64  \n",
      " 39  sad                                  int64  \n",
      " 40  likes                                int64  \n",
      " 41  disagree                             int64  \n",
      " 42  sexual_explicit                      float64\n",
      " 43  identity_annotator_count             int64  \n",
      " 44  toxicity_annotator_count             int64  \n",
      " 45  comment_text_pre                     object \n",
      " 46  class                                object \n",
      "dtypes: float64(32), int64(10), object(5)\n",
      "memory usage: 658.4+ MB\n"
     ]
    }
   ],
   "source": [
    "Train_Final_pre_text_data_DL.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "# https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(Train_Final_pre_text_data_DL['comment_text_pre'].values, Train_Final_pre_text_data_DL['class'].values, test_size=0.15,stratify=Train_Final_pre_text_data_DL['class'].values ,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_cv, y_train, y_cv = train_test_split(x_train,y_train,stratify=y_train, test_size=0.15, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Data sets\n",
    "* x_train , y_train\n",
    "* x_cv    , y_cv\n",
    "* x_test  , y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "PjFSlqosX53l"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> here i am tokenize data based on x_train (assign token to each unique word) and use same token to padd x_text to avoide data leakage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "YLcKFaYnX53l"
   },
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(oov_token=\"<OOV>\")\n",
    "tokenizer.fit_on_texts(x_train)\n",
    "word_index_train= tokenizer.word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "111590"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(word_index_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# x_test_submission_sequences = tokenizer.texts_to_sequences(x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> lets calculate maxlen for test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "lengths = [len(i.split()) for i in x_train]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1298990"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Its recommended that to keep maxlen as  90 percentile of total lengths of all texts. as we also saw in EDA ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://www.geeksforgeeks.org/numpy-percentile-in-python/\n",
    "max_length = np.percentile(lengths, 90)\n",
    "max_length = int(max_length)\n",
    "max_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Quick access to Test Data (Submission File)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "Test_Submission = pd.read_csv(r'Test_Final_pre_text_data_DL_1.csv')\n",
    "Test_Submission['comment_text_pre'] = Test_Submission['comment_text_pre'].astype('string')\n",
    "## Droping NAN values\n",
    "Test_Submission.dropna(subset = [\"comment_text_pre\"],inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# len(x_train_sequences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<StringArray>\n",
       "['integrity means pay debts apply president trump']\n",
       "Length: 1, dtype: string"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Test_Submission['comment_text_pre'].values[:1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zFEcb8waX53l",
    "outputId": "292ddc39-de9f-4337-f0f1-56e54f4e75aa"
   },
   "outputs": [],
   "source": [
    "x_train_sequences = tokenizer.texts_to_sequences(x_train)\n",
    "x_cv_sequences = tokenizer.texts_to_sequences(x_cv)\n",
    "x_test_sequences = tokenizer.texts_to_sequences(x_test)\n",
    "Test_Submission_sequences = tokenizer.texts_to_sequences(Test_Submission['comment_text_pre'].values)\n",
    "\n",
    "x_train_padding= pad_sequences(x_train_sequences,padding='pre',maxlen=max_length)\n",
    "x_cv_padding= pad_sequences(x_cv_sequences,padding='pre',maxlen=max_length)\n",
    "x_test_padding= pad_sequences(x_test_sequences,padding='pre',maxlen=max_length)\n",
    "Test_Submission_padding= pad_sequences(Test_Submission_sequences,padding='pre',maxlen=max_length)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train_padding: (1298990, 65)\n",
      "x_cv_padding: (229234, 65)\n",
      "x_test_padding: (269687, 65)\n",
      "Test_Submission_padding: (96980, 65)\n"
     ]
    }
   ],
   "source": [
    "print('x_train_padding:',x_train_padding.shape)\n",
    "print('x_cv_padding:',x_cv_padding.shape)\n",
    "print('x_test_padding:',x_test_padding.shape)\n",
    "print('Test_Submission_padding:',Test_Submission_padding.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Saving Tokeniz data into pickle file "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Pb37y-13bwXm"
   },
   "source": [
    "\n",
    "* here we are using Pretrained Glove vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open('data_Tokeniz.pkl', 'wb') as file:\n",
    "    pickle.dump([x_train_padding , x_cv_padding , x_test_padding, y_train, y_cv, y_test,Test_Submission_padding], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Loding data from pickel file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import pickle\n",
    "  \n",
    "# Open the file in binary mode\n",
    "with open('data_Tokeniz.pkl', 'rb') as file:\n",
    "      \n",
    "    # Call load method to deserialze\n",
    "    x_train_padding , x_cv_padding , x_test_padding, y_train, y_cv, y_test,Test_Submission_padding = pickle.load(file)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ## Files that we got after Preprocessing:\n",
    "\n",
    "* Preprocessing Files:\n",
    "\n",
    "    * Train_Final_pre_text_data_ML : Train data for ML models\n",
    "    * Test_Final_pre_text_data_ML : Test (Submission) Data \n",
    "    * Train_Final_pre_text_data_DL : Train data for DL models\n",
    "    * Test_Final_pre_text_data_DL : Test (Submission) Data\n",
    "    \n",
    "* Numeric Data (After Feature Engineering):  \n",
    "\n",
    "    * BOW : \n",
    "        * x_train_BOW ,y_train\n",
    "        * x_cv_BOW , y_cv\n",
    "        * x_test_BOW ,y_test\n",
    "        * Test_Final_pre_text_data_ML_BOW (Test Submission File data )\n",
    "        \n",
    "     * TFIDF :\n",
    "        * x_train_TFIDF ,y_train\n",
    "        * x_cv_TFIDF , y_cv\n",
    "        * x_test_TFIDF ,y_test\n",
    "        * Test_Final_pre_text_data_ML_TFIDF (Test Submission File data)\n",
    "     \n",
    "     * Tokanizer:\n",
    "        * x_train_padding\n",
    "        * x_test_padding\n",
    "        * x_cv_padding\n",
    "        * Test_Submission_padding  (Test Submission File data )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> ##  End of Preprocessing and Feature Engineering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Refrences:\n",
    "* https://www.kaggle.com/sudalairajkumar/getting-started-with-text-preprocessing#Stemming\n",
    "* https://www.kaggle.com/christofhenkel/how-to-preprocessing-for-glove-part1-eda\n",
    "* https://www.guru99.com/stemming-lemmatization-python-nltk.html\n",
    "* https://www.datacamp.com/community/tutorials/markdown-in-jupyter-notebook?utm_source=adwords_ppc&utm_medium=cpc&utm_campaignid=1455363063&utm_adgroupid=65083631748&utm_device=c&utm_keyword=&utm_matchtype=&utm_network=g&utm_adpostion=&utm_creative=332602034364&utm_targetid=aud-299261629574:dsa-429603003980&utm_loc_interest_ms=&utm_loc_physical_ms=9040235&gclid=Cj0KCQiA09eQBhCxARIsAAYRiymKqU2B4rJ-VZgZ8aRntOfA0RQfMCobAyeRTeD2Xxg49PFgs5Smt8saAvv-EALw_wcB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}