Web Scraping Topics/Recognizing Captcha/solving captcha with ocr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Crack captcha with OCR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the captchas(run this only once)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "import random\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "user_agent = \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "response = requests.get(\"https://captcha.com/captcha-examples.html?cst=corg\",headers = {\"USER-AGENT\":user_agent})\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "soup = BeautifulSoup(response.text,\"lxml\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "base_url = \"https://captcha.com/\"\n",
    "image_urls = [base_url + x[\"src\"] for x in soup.select(\".captcha_sample\")]\n",
    "image_captcha_type = [x[\"alt\"].split(\" \")[2] for x in soup.select(\".captcha_sample\")]\n",
    "df = pd.DataFrame({\"captcha_type\":image_captcha_type,\n",
    "                 \"captcha_url\": image_urls})\n",
    "df[\"local_path\"] = np.nan\n",
    "df[\"text\"] = np.nan\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# I already generated a captchas folder\n",
    "for index,row in df.iterrows():\n",
    "    response = requests.get(row[\"captcha_url\"],headers = {\"USER-AGENT\":user_agent})\n",
    "    local_path = f\"captchas/{row['captcha_type']}.jpg\"\n",
    "    row[\"local_path\"] = local_path\n",
    "    with open(local_path,\"wb\") as f:\n",
    "        f.write(response.content)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label them manually(By modifying the file name)\n",
    "\n",
    "original file name - text.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load all information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_path = [\"captchas/\" + x for x in os.listdir(\"captchas\")]\n",
    "images_df = pd.DataFrame({\"image_path\":image_path})\n",
    "images_df[\"captcha_type\"] = [x.split(\"-\")[0] for x in os.listdir(\"captchas\")]\n",
    "images_df[\"text\"] = [x.split(\"-\")[1].split(\".\")[0] for x in os.listdir(\"captchas\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OCR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## No Pre Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OCR - pytesseract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import pytesseract\n",
    "pytesseract.pytesseract.tesseract_cmd = r'D:\\Working_Space\\Tesseract-OCR\\tesseract'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"pytesseract_no\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    text = pytesseract.image_to_string(Image.open(row[\"image_path\"])).strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"pytesseract_no\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"pytesseract_no\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"pytesseract_no\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All failed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Cloud Vision API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import os\n",
    "\n",
    "# Imports the Google Cloud client library\n",
    "from google.cloud import vision\n",
    "from google.cloud.vision import types\n",
    "\n",
    "# Instantiates a client\n",
    "client = vision.ImageAnnotatorClient.from_service_account_json(\"credential/cloud-vision-key.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detect_text(path):\n",
    "    image_PIl = Image.open(path)\n",
    "    with io.open(path, 'rb') as image_file:\n",
    "        content = image_file.read()\n",
    "\n",
    "    image = vision.types.Image(content=content)\n",
    "    image_context = vision.types.ImageContext(language_hints=[\"en\",\"zh\"])\n",
    "\n",
    "    response = client.text_detection(image=image,image_context=image_context)\n",
    "    texts = response.text_annotations\n",
    "    if len(texts) == 0:\n",
    "        return(\"\")\n",
    "    else:\n",
    "        return(texts[0].description.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"api_no\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    text = detect_text(row[\"image_path\"]).strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"api_no\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"api_no\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06666666666666667"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"api_no\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "About 7% Success Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing -gray,remove noise, Gaussian blur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OCR - pytesseract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"pytesseract_1\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.GaussianBlur(img, (5, 5), 0)\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = pytesseract.image_to_string(Image.open(\"temp.jpg\")).strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"pytesseract_1\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"pytesseract_1\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"pytesseract_1\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Failed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Cloud Vision API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"api_1\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.GaussianBlur(img, (5, 5), 0)\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = detect_text(\"temp.jpg\").strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"api_1\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"api_1\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"api_1\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10% success rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing --gray,remove noise,Gaussian blur,binary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OCR - pytesseract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"pytesseract_2\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.GaussianBlur(img, (5, 5), 0)\n",
    "    # Apply threshold to get image with only b&w (binarization)\n",
    "    img = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = pytesseract.image_to_string(Image.open(\"temp.jpg\")).strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"pytesseract_2\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"pytesseract_2\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.016666666666666666"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"pytesseract_2\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Success Rate 2%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Cloud Vision API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"api_2\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.GaussianBlur(img, (5, 5), 0)\n",
    "    # Apply threshold to get image with only b&w (binarization)\n",
    "    img = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = detect_text(\"temp.jpg\").strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"api_2\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"api_2\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.05"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"api_2\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Success Rate Dropped to 5%. It seems like binarization doesn't work well for google cloud vision api."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reference:\n",
    "1. https://medium.freecodecamp.org/getting-started-with-tesseract-part-i-2a6a6b1cf75e\n",
    "2. https://medium.freecodecamp.org/getting-started-with-tesseract-part-ii-f7f9a0899b3f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proprocessing - Median blurring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OCR - pytesseract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"pytesseract_3\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.medianBlur(img, 3)\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = pytesseract.image_to_string(Image.open(\"temp.jpg\")).strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"pytesseract_3\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"pytesseract_3\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.016666666666666666"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"pytesseract_3\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2%"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Cloud Vision API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"api_3\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.medianBlur(img, 3)\n",
    "    # Apply threshold to get image with only b&w (binarization)\n",
    "    img = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = detect_text(\"temp.jpg\").strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"api_3\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"api_3\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"api_3\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10%, this blur method is similar to gaussian"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Proprocessing - Bilateral  blurring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OCR - pytesseract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"pytesseract_4\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.bilateralFilter(img,9,75,75)\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = pytesseract.image_to_string(Image.open(\"temp.jpg\")).strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"pytesseract_4\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"pytesseract_4\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.016666666666666666"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"pytesseract_4\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Cloud Vision API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"api_4\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.bilateralFilter(img,9,75,75)\n",
    "    # Apply threshold to get image with only b&w (binarization)\n",
    "    img = cv2.threshold(img, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = detect_text(\"temp.jpg\").strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"api_4\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"api_4\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"api_4\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Adaptive - Simple Thresholds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OCR - pytesseract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"pytesseract_5\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.bilateralFilter(img,9,75,75)\n",
    "    # Apply threshold to get image with only b&w (binarization)\n",
    "    img = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 31, 2)\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = pytesseract.image_to_string(Image.open(\"temp.jpg\")).strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"pytesseract_5\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"pytesseract_5\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"pytesseract_5\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Google Cloud Vision API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_df[\"api_5\"] = np.nan\n",
    "for index,row in images_df.iterrows():\n",
    "    #read image\n",
    "    img = cv2.imread(row[\"image_path\"])\n",
    "    # Convert to gray\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
    "    # Apply dilation and erosion to remove some noise\n",
    "    kernel = np.ones((1, 1), np.uint8)\n",
    "    img = cv2.dilate(img, kernel, iterations=1)\n",
    "    img = cv2.erode(img, kernel, iterations=1)\n",
    "    # Apply blur to smooth out the edges\n",
    "    img = cv2.bilateralFilter(img,9,75,75)\n",
    "    # Apply threshold to get image with only b&w (binarization)\n",
    "    img = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 31, 2)\n",
    "    #write to temp file\n",
    "    cv2.imwrite(\"temp.jpg\",img)\n",
    "    text = detect_text(\"temp.jpg\").strip()\n",
    "    if text == row[\"text\"]:\n",
    "        images_df.at[index,\"api_5\"] = int(1)\n",
    "    else:\n",
    "        images_df.at[index,\"api_5\"] = int(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.03333333333333333"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images_df[\"api_5\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion:\n",
    "\n",
    "It's hard to recognize captcha with OCR or text recgonizing services since captcha is designed to not be able to be recognized by machines. The highest success rate is 10% using google cloud vision(text recognition) services.\n",
    "\n",
    "However, if we really need to solve a kind of captcha, we can use captcha solving services(peole mannual sovling captcha for us) or machine learning after we collect enough samples."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}