Analise_de_Padroes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Curso de Auditoria de Dados</center>\n",
    "## <center>Flávio Brito</center>\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<H1>Análise de Padrões</H1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "O objetivo deste estudo é apresentar algumas técnicas para análise de padrões em dados. Utilizaremos a linguagem R e seus pacotes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos utilizar o pacote bpa - Basic Pattern Analysis. Este pacote possui funções especiais em análise de padrões em dados. \n",
    "É um conjunto de ferraamentas para pré-processamento de dados que ajuda a economizar linhas e linhas de código em busca de padrões de dados sejam conjuntos de caracteres, números ou combinações de números e caracteres. É util para limpeza de dados e estudo de padrões e anomalias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "package 'bpa' successfully unpacked and MD5 sums checked\n",
      "\n",
      "The downloaded binary packages are in\n",
      "\tC:\\Users\\flavi\\AppData\\Local\\Temp\\RtmpAzlaRF\\downloaded_packages\n"
     ]
    }
   ],
   "source": [
    "install.packages(\"bpa\", repos = \"https://cloud.r-project.org\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Carregando a biblioteca bpa  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "library(bpa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Carregando os dados de exemplo do pacote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data(messy, package = \"bpa\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>Gender</th><th scope=col>Date</th><th scope=col>Phone</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>Female     </td><td>16Aug2001  </td><td>571 972 510</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>Male       </td><td>2009-08-21 </td><td>090 875 285</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>M            </td><td>01/22/2004   </td><td>(079)-200-150</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>male         </td><td>09/02/2008   </td><td>(129)-006-808</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>Male            </td><td>February 19 2006</td><td>016 651 851     </td></tr>\n",
       "\t<tr><th scope=row>6</th><td>Female       </td><td>16Nov2015    </td><td>(492)-455-108</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       "  & Gender & Date & Phone\\\\\n",
       "\\hline\n",
       "\t1 & Female      & 16Aug2001   & 571 972 510\\\\\n",
       "\t2 & Male        & 2009-08-21  & 090 875 285\\\\\n",
       "\t3 & M             & 01/22/2004    & (079)-200-150\\\\\n",
       "\t4 & male          & 09/02/2008    & (129)-006-808\\\\\n",
       "\t5 & Male             & February 19 2006 & 016 651 851     \\\\\n",
       "\t6 & Female        & 16Nov2015     & (492)-455-108\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "  Gender             Date         Phone\n",
       "1 Female        16Aug2001   571 972 510\n",
       "2   Male       2009-08-21   090 875 285\n",
       "3      M       01/22/2004 (079)-200-150\n",
       "4   male       09/02/2008 (129)-006-808\n",
       "5   Male February 19 2006   016 651 851\n",
       "6 Female        16Nov2015 (492)-455-108"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(messy)\n",
    "attach(messy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pode-se perceber imediatamente que os valores possuem formatos mistos. Isto é muito comum quando os dados são inseridos manualmente dentro de um sistema e por não ter uma crística de dentrada acabam sendo aceitos pelo sistema transacional, dificultando assim a padronização dos mesmos."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilização básica"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"(999)-999-999\""
      ],
      "text/latex": [
       "\"(999)-999-999\""
      ],
      "text/markdown": [
       "\"(999)-999-999\""
      ],
      "text/plain": [
       "[1] \"(999)-999-999\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "get_pattern(\"(079)-200-150\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"999w999w999\""
      ],
      "text/latex": [
       "\"999w999w999\""
      ],
      "text/markdown": [
       "\"999w999w999\""
      ],
      "text/plain": [
       "[1] \"999w999w999\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "get_pattern(\"016 651 851\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"999&lt;BRANCO&gt;999&lt;BRANCO&gt;999\""
      ],
      "text/latex": [
       "\"999<BRANCO>999<BRANCO>999\""
      ],
      "text/markdown": [
       "\"999&lt;BRANCO&gt;999&lt;BRANCO&gt;999\""
      ],
      "text/plain": [
       "[1] \"999<BRANCO>999<BRANCO>999\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    " get_pattern(\"016 651 851\", ws_char = \"<BRANCO>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Buscando padrões dentro de um Data Frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Agora examinemos algumas variáveis, buscando não só descobrir os padrões como também a frequência de cada um na variável. \n",
    "Voltemos aos dados. Busquemos padrões na variável Date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>.</th><th scope=col>Freq</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>99/99/9999</td><td>259       </td></tr>\n",
       "\t<tr><th scope=row>2</th><td>9999-99-99</td><td>262       </td></tr>\n",
       "\t<tr><th scope=row>3</th><td>99Aaa9999</td><td>241      </td></tr>\n",
       "\t<tr><th scope=row>4</th><td>Aaaaaaaaaw99w9999</td><td>19               </td></tr>\n",
       "\t<tr><th scope=row>5</th><td>Aaaaaaaaw99w9999</td><td>56              </td></tr>\n",
       "\t<tr><th scope=row>6</th><td>Aaaaaaaw99w9999</td><td>45             </td></tr>\n",
       "\t<tr><th scope=row>7</th><td>Aaaaaaw99w9999</td><td>24            </td></tr>\n",
       "\t<tr><th scope=row>8</th><td>Aaaaaw99w9999</td><td>36           </td></tr>\n",
       "\t<tr><th scope=row>9</th><td>Aaaaw99w9999</td><td>42          </td></tr>\n",
       "\t<tr><th scope=row>10</th><td>Aaaw99w9999</td><td>16         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       "  & . & Freq\\\\\n",
       "\\hline\n",
       "\t1 & 99/99/9999 & 259       \\\\\n",
       "\t2 & 9999-99-99 & 262       \\\\\n",
       "\t3 & 99Aaa9999 & 241      \\\\\n",
       "\t4 & Aaaaaaaaaw99w9999 & 19               \\\\\n",
       "\t5 & Aaaaaaaaw99w9999 & 56              \\\\\n",
       "\t6 & Aaaaaaaw99w9999 & 45             \\\\\n",
       "\t7 & Aaaaaaw99w9999 & 24            \\\\\n",
       "\t8 & Aaaaaw99w9999 & 36           \\\\\n",
       "\t9 & Aaaaw99w9999 & 42          \\\\\n",
       "\t10 & Aaaw99w9999 & 16         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "                   . Freq\n",
       "1         99/99/9999  259\n",
       "2         9999-99-99  262\n",
       "3          99Aaa9999  241\n",
       "4  Aaaaaaaaaw99w9999   19\n",
       "5   Aaaaaaaaw99w9999   56\n",
       "6    Aaaaaaaw99w9999   45\n",
       "7     Aaaaaaw99w9999   24\n",
       "8      Aaaaaw99w9999   36\n",
       "9       Aaaaw99w9999   42\n",
       "10       Aaaw99w9999   16"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messy$Date %>%\n",
    "  get_pattern %>%  # extrai o padrão\n",
    "  table %>%        # tabula as fraquências\n",
    "  as.data.frame    # apresenta o resultado como um data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perceba que temos 259 registros no campo Date com formato 99/99/9999 (Ex:01/22/2004) e 241 com formato 99Aaa9999(Ex:16Aug2001) , por exemplo. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analisando os dados da variável alvo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>\"16Aug2001\"</li>\n",
       "\t<li>\"2009-08-21\"</li>\n",
       "\t<li>\"01/22/2004\"</li>\n",
       "\t<li>\"09/02/2008\"</li>\n",
       "\t<li>\"February 19 2006\"</li>\n",
       "\t<li>\"16Nov2015\"</li>\n",
       "\t<li>\"2004-07-12\"</li>\n",
       "\t<li>\"24Sep2013\"</li>\n",
       "\t<li>\"November 01 2007\"</li>\n",
       "\t<li>\"2007-08-15\"</li>\n",
       "\t<li>\"August 22 2009\"</li>\n",
       "\t<li>\"2003-05-29\"</li>\n",
       "\t<li>\"2011-04-02\"</li>\n",
       "\t<li>\"October 29 2004\"</li>\n",
       "\t<li>\"2004-09-13\"</li>\n",
       "\t<li>\"25Nov2009\"</li>\n",
       "\t<li>\"02/12/2010\"</li>\n",
       "\t<li>\"23Jan2008\"</li>\n",
       "\t<li>\"06/30/2006\"</li>\n",
       "\t<li>\"2009-07-24\"</li>\n",
       "\t<li>\"December 02 2013\"</li>\n",
       "\t<li>\"14Jun2014\"</li>\n",
       "\t<li>\"2009-04-19\"</li>\n",
       "\t<li>\"09/03/2009\"</li>\n",
       "\t<li>\"November 25 2007\"</li>\n",
       "\t<li>\"18Jan2015\"</li>\n",
       "\t<li>\"2009-03-05\"</li>\n",
       "\t<li>\"10/24/2006\"</li>\n",
       "\t<li>\"09/09/2008\"</li>\n",
       "\t<li>\"September 08 2012\"</li>\n",
       "\t<li>\"12/31/2011\"</li>\n",
       "\t<li>\"11/12/2009\"</li>\n",
       "\t<li>\"09/23/2002\"</li>\n",
       "\t<li>\"10/09/2002\"</li>\n",
       "\t<li>\"2004-12-11\"</li>\n",
       "\t<li>\"2001-08-23\"</li>\n",
       "\t<li>\"02/09/2004\"</li>\n",
       "\t<li>\"November 02 2010\"</li>\n",
       "\t<li>\"03/14/2007\"</li>\n",
       "\t<li>\"24Dec2004\"</li>\n",
       "\t<li>\"March 10 2004\"</li>\n",
       "\t<li>\"November 05 2012\"</li>\n",
       "\t<li>\"2010-02-11\"</li>\n",
       "\t<li>\"12/25/2009\"</li>\n",
       "\t<li>\"October 25 2007\"</li>\n",
       "\t<li>\"2011-07-22\"</li>\n",
       "\t<li>\"April 01 2014\"</li>\n",
       "\t<li>\"18Mar2009\"</li>\n",
       "\t<li>\"06Feb2015\"</li>\n",
       "\t<li>\"2015-12-10\"</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item \"16Aug2001\"\n",
       "\\item \"2009-08-21\"\n",
       "\\item \"01/22/2004\"\n",
       "\\item \"09/02/2008\"\n",
       "\\item \"February 19 2006\"\n",
       "\\item \"16Nov2015\"\n",
       "\\item \"2004-07-12\"\n",
       "\\item \"24Sep2013\"\n",
       "\\item \"November 01 2007\"\n",
       "\\item \"2007-08-15\"\n",
       "\\item \"August 22 2009\"\n",
       "\\item \"2003-05-29\"\n",
       "\\item \"2011-04-02\"\n",
       "\\item \"October 29 2004\"\n",
       "\\item \"2004-09-13\"\n",
       "\\item \"25Nov2009\"\n",
       "\\item \"02/12/2010\"\n",
       "\\item \"23Jan2008\"\n",
       "\\item \"06/30/2006\"\n",
       "\\item \"2009-07-24\"\n",
       "\\item \"December 02 2013\"\n",
       "\\item \"14Jun2014\"\n",
       "\\item \"2009-04-19\"\n",
       "\\item \"09/03/2009\"\n",
       "\\item \"November 25 2007\"\n",
       "\\item \"18Jan2015\"\n",
       "\\item \"2009-03-05\"\n",
       "\\item \"10/24/2006\"\n",
       "\\item \"09/09/2008\"\n",
       "\\item \"September 08 2012\"\n",
       "\\item \"12/31/2011\"\n",
       "\\item \"11/12/2009\"\n",
       "\\item \"09/23/2002\"\n",
       "\\item \"10/09/2002\"\n",
       "\\item \"2004-12-11\"\n",
       "\\item \"2001-08-23\"\n",
       "\\item \"02/09/2004\"\n",
       "\\item \"November 02 2010\"\n",
       "\\item \"03/14/2007\"\n",
       "\\item \"24Dec2004\"\n",
       "\\item \"March 10 2004\"\n",
       "\\item \"November 05 2012\"\n",
       "\\item \"2010-02-11\"\n",
       "\\item \"12/25/2009\"\n",
       "\\item \"October 25 2007\"\n",
       "\\item \"2011-07-22\"\n",
       "\\item \"April 01 2014\"\n",
       "\\item \"18Mar2009\"\n",
       "\\item \"06Feb2015\"\n",
       "\\item \"2015-12-10\"\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. \"16Aug2001\"\n",
       "2. \"2009-08-21\"\n",
       "3. \"01/22/2004\"\n",
       "4. \"09/02/2008\"\n",
       "5. \"February 19 2006\"\n",
       "6. \"16Nov2015\"\n",
       "7. \"2004-07-12\"\n",
       "8. \"24Sep2013\"\n",
       "9. \"November 01 2007\"\n",
       "10. \"2007-08-15\"\n",
       "11. \"August 22 2009\"\n",
       "12. \"2003-05-29\"\n",
       "13. \"2011-04-02\"\n",
       "14. \"October 29 2004\"\n",
       "15. \"2004-09-13\"\n",
       "16. \"25Nov2009\"\n",
       "17. \"02/12/2010\"\n",
       "18. \"23Jan2008\"\n",
       "19. \"06/30/2006\"\n",
       "20. \"2009-07-24\"\n",
       "21. \"December 02 2013\"\n",
       "22. \"14Jun2014\"\n",
       "23. \"2009-04-19\"\n",
       "24. \"09/03/2009\"\n",
       "25. \"November 25 2007\"\n",
       "26. \"18Jan2015\"\n",
       "27. \"2009-03-05\"\n",
       "28. \"10/24/2006\"\n",
       "29. \"09/09/2008\"\n",
       "30. \"September 08 2012\"\n",
       "31. \"12/31/2011\"\n",
       "32. \"11/12/2009\"\n",
       "33. \"09/23/2002\"\n",
       "34. \"10/09/2002\"\n",
       "35. \"2004-12-11\"\n",
       "36. \"2001-08-23\"\n",
       "37. \"02/09/2004\"\n",
       "38. \"November 02 2010\"\n",
       "39. \"03/14/2007\"\n",
       "40. \"24Dec2004\"\n",
       "41. \"March 10 2004\"\n",
       "42. \"November 05 2012\"\n",
       "43. \"2010-02-11\"\n",
       "44. \"12/25/2009\"\n",
       "45. \"October 25 2007\"\n",
       "46. \"2011-07-22\"\n",
       "47. \"April 01 2014\"\n",
       "48. \"18Mar2009\"\n",
       "49. \"06Feb2015\"\n",
       "50. \"2015-12-10\"\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       " [1] \"16Aug2001\"         \"2009-08-21\"        \"01/22/2004\"       \n",
       " [4] \"09/02/2008\"        \"February 19 2006\"  \"16Nov2015\"        \n",
       " [7] \"2004-07-12\"        \"24Sep2013\"         \"November 01 2007\" \n",
       "[10] \"2007-08-15\"        \"August 22 2009\"    \"2003-05-29\"       \n",
       "[13] \"2011-04-02\"        \"October 29 2004\"   \"2004-09-13\"       \n",
       "[16] \"25Nov2009\"         \"02/12/2010\"        \"23Jan2008\"        \n",
       "[19] \"06/30/2006\"        \"2009-07-24\"        \"December 02 2013\" \n",
       "[22] \"14Jun2014\"         \"2009-04-19\"        \"09/03/2009\"       \n",
       "[25] \"November 25 2007\"  \"18Jan2015\"         \"2009-03-05\"       \n",
       "[28] \"10/24/2006\"        \"09/09/2008\"        \"September 08 2012\"\n",
       "[31] \"12/31/2011\"        \"11/12/2009\"        \"09/23/2002\"       \n",
       "[34] \"10/09/2002\"        \"2004-12-11\"        \"2001-08-23\"       \n",
       "[37] \"02/09/2004\"        \"November 02 2010\"  \"03/14/2007\"       \n",
       "[40] \"24Dec2004\"         \"March 10 2004\"     \"November 05 2012\" \n",
       "[43] \"2010-02-11\"        \"12/25/2009\"        \"October 25 2007\"  \n",
       "[46] \"2011-07-22\"        \"April 01 2014\"     \"18Mar2009\"        \n",
       "[49] \"06Feb2015\"         \"2015-12-10\"       "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messy$Date %>%\n",
    "  unique %>%    # extract unique values\n",
    "  head(50)      # look at first 50 observations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Padronizar os dados através de basic pattern analysis fornece uma representação muito mais clara a respeito dos dados \n",
    "que são frequentemente mais usados durante a etapa de pré-processamento"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analisando todas as variáveis do Data frame de uma só vez"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>Gender</th><th scope=col>Date</th><th scope=col>Phone</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>Aaaaaa     </td><td>99Aaa9999  </td><td>999w999w999</td></tr>\n",
       "\t<tr><th scope=row>2</th><td>Aaaa       </td><td>9999-99-99 </td><td>999w999w999</td></tr>\n",
       "\t<tr><th scope=row>3</th><td>A            </td><td>99/99/9999   </td><td>(999)-999-999</td></tr>\n",
       "\t<tr><th scope=row>4</th><td>aaaa         </td><td>99/99/9999   </td><td>(999)-999-999</td></tr>\n",
       "\t<tr><th scope=row>5</th><td>Aaaa            </td><td>Aaaaaaaaw99w9999</td><td>999w999w999     </td></tr>\n",
       "\t<tr><th scope=row>6</th><td>Aaaaaa       </td><td>99Aaa9999    </td><td>(999)-999-999</td></tr>\n",
       "\t<tr><th scope=row>7</th><td>Aaaaaa       </td><td>9999-99-99   </td><td>(999)w999w999</td></tr>\n",
       "\t<tr><th scope=row>8</th><td>Aaaaaa     </td><td>99Aaa9999  </td><td>999-999-999</td></tr>\n",
       "\t<tr><th scope=row>9</th><td>aaaa            </td><td>Aaaaaaaaw99w9999</td><td>999w999w999     </td></tr>\n",
       "\t<tr><th scope=row>10</th><td>Aaaaaa     </td><td>9999-99-99 </td><td>999-999-999</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lll}\n",
       "  & Gender & Date & Phone\\\\\n",
       "\\hline\n",
       "\t1 & Aaaaaa      & 99Aaa9999   & 999w999w999\\\\\n",
       "\t2 & Aaaa        & 9999-99-99  & 999w999w999\\\\\n",
       "\t3 & A             & 99/99/9999    & (999)-999-999\\\\\n",
       "\t4 & aaaa          & 99/99/9999    & (999)-999-999\\\\\n",
       "\t5 & Aaaa             & Aaaaaaaaw99w9999 & 999w999w999     \\\\\n",
       "\t6 & Aaaaaa        & 99Aaa9999     & (999)-999-999\\\\\n",
       "\t7 & Aaaaaa        & 9999-99-99    & (999)w999w999\\\\\n",
       "\t8 & Aaaaaa      & 99Aaa9999   & 999-999-999\\\\\n",
       "\t9 & aaaa             & Aaaaaaaaw99w9999 & 999w999w999     \\\\\n",
       "\t10 & Aaaaaa      & 9999-99-99  & 999-999-999\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "   Gender             Date         Phone\n",
       "1  Aaaaaa        99Aaa9999   999w999w999\n",
       "2    Aaaa       9999-99-99   999w999w999\n",
       "3       A       99/99/9999 (999)-999-999\n",
       "4    aaaa       99/99/9999 (999)-999-999\n",
       "5    Aaaa Aaaaaaaaw99w9999   999w999w999\n",
       "6  Aaaaaa        99Aaa9999 (999)-999-999\n",
       "7  Aaaaaa       9999-99-99 (999)w999w999\n",
       "8  Aaaaaa        99Aaa9999   999-999-999\n",
       "9    aaaa Aaaaaaaaw99w9999   999w999w999\n",
       "10 Aaaaaa       9999-99-99   999-999-999"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messy %>%\n",
    "    bpa %>%\n",
    "    head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Com muitos dados, muitas vezes será mais útil exibir uma lista contendo apenas os padrões únicos para cada coluna de um frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "$Gender\n",
       "\n",
       "     A   aaaa   Aaaa aaaaaa Aaaaaa \n",
       "   189    110    312     98    291 \n",
       "\n",
       "$Date\n",
       "\n",
       "       99/99/9999        9999-99-99         99Aaa9999 Aaaaaaaaaw99w9999 \n",
       "              259               262               241                19 \n",
       " Aaaaaaaaw99w9999   Aaaaaaaw99w9999    Aaaaaaw99w9999     Aaaaaw99w9999 \n",
       "               56                45                24                36 \n",
       "     Aaaaw99w9999       Aaaw99w9999 \n",
       "               42                16 \n",
       "\n",
       "$Phone\n",
       "\n",
       "(999)-999-999 (999)w999w999   999-999-999   999w999w999 \n",
       "          242           250           276           232 \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Retorna somente os padôes únicos em formato de lista\n",
    "bpa(messy, unique_only = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finalmente, nós incluímos a função match_pattern para extrair os valores de um vetor que coincidam com um padrão especificado.\n",
    "Por exemplo, o fragmento de código a seguir irá extrair os valores exclusivos de que coincidirem com o padrão padronizado Gender Aaaa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\"Male\""
      ],
      "text/latex": [
       "\"Male\""
      ],
      "text/markdown": [
       "\"Male\""
      ],
      "text/plain": [
       "[1] \"Male\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Extrai de dentro de Gender os valores que coincidem com o padrão \"Aaaa\"\n",
    "match_pattern(messy$Gender, pattern = \"Aaaa\", unique_only = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estudo de Caso - Pagamento de Impostos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(bpa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nf <-read.csv2(\"./dados/NotaFiscal.csv\", header = TRUE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>CNPJ</th><th scope=col>UF</th><th scope=col>Data</th><th scope=col>NF</th><th scope=col>Classificacao</th><th scope=col>Valor</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>21.704.783/0001-64</td><td>RJ                </td><td>01/09/2016        </td><td>90                </td><td>ME                </td><td> R$30.000,00      </td></tr>\n",
       "\t<tr><th scope=row>2</th><td>21.704.783/0001-64</td><td>RJ                </td><td>02/09/2016        </td><td>91                </td><td>ME                </td><td> R$40.000,00      </td></tr>\n",
       "\t<tr><th scope=row>3</th><td>03.117.734/0001-43</td><td>SP                </td><td>03/09/2016        </td><td>2000              </td><td>MEI               </td><td> R$444,00         </td></tr>\n",
       "\t<tr><th scope=row>4</th><td>76.313.316/0001-03</td><td>SP                </td><td>04/09/2016        </td><td>111               </td><td>MEI               </td><td> R$4,00           </td></tr>\n",
       "\t<tr><th scope=row>5</th><td>06.062.510/0001-70</td><td>SP                </td><td>05/09/2016        </td><td>222               </td><td>                  </td><td> R$4.444,00       </td></tr>\n",
       "\t<tr><th scope=row>6</th><td>16.474.789/0001-52</td><td>GO                </td><td>06/09/2016        </td><td>33333             </td><td>                  </td><td> R$55,00          </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       "  & CNPJ & UF & Data & NF & Classificacao & Valor\\\\\n",
       "\\hline\n",
       "\t1 & 21.704.783/0001-64 & RJ                 & 01/09/2016         & 90                 & ME                 &  R\\$30.000,00    \\\\\n",
       "\t2 & 21.704.783/0001-64 & RJ                 & 02/09/2016         & 91                 & ME                 &  R\\$40.000,00    \\\\\n",
       "\t3 & 03.117.734/0001-43 & SP                 & 03/09/2016         & 2000               & MEI                &  R\\$444,00       \\\\\n",
       "\t4 & 76.313.316/0001-03 & SP                 & 04/09/2016         & 111                & MEI                &  R\\$4,00         \\\\\n",
       "\t5 & 06.062.510/0001-70 & SP                 & 05/09/2016         & 222                &                    &  R\\$4.444,00     \\\\\n",
       "\t6 & 16.474.789/0001-52 & GO                 & 06/09/2016         & 33333              &                    &  R\\$55,00        \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "                CNPJ UF       Data    NF Classificacao         Valor\n",
       "1 21.704.783/0001-64 RJ 01/09/2016    90            ME  R$30.000,00 \n",
       "2 21.704.783/0001-64 RJ 02/09/2016    91            ME  R$40.000,00 \n",
       "3 03.117.734/0001-43 SP 03/09/2016  2000           MEI     R$444,00 \n",
       "4 76.313.316/0001-03 SP 04/09/2016   111           MEI       R$4,00 \n",
       "5 06.062.510/0001-70 SP 05/09/2016   222                 R$4.444,00 \n",
       "6 16.474.789/0001-52 GO 06/09/2016 33333                    R$55,00 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(nf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>CNPJ</th><th scope=col>UF</th><th scope=col>Data</th><th scope=col>NF</th><th scope=col>Classificacao</th><th scope=col>Valor</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>99                </td><td>AA                </td><td>wA$99.999,99w     </td></tr>\n",
       "\t<tr><th scope=row>2</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>99                </td><td>AA                </td><td>wA$99.999,99w     </td></tr>\n",
       "\t<tr><th scope=row>3</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>9999              </td><td>AAA               </td><td>wA$999,99w        </td></tr>\n",
       "\t<tr><th scope=row>4</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>999               </td><td>AAA               </td><td>wA$9,99w          </td></tr>\n",
       "\t<tr><th scope=row>5</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>999               </td><td>                  </td><td>wA$9.999,99w      </td></tr>\n",
       "\t<tr><th scope=row>6</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>99999             </td><td>                  </td><td>wA$99,99w         </td></tr>\n",
       "\t<tr><th scope=row>7</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>9999999           </td><td>AAA               </td><td>wA$999,99w        </td></tr>\n",
       "\t<tr><th scope=row>8</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>999               </td><td>AA                </td><td>wA$999,99w        </td></tr>\n",
       "\t<tr><th scope=row>9</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>999               </td><td>                  </td><td>wA$9.999,99w      </td></tr>\n",
       "\t<tr><th scope=row>10</th><td>99.999.999/9999-99</td><td>AA                </td><td>99/99/9999        </td><td>99                </td><td>AAA               </td><td>wA$9.999.999,99w  </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllll}\n",
       "  & CNPJ & UF & Data & NF & Classificacao & Valor\\\\\n",
       "\\hline\n",
       "\t1 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 99                 & AA                 & wA\\$99.999,99w   \\\\\n",
       "\t2 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 99                 & AA                 & wA\\$99.999,99w   \\\\\n",
       "\t3 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 9999               & AAA                & wA\\$999,99w      \\\\\n",
       "\t4 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 999                & AAA                & wA\\$9,99w        \\\\\n",
       "\t5 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 999                &                    & wA\\$9.999,99w    \\\\\n",
       "\t6 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 99999              &                    & wA\\$99,99w       \\\\\n",
       "\t7 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 9999999            & AAA                & wA\\$999,99w      \\\\\n",
       "\t8 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 999                & AA                 & wA\\$999,99w      \\\\\n",
       "\t9 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 999                &                    & wA\\$9.999,99w    \\\\\n",
       "\t10 & 99.999.999/9999-99 & AA                 & 99/99/9999         & 99                 & AAA                & wA\\$9.999.999,99w\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "                 CNPJ UF       Data      NF Classificacao            Valor\n",
       "1  99.999.999/9999-99 AA 99/99/9999      99            AA    wA$99.999,99w\n",
       "2  99.999.999/9999-99 AA 99/99/9999      99            AA    wA$99.999,99w\n",
       "3  99.999.999/9999-99 AA 99/99/9999    9999           AAA       wA$999,99w\n",
       "4  99.999.999/9999-99 AA 99/99/9999     999           AAA         wA$9,99w\n",
       "5  99.999.999/9999-99 AA 99/99/9999     999                   wA$9.999,99w\n",
       "6  99.999.999/9999-99 AA 99/99/9999   99999                      wA$99,99w\n",
       "7  99.999.999/9999-99 AA 99/99/9999 9999999           AAA       wA$999,99w\n",
       "8  99.999.999/9999-99 AA 99/99/9999     999            AA       wA$999,99w\n",
       "9  99.999.999/9999-99 AA 99/99/9999     999                   wA$9.999,99w\n",
       "10 99.999.999/9999-99 AA 99/99/9999      99           AAA wA$9.999.999,99w"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nf %>%\n",
    "    bpa%>%\n",
    "head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "$CNPJ\n",
       "\n",
       "99.999.999/9999-99 \n",
       "                10 \n",
       "\n",
       "$UF\n",
       "\n",
       "AA \n",
       "10 \n",
       "\n",
       "$Data\n",
       "\n",
       "99/99/9999 \n",
       "        10 \n",
       "\n",
       "$NF\n",
       "\n",
       "     99     999    9999   99999 9999999 \n",
       "      3       4       1       1       1 \n",
       "\n",
       "$Classificacao\n",
       "\n",
       "     AA AAA \n",
       "  3   3   4 \n",
       "\n",
       "$Valor\n",
       "\n",
       "        wA$9,99w     wA$9.999,99w wA$9.999.999,99w        wA$99,99w \n",
       "               1                2                1                1 \n",
       "   wA$99.999,99w       wA$999,99w \n",
       "               2                3 \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Retorna somente os padôes únicos em formato de lista\n",
    "bpa(nf, unique_only = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Busca NF por tamanho"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>.</th><th scope=col>Freq</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>1</th><td>99</td><td>3 </td></tr>\n",
       "\t<tr><th scope=row>2</th><td>999</td><td>4  </td></tr>\n",
       "\t<tr><th scope=row>3</th><td>9999</td><td>1   </td></tr>\n",
       "\t<tr><th scope=row>4</th><td>99999</td><td>1    </td></tr>\n",
       "\t<tr><th scope=row>5</th><td>9999999</td><td>1      </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ll}\n",
       "  & . & Freq\\\\\n",
       "\\hline\n",
       "\t1 & 99 & 3 \\\\\n",
       "\t2 & 999 & 4  \\\\\n",
       "\t3 & 9999 & 1   \\\\\n",
       "\t4 & 99999 & 1    \\\\\n",
       "\t5 & 9999999 & 1      \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "        . Freq\n",
       "1      99    3\n",
       "2     999    4\n",
       "3    9999    1\n",
       "4   99999    1\n",
       "5 9999999    1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "nf$NF %>%\n",
    "  get_pattern %>%  # extrai o padrão\n",
    "  table %>%        # tabula frequências\n",
    "  as.data.frame    # apresenta o dado como data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Encontrar NF com padrão 99"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>90</li>\n",
       "\t<li>91</li>\n",
       "\t<li>89</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 90\n",
       "\\item 91\n",
       "\\item 89\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 90\n",
       "2. 91\n",
       "3. 89\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 90 91 89"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Extrai de dentro de Gender os valores que coincidem com o padrão \"99\"\n",
    "match_pattern(nf$NF, pattern = \"99\", unique_only = TRUE)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.1"
  },
  "toc": {
   "nav_menu": {
    "height": "191px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": false,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}