Day4-Python_regular_expression_作業.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3.7.6 64-bit (conda)","metadata":{"interpreter":{"hash":"7b4b4feff2f24a0f0a34464dbe537a36fda679851528fb8735cb41fa49dffb2d"}}},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6-final"},"colab":{"name":"Python_regular_expression_作業.ipynb","provenance":[]}},"cells":[{"cell_type":"markdown","metadata":{"id":"tTJolmzeBLGr","colab_type":"text"},"source":["# 作業目標: 使用python正規表達式對資料進行清洗處理\n","\n","這份作業我們會使用詐欺郵件的文本資料來作為清洗與處理的操作。\n","[資料集](https://www.kaggle.com/rtatman/fraudulent-email-corpus/data#)"]},{"cell_type":"markdown","metadata":{"id":"vhv0OElTBLGs","colab_type":"text"},"source":["### 讀入資料文本\n","因原始文本較大，先使用部份擷取的**sample_emails.txt**來進行練習"]},{"cell_type":"code","metadata":{"id":"a0BdDqt1BLGt","colab_type":"code","colab":{}},"source":["#讀取文本資料\n","#<your code>#\n","with open('sample_emails.txt', 'r', encoding= 'UTF8') as f:\n","    sample_corpus = f.read()"],"execution_count":1,"outputs":[]},{"cell_type":"code","metadata":{"scrolled":true,"id":"XQbjXNKdBLGx","colab_type":"code","colab":{},"outputId":"97497027-4c29-4dcf-9cf4-94b0081dac54"},"source":["sample_corpus"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'From r  Wed Oct 30 21:41:56 2002\\nReturn-Path: <james_ngola2002@maktoob.com>\\nX-Sieve: cmu-sieve 2.0\\nReturn-Path: <james_ngola2002@maktoob.com>\\nMessage-Id: <200210310241.g9V2fNm6028281@cs.CU>\\nFrom: \"MR. JAMES NGOLA.\" <james_ngola2002@maktoob.com>\\nReply-To: james_ngola2002@maktoob.com\\nTo: webmaster@aclweb.org\\nDate: Thu, 31 Oct 2002 02:38:20 +0000\\nSubject: URGENT BUSINESS ASSISTANCE AND PARTNERSHIP\\nX-Mailer: Microsoft Outlook Express 5.00.2919.6900 DM\\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=\"us-ascii\"\\nContent-Transfer-Encoding: 8bit\\nX-MIME-Autoconverted: from quoted-printable to 8bit by sideshowmel.si.UM id g9V2foW24311\\nStatus: O\\n\\nFROM:MR. JAMES NGOLA.\\nCONFIDENTIAL TEL: 233-27-587908.\\nE-MAIL: (james_ngola2002@maktoob.com).\\n\\nURGENT BUSINESS ASSISTANCE AND PARTNERSHIP.\\n\\n\\nDEAR FRIEND,\\n\\nI AM ( DR.) JAMES NGOLA, THE PERSONAL ASSISTANCE TO THE LATE CONGOLESE (PRESIDENT LAURENT KABILA) WHO WAS ASSASSINATED BY HIS BODY GUARD ON 16TH JAN. 2001.\\n\\n\\nTHE INCIDENT OCCURRED IN OUR PRESENCE WHILE WE WERE HOLDING MEETING WITH HIS EXCELLENCY OVER THE FINANCIAL RETURNS FROM THE DIAMOND SALES IN THE AREAS CONTROLLED BY (D.R.C.) DEMOCRATIC REPUBLIC OF CONGO FORCES AND THEIR FOREIGN ALLIES ANGOLA AND ZIMBABWE, HAVING RECEIVED THE PREVIOUS DAY (USD$100M) ONE HUNDRED MILLION UNITED STATES DOLLARS, CASH IN THREE DIPLOMATIC BOXES ROUTED THROUGH ZIMBABWE.\\n\\nMY PURPOSE OF WRITING YOU THIS LETTER IS TO SOLICIT FOR YOUR ASSISTANCE AS TO BE A COVER TO THE FUND AND ALSO COLLABORATION IN MOVING THE SAID FUND INTO YOUR BANK ACCOUNT THE SUM OF (USD$25M) TWENTY FIVE MILLION UNITED STATES DOLLARS ONLY, WHICH I DEPOSITED WITH A SECURITY COMPANY IN GHANA, IN A DIPLOMATIC BOX AS GOLDS WORTH (USD$25M) TWENTY FIVE MILLION UNITED STATES DOLLARS ONLY FOR SAFE KEEPING IN A SECURITY VAULT FOR ANY FURTHER INVESTMENT PERHAPS IN YOUR COUNTRY. \\n\\nYOU WERE INTRODUCED TO ME BY A RELIABLE FRIEND OF MINE WHO IS A TRAVELLER,AND ALSO A MEMBER OF CHAMBER OF COMMERCE AS A RELIABLE AND TRUSTWORTHY PERSON WHOM I CAN RELY ON AS FOREIGN PARTNER, EVEN THOUGH THE NATURE OF THE TRANSACTION WAS NOT REVEALED TO HIM FOR SECURITY REASONS.\\n\\n\\nTHE (USD$25M) WAS PART OF A PROCEEDS FROM DIAMOND TRADE MEANT FOR THE LATE PRESIDENT LAURENT KABILA WHICH WAS DELIVERED THROUGH ZIMBABWE IN DIPLOMATIC BOXES. THE BOXES WERE KEPT UNDER MY CUSTODY BEFORE THE SAD EVENT THAT TOOK THE LIFE OF (MR. PRESIDENT).THE CONFUSION THAT ENSUED AFTER THE ASSASSINATION AND THE SPORADIC SHOOTING AMONG THE FACTIONS, I HAVE TO RUN AWAY FROM THE COUNTRY FOR MY DEAR LIFE AS I AM NOT A SOLDIER BUT A CIVIL SERVANT I CROSSED RIVER CONGO TO OTHER SIDE OF CONGO LIBREVILLE FROM THERE I MOVED TO THE THIRD COUNTRY GHANA WHERE I AM PRESENTLY TAKING REFUGE. \\n\\nAS A MATTER OF FACT, WHAT I URGENTLY NEEDED FROM YOU IS YOUR ASSISTANCE IN MOVING THIS MONEY INTO YOUR ACCOUNT IN YOUR COUNTRY FOR INVESTMENT WITHOUT RAISING EYEBROW. FOR YOUR ASSISTANCE I WILL GIVE YOU 20% OF THE TOTAL SUM AS YOUR OWN SHARE WHEN THE MONEY GETS TO YOUR ACCOUNT, WHILE 75% WILL BE FOR ME, OF WHICH WITH YOUR KIND ADVICE I HOPE TO INVEST IN PROFITABLE VENTURE IN YOUR COUNTRY IN OTHER TO SETTLE DOWN FOR MEANINGFUL LIFE, AS I AM TIRED OF LIVING IN A WAR ENVIRONMENT. \\n\\nTHE REMAINING 5% WILL BE USED TO OFFSET ANY COST INCURRED IN THE CAUSE OF MOVING THE MONEY TO YOUR ACCOUNT. IF THE PROPOSAL IS ACCEPTABLE TO YOU PLEASE CONTACT ME IMMEDIATELY THROUGH THE ABOVE TELEPHONE AND E-MAIL, TO ENABLE ME ARRANGE FACE TO FACE MEETING WITH YOU IN GHANA FOR THE CLEARANCE OF THE FUNDS BEFORE TRANSFRING IT TO YOUR BANK ACCOUNT AS SEEING IS BELIEVING. \\n\\nFINALLY, IT IS IMPORTANT ALSO THAT I LET YOU UNDERSTAND THAT THERE IS NO RISK INVOLVED WHATSOEVER AS THE MONEY HAD NO RECORD IN KINSHASA FOR IT WAS MEANT FOR THE PERSONAL USE OF (MR. PRESIDEND ) BEFORE THE NEFARIOUS INCIDENT OCCURRED, AND ALSO I HAVE ALL THE NECESSARY DOCUMENTS AS REGARDS TO THE FUNDS INCLUDING THE (CERTIFICATE OF DEPOSIT), AS I AM THE DEPOSITOR OF THE CONSIGNMENT.\\n\\n\\nLOOKING FORWARD TO YOUR URGENT RESPONSE.\\n\\nYOUR SINCERELY,\\n\\nMR. JAMES NGOLA. \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nFrom r  Thu Oct 31 08:11:39 2002\\nReturn-Path: <bensul2004nng@spinfinder.com>\\nX-Sieve: cmu-sieve 2.0\\nReturn-Path: <bensul2004nng@spinfinder.com>\\nMessage-Id: <200210311310.g9VDANt24674@bloodwork.mr.itd.UM>\\nFrom: \"Mr. Ben Suleman\" <bensul2004nng@spinfinder.com>\\nDate: Thu, 31 Oct 2002 05:10:00\\nTo: R@M\\nSubject: URGENT ASSISTANCE /RELATIONSHIP (P)\\nMIME-Version: 1.0\\nContent-Type: text/plain;charset=\"iso-8859-1\"\\nContent-Transfer-Encoding: 7bit\\nStatus: O\\n\\nDear Friend,\\n\\nI am Mr. Ben Suleman a custom officer and work as Assistant controller of the Customs and Excise department Of the Federal Ministry of Internal Affairs stationed at the Murtala Mohammed International Airport, Ikeja, Lagos-Nigeria.\\n\\nAfter the sudden death of the former Head of state of Nigeria General Sanni Abacha on June 8th 1998 his aides and immediate members of his family were arrested while trying to escape from Nigeria in a Chartered jet to Saudi Arabia with 6 trunk boxes Marked \"Diplomatic Baggage\". Acting on a tip-off as they attempted to board the Air Craft,my officials carried out a thorough search on the air craft and discovered that the 6 trunk boxes contained foreign currencies amounting to US$197,570,000.00(One Hundred and  Ninety-Seven Million Five Hundred Seventy Thousand United States Dollars).\\n\\nI declared only (5) five boxes to the government and withheld one (1) in my custody containing the sum of (US$30,000,000.00) Thirty Million United States Dollars Only, which has been disguised to prevent their being discovered during transportation process.Due to several media reports on the late head of state about all the money him and his co-government officials stole from our government treasury amounting\\nto US$55 Billion Dollars (ref:ngrguardiannews.com) of July 2nd 1999. Even the London times of July 1998 reported that General Abacha has over US$3.Billion dollars in one account overseas. We decided to conceal this one (1)box till the situation is calm and quite on the issue. The box was thus deposited with a security company here in Nigeria and tagged as \"Precious Stones and Jewellry\" in other that its\\ncontent will not be discovered. Now that all is calm, we (myself and two of my colleagues in the operations team) are now ready to move this box out of the country through a diplomatic arrangement which is the safest means. \\n\\nHowever as government officials the Civil Service Code of Conduct does not allow us by law to operate any foreign account or own foreign investment and the amount of money that can be found in our account\\ncannot be more than our salary on the average, thus our handicapp and our need for your assistance to help collect and keep safely in your account this money.\\n\\nTherefore we want you to assist us in moving this money out of Nigeria. We shall definitely compensate you handsomely for the assistance. We can do this by instructing the Security Company here in Nigeria to\\nmove the consignment to their affiliate branch office outside Nigeria through diplomatic means and the consignment will be termed as Precious Stones and Jewelleries\" which you bought during your visit to Nigeria and is being transfered to your country from here for safe keeping. Then we can arrange to meet at the destination country to take the delivery of the consignment. You will thereafter open an account there and lodge the Money there and gradually instruct remittance to your Country. \\n\\nThis business is 100% risk free for you so please treat this matter with utmost confidentiality .If you indicate your interest to assist us please just e-mail me for more Explanation on how we plan to execute the transaction.\\n\\nExpecting your response urgently.\\n\\nBest regards,\\n\\nMr. Ben Suleman\\n\\nFrom r  Thu Oct 31 17:27:16 2002\\nReturn-Path: <obong_715@epatra.com>\\nX-Sieve: cmu-sieve 2.0\\nReturn-Path: <obong_715@epatra.com>\\nMessage-Id: <200210312227.g9VMQvDj017948@bluewhale.cs.CU>\\nFrom: \"PRINCE OBONG ELEME\" <obong_715@epatra.com>\\nReply-To: obong_715@epatra.com\\nTo: webmaster@aclweb.org\\nDate: Thu, 31 Oct 2002 22:17:55 +0100\\nSubject: GOOD DAY TO YOU\\nX-Mailer: Microsoft Outlook Express 5.00.2919.6900DM\\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=\"us-ascii\"\\nContent-Transfer-Encoding: 8bit\\nX-MIME-Autoconverted: from quoted-printable to 8bit by sideshowmel.si.UM id g9VMRBW20642\\nStatus: RO\\n\\nFROM HIS ROYAL MAJESTY (HRM) CROWN RULER OF ELEME KINGDOM \\nCHIEF DANIEL ELEME, PHD, EZE 1 OF ELEME.E-MAIL \\nADDRESS:obong_715@epatra.com  \\n\\nATTENTION:PRESIDENT,CEO Sir/ Madam. \\n\\nThis letter might surprise you because we have met\\nneither in person nor by correspondence. But I believe\\nit is one day that you got to know somebody either in\\nphysical or through correspondence. \\n\\nI got your contact through discreet inquiry from the\\nchambers of commerce and industry of your country on\\nthe net, you and your organization were revealed as\\nbeing quite astute in private entrepreneurship, one\\nhas no doubt in your ability to handle a financialbusiness transaction. \\n\\nHowever, I am the first son of His Royal\\nmajesty,Obong.D. Eleme , and the traditional Ruler of\\nEleme Province in the oil producing area of River\\nState of Nigeria. I am making this contact to you in\\nrespect of US$60,000,000.00 (Sixty Million United\\nState Dollars), which I inherited, from my latefather. \\n\\nThis money was accumulated from royalties paid to my\\nfather as compensation by the oil firms located in our\\narea as a result of oil presence on our land, which\\nhamper agriculture, which is our major source oflivelihood. \\n\\nUnfortunately my father died from protracted\\ndiabetes.But before his death he called my attention\\nand informed me that he lodged some funds on a two\\nboxes with a security firm with an open beneficiary\\nstatus. The lodgment security code number was also\\nrevealed to me, he then advised me to look for a\\nreliable business partner abroad, that will assist me\\nin investing the money in a lucrative business as a\\nresult of economic instability in Nigeria. So this is\\nthe main reason why I am contacting you for us to move\\nthis money from the security firm to any Country of\\nyour choice for investment purpose. \\n\\nSo I will like you to be the ultimate beneficiary, so\\nthat the funds can be moved in your name and\\nparticulars to any Country of your choice where it\\nwill be claimed and invested. Hence my father have had\\nintimated the security firm personnel that the\\nbeneficiary of the box is his foreign partner whose\\nparticulars will be forwarded to the firm when due. \\n\\nBut I will guide you Accordingly. As soon as the funds\\nreach, I will then come over to meet you in person, so\\nthat we can discuss physically on investment\\npotentials. Based on this assistance my Family and I\\nhave unanimously decided to give you 30% of the total\\nmoney, 5% for Charity home, 10% for expenses, which\\nmay arise during this transaction, Fax and phone bills\\ninclusive. The balance of 55% you will invest andmanaged for my Family. \\n\\nI hereby guarantee you that this is not government\\nmoney, it is not drug money and it is not money from\\narms deal. Though you have to maintain high degree of\\nconfidentiality on this matter. I will give more\\ndetails about the proceedings of this transaction as\\nsoon as I receive your favorable reply. \\n\\nPlease reply to my Email Address:obong_715@epatra.com\\nI hope this will be the beginning of a prosperous\\nrelationship between my family and your family. \\n\\nNevertheless if you are for any reason not interested,\\nkindly inform me immediately so that I will look foranother contact. \\n\\nI am waiting for your quick response. \\n\\nYours faithfully, \\n\\nPrince Obong Abbot \\n'"]},"metadata":{},"execution_count":2}]},{"cell_type":"markdown","metadata":{"id":"aUg8HenSBLG2","colab_type":"text"},"source":["### 讀取寄件者資訊\n","觀察文本資料可以發現, 寄件者資訊都符合以下格式\n","\n","`From: <收件者姓名> <收件者電子郵件>`"]},{"cell_type":"code","metadata":{"id":"YgsUuHsfBLG2","colab_type":"code","colab":{}},"source":["#<your code>#\n","import re\n","pattern= r'From: .*'\n","match= re.findall(pattern,sample_corpus)"],"execution_count":5,"outputs":[]},{"cell_type":"code","metadata":{"id":"pofRrOGhBLG5","colab_type":"code","colab":{},"outputId":"28eb134d-2dfa-4743-fae8-1f0abda25968"},"source":["match"],"execution_count":6,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['From: \"MR. JAMES NGOLA.\" <james_ngola2002@maktoob.com>',\n"," 'From: \"Mr. Ben Suleman\" <bensul2004nng@spinfinder.com>',\n"," 'From: \"PRINCE OBONG ELEME\" <obong_715@epatra.com>']"]},"metadata":{},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"Ac8KHAqTBLG7","colab_type":"text"},"source":["### 只讀取寄件者姓名"]},{"cell_type":"code","metadata":{"id":"coBDE2iBBLG8","colab_type":"code","colab":{},"outputId":"aada7b60-56bb-45ae-f25f-941218a541c2"},"source":["#<your code>#\n","pattern=r\"\\\".*\\\"\"\n","for i in match:\n","    print(re.search(pattern,i).group())"],"execution_count":21,"outputs":[{"output_type":"stream","name":"stdout","text":["\"MR. JAMES NGOLA.\"\n\"Mr. Ben Suleman\"\n\"PRINCE OBONG ELEME\"\n"]}]},{"cell_type":"markdown","metadata":{"id":"gaJefecaBLG-","colab_type":"text"},"source":["### 只讀取寄件者電子信箱"]},{"cell_type":"code","metadata":{"id":"G55ADKW_BLG_","colab_type":"code","colab":{},"outputId":"7f3d9657-87b9-4c78-d324-3df919ee84a7"},"source":["#<your code>#\n","pattern= r'\\w+@\\w*\\.com'\n","for i in match:\n","    print(re.search(pattern,i).group())"],"execution_count":22,"outputs":[{"output_type":"stream","name":"stdout","text":["james_ngola2002@maktoob.com\nbensul2004nng@spinfinder.com\nobong_715@epatra.com\n"]}]},{"cell_type":"markdown","metadata":{"id":"t-RXjNLiBLHC","colab_type":"text"},"source":["### 只讀取電子信箱中的寄件機構資訊\n","ex: james_ngola2002@maktoob.com --> 取maktoob"]},{"cell_type":"code","metadata":{"id":"qC2Sd6_SBLHC","colab_type":"code","colab":{},"outputId":"14e4e148-a660-4a15-88ac-e781300c9767"},"source":["#<your code>#\n","pattern=r'(?<=@)\\w+'\n","for i in match:\n","    print(re.search(pattern,i).group())"],"execution_count":24,"outputs":[{"output_type":"stream","name":"stdout","text":["maktoob\nspinfinder\nepatra\n"]}]},{"cell_type":"markdown","metadata":{"id":"vfzkuLCWBLHG","colab_type":"text"},"source":["### 結合上面的配對方式, 將寄件者的帳號與機構訊返回\n","ex: james_ngola2002@maktoob.com --> [james_ngola2002, maktoob]"]},{"cell_type":"code","metadata":{"id":"t3lTRThcBLHG","colab_type":"code","colab":{},"outputId":"e94f295a-ffa4-4897-dc43-e93991aac20e"},"source":["#<your code>#\n","pattern_1= r'\\w+(?=@)'\n","pattern_2= r'(?<=@)\\w+'\n","for i in match:\n","    print(re.search(pattern_1,i).group(),',',re.search(pattern_2,i).group())"],"execution_count":31,"outputs":[{"output_type":"stream","name":"stdout","text":["james_ngola2002 , maktoob\nbensul2004nng , spinfinder\nobong_715 , epatra\n"]}]},{"cell_type":"markdown","metadata":{"id":"Yl-BMduNBLHJ","colab_type":"text"},"source":["### 使用正規表達式對email資料進行處理\n","這裡我們會使用到python其他的套件協助處理(ex: pandas, email, etc)，這裡我們只需要專注在正規表達式上即可，其他的套件是方便我們整理與處理資料。"]},{"cell_type":"markdown","metadata":{"id":"kqCacQ9tBLHJ","colab_type":"text"},"source":["### 讀取與切分Email\n","讀入的email為一個長字串，利用正規表達式切割讀入的資料成一封一封的email，並將結果以list表示。\n","\n","輸出: [email_1, email_2, email_3, ....]"]},{"cell_type":"code","metadata":{"id":"k9BCVTqoBLHK","colab_type":"code","colab":{},"outputId":"656ffc81-cb9b-4174-8511-9013985c4cad"},"source":["import re\n","import pandas as pd\n","import email\n","\n","###讀取文本資料:fradulent_emails.txt###\n","#<your code>#\n","with open('all_emails.txt','r', encoding='UTF8',errors='ignore') as f:\n","    data=f.read()\n","###切割讀入的資料成一封一封的email###\n","###我們可以使用list來儲存每一封email###\n","###注意！這裡請仔細觀察sample資料，看資料是如何切分不同email###\n","#<your code>#\n","emails=re.split(r\"From r\",data, flags=re.M)\n","emails=emails[1:]\n","len(emails) #查看有多少封email"],"execution_count":35,"outputs":[{"output_type":"execute_result","data":{"text/plain":["3977"]},"metadata":{},"execution_count":35}]},{"cell_type":"markdown","metadata":{"id":"e9IN-2o0BLHM","colab_type":"text"},"source":["### 從文本中擷取所有寄件者與收件者的姓名和地址"]},{"cell_type":"code","metadata":{"id":"d_WRYFpcBLHM","colab_type":"code","colab":{}},"source":["emails_list = [] #創建空list來儲存所有email資訊\n","\n","for mail in emails[:20]: #只取前20筆資料 (處理速度比較快)\n","    emails_dict = dict() #創建空字典儲存資訊\n","    ###取的寄件者姓名與地址###\n","    \n","    #Step1: 取的寄件者資訊 (hint: From:)\n","    #<your code>#\n","    send= re.search(r'From:.*',mail)\n","    #Step2: 取的姓名與地址 (hint: 要注意有時會有沒取到配對的情況)\n","    #<your code>#\n","    if send is not None:\n","        send_mail= re.search(r'\\w+@.*', send.group())\n","        send_name= re.search(r'\\\".*\\\"',send.group())\n","    else:\n","        send_mail= None\n","        send_name= None\n","    #Step3: 將取得的姓名與地址存入字典中\n","    #<your code>#\n","    if send_mail is not None:\n","        emails_dict['send_mail']= send_mail.group()\n","    else:\n","        emails_dict['send_mail']= send_mail\n","    \n","    if send_name is not None:\n","        emails_dict['send_name']= send_name.group()\n","    else:\n","        emails_dict['send_name']= send_name\n","    \n","    ###取的收件者姓名與地址###\n","    #Step1: 取的寄件者資訊 (hint: To:)\n","    #<your code>#\n","    receiver= re.search(r'To:.*',mail)\n","    #Step2: 取的姓名與地址 (hint: 要注意有時會有沒取到配對的情況)\n","    #<your code>#\n","    if receiver is not None:\n","        receiver_mail= re.search(r'\\w+@\\w+', receiver.group())\n","        receiver_name= re.search(r'\\\".*\\\"',receiver.group())\n","    else:\n","        receiver_mail= None\n","        receiver_name= None\n","    #Step3: 將取得的姓名與地址存入字典中\n","    #<your code>#\n","    if receiver_mail is not None:\n","        emails_dict['receiver_mail']= receiver_mail.group()\n","    else:\n","        emails_dict['receiver_mail']= receiver_mail\n","    \n","    if receiver_name is not None:\n","        emails_dict['receiver_name']= receiver_name.group()\n","    else:\n","        emails_dict['receiver_name']= receiver_name        \n","        \n","    ###取得信件日期###\n","    #Step1: 取得日期資訊 (hint: To:)\n","    #<your code>#\n","    Date= re.search(r'Date:.*',mail)\n","    #Step2: 取得詳細日期(只需取得DD MMM YYYY)\n","    #<your code>#\n","    if Date is not None:\n","        Date_mail= re.search(r'\\d+\\s\\w+\\s\\d+', Date.group())\n","    else:\n","        Date_mail= None      \n","    #Step3: 將取得的日期資訊存入字典中\n","    #<your code>#\n","    if Date_mail is not None:\n","        emails_dict['Date']= Date_mail.group()\n","    else:\n","        emails_dict['Date']= Date_mail         \n","        \n","    ###取得信件主旨###\n","    #Step1: 取得主旨資訊 (hint: Subject:)\n","    #<your code>#\n","    subject= re.search(r'Subject: .*',mail)\n","    #Step2: 移除不必要文字 (hint: Subject: )\n","    #<your code>#\n","    if subject is not None:\n","        subject_mail= re.sub(r'Subject: ', \"\", subject.group())\n","    else:\n","        subject_mail= None      \n","    #Step3: 將取得的主旨存入字典中\n","    #<your code>#\n","    emails_dict['Subject']= subject_mail\n","\n","    \n","    ###取得信件內文###\n","    #這裡我們使用email package來取出email內文 (可以不需深究，本章節重點在正規表達式)\n","    try:\n","        full_email = email.message_from_string(mail)\n","        body = full_email.get_payload()\n","        emails_dict[\"email_body\"] = body\n","    except:\n","        emails_dict[\"email_body\"] = None\n","    \n","    ###將字典加入list###\n","    #<your code>#\n","    emails_list.append(emails_dict)"],"execution_count":54,"outputs":[]},{"cell_type":"code","metadata":{"id":"CBfQRiYPBLHP","colab_type":"code","colab":{}},"source":["#將處理結果轉化為dataframe\n","emails_df = pd.DataFrame(emails_list)\n","emails_df"],"execution_count":55,"outputs":[{"output_type":"execute_result","data":{"text/plain":["                         send_mail                 send_name  \\\n","0     james_ngola2002@maktoob.com>        \"MR. JAMES NGOLA.\"   \n","1    bensul2004nng@spinfinder.com>         \"Mr. Ben Suleman\"   \n","2            obong_715@epatra.com>      \"PRINCE OBONG ELEME\"   \n","3            obong_715@epatra.com>      \"PRINCE OBONG ELEME\"   \n","4              m_abacha03@www.com>           \"Maryam Abacha\"   \n","5          davidkuta@postmark.net>                      None   \n","6          tunde_dosumu@lycos.com>  \"Barrister tunde dosumu\"   \n","7   william2244drallo@maktoob.com>          \"William Drallo\"   \n","8        abdul_817@rediffmail.com>          \"MR USMAN ABDUL\"   \n","9          barrister_td@lycos.com>           \"Tunde  Dosumu\"   \n","10    temijohnson2@rediffmail.com>                      None   \n","11          sjordan@diplomats.com>           \"Dr.Sam jordan\"   \n","12             p_brown2@lawyer.com                      None   \n","13                 mic_k1@post.com                      None   \n","14     mikebunduu1@rediffmail.com>      \"COL. MICHAEL BUNDU\"   \n","15            elixwilliam@usa.com>       \"MRS MARIAM ABACHA\"   \n","16          anayoawka@hotmail.com>        \" DR. ANAYO AWKA \"   \n","17          anayoawka@hotmail.com>        \" DR. ANAYO AWKA \"   \n","18        victorloma@netscape.net>            \"Victor Aloma\"   \n","19        victorloma@netscape.net>            \"Victor Aloma\"   \n","\n","                 receiver_mail receiver_name         Date  \\\n","0      james_ngola2002@maktoob          None  31 Oct 2002   \n","1                          R@M          None  31 Oct 2002   \n","2             obong_715@epatra          None  31 Oct 2002   \n","3             webmaster@aclweb          None  31 Oct 2002   \n","4               m_abacha03@www          None   1 Nov 2002   \n","5              davidkuta@yahoo          None  02 Nov 2002   \n","6           tunde_dosumu@lycos          None         None   \n","7    william2244drallo@maktoob          None   3 Nov 2002   \n","8                          R@M          None  04 Nov 2002   \n","9           barrister_td@lycos          None         None   \n","10                         R@E          None   6 Nov 2001   \n","11                         R@M          None  08 Nov 2002   \n","12             p_brown2@lawyer          None   8 Nov 2002   \n","13                         R@M          None  11 Nov 2002   \n","14                         R@E          None  13 Nov 2001   \n","15  presidentwifeng@indiatimes          None  14 Nov 2002   \n","16            webmaster@aclweb          None  15 Nov 2002   \n","17            webmaster@aclweb          None  15 Nov 2002   \n","18               listmem@newns          None  16 Nov 2002   \n","19               outgoing@mail          None  16 Nov 2002   \n","\n","                                       Subject  \\\n","0   URGENT BUSINESS ASSISTANCE AND PARTNERSHIP   \n","1          URGENT ASSISTANCE /RELATIONSHIP (P)   \n","2                              GOOD DAY TO YOU   \n","3                              GOOD DAY TO YOU   \n","4                      I Need Your Assistance.   \n","5                                  Partnership   \n","6                             Urgent Attention   \n","7                      URGENT BUSINESS PRPOSAL   \n","8                                    THANK YOU   \n","9                            Urgent Assistance   \n","10                                   IMPORTANT   \n","11                          URGENT ASSISTANCE.   \n","12                 From: Barrister Peter Brown   \n","13                                     MICHAEL   \n","14                    *****SPAM***** IMPORTANT   \n","15                           TRUST TRANSACTION   \n","16      REQUEST FOR YOUR UNRESERVED ASSISTANCE   \n","17      REQUEST FOR YOUR UNRESERVED ASSISTANCE   \n","18                           Urgent Assistance   \n","19                           Urgent Assistance   \n","\n","                                           email_body  \n","0   FROM:MR. JAMES NGOLA.\\nCONFIDENTIAL TEL: 233-2...  \n","1   Dear Friend,\\n\\nI am Mr. Ben Suleman a custom ...  \n","2   FROM HIS ROYAL MAJESTY (HRM) CROWN RULER OF EL...  \n","3   FROM HIS ROYAL MAJESTY (HRM) CROWN RULER OF EL...  \n","4   Dear sir, \\n \\nIt is with a heart full of hope...  \n","5   ATTENTION:                                    ...  \n","6   Dear Sir,\\n\\nI am Barrister Tunde Dosumu (SAN)...  \n","7   FROM: WILLIAM DRALLO.\\nCONFIDENTIAL TEL: 233-2...  \n","8   CHALLENGE SECURITIES LTD.\\nLAGOS, NIGERIA\\n\\n\\...  \n","9   Dear Sir,\\n\\nI am Barrister Tunde Dosumu (SAN)...  \n","10  FROM: MR TEMI JOHNSON\\nDEMOCRATIC REPUBLIC OF ...  \n","11  FROM THE DESK OF DR.SAM JORDAN\\nTHE MANAGER\\nH...  \n","12  \\nSENIOR ADVOCATE OF NIGERIA\\nBARR. PETER BROW...  \n","13  From;Mr.Michael Kamah and Family,\\n          J...  \n","14  FROM: COL. MICHAEL BUNDU. \\nDEMOCRATIC REPUBLI...  \n","15  Dear Sir,\\n\\nA Very Good day to you   \\n\\nI am...  \n","16  FROM: DR. ANAYO AWKA BANK MANAGER \\n(UNION BAN...  \n","17  FROM: DR. ANAYO AWKA BANK MANAGER \\n(UNION BAN...  \n","18  The Director,\\n\\n\\n\\n            SEEKING FOR I...  \n","19  The Director,\\n\\n\\n\\n            SEEKING FOR I...  "],"text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>send_mail</th>\n      <th>send_name</th>\n      <th>receiver_mail</th>\n      <th>receiver_name</th>\n      <th>Date</th>\n      <th>Subject</th>\n      <th>email_body</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>james_ngola2002@maktoob.com&gt;</td>\n      <td>\"MR. JAMES NGOLA.\"</td>\n      <td>james_ngola2002@maktoob</td>\n      <td>None</td>\n      <td>31 Oct 2002</td>\n      <td>URGENT BUSINESS ASSISTANCE AND PARTNERSHIP</td>\n      <td>FROM:MR. JAMES NGOLA.\\nCONFIDENTIAL TEL: 233-2...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>bensul2004nng@spinfinder.com&gt;</td>\n      <td>\"Mr. Ben Suleman\"</td>\n      <td>R@M</td>\n      <td>None</td>\n      <td>31 Oct 2002</td>\n      <td>URGENT ASSISTANCE /RELATIONSHIP (P)</td>\n      <td>Dear Friend,\\n\\nI am Mr. Ben Suleman a custom ...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>obong_715@epatra.com&gt;</td>\n      <td>\"PRINCE OBONG ELEME\"</td>\n      <td>obong_715@epatra</td>\n      <td>None</td>\n      <td>31 Oct 2002</td>\n      <td>GOOD DAY TO YOU</td>\n      <td>FROM HIS ROYAL MAJESTY (HRM) CROWN RULER OF EL...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>obong_715@epatra.com&gt;</td>\n      <td>\"PRINCE OBONG ELEME\"</td>\n      <td>webmaster@aclweb</td>\n      <td>None</td>\n      <td>31 Oct 2002</td>\n      <td>GOOD DAY TO YOU</td>\n      <td>FROM HIS ROYAL MAJESTY (HRM) CROWN RULER OF EL...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>m_abacha03@www.com&gt;</td>\n      <td>\"Maryam Abacha\"</td>\n      <td>m_abacha03@www</td>\n      <td>None</td>\n      <td>1 Nov 2002</td>\n      <td>I Need Your Assistance.</td>\n      <td>Dear sir, \\n \\nIt is with a heart full of hope...</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>davidkuta@postmark.net&gt;</td>\n      <td>None</td>\n      <td>davidkuta@yahoo</td>\n      <td>None</td>\n      <td>02 Nov 2002</td>\n      <td>Partnership</td>\n      <td>ATTENTION:                                    ...</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>tunde_dosumu@lycos.com&gt;</td>\n      <td>\"Barrister tunde dosumu\"</td>\n      <td>tunde_dosumu@lycos</td>\n      <td>None</td>\n      <td>None</td>\n      <td>Urgent Attention</td>\n      <td>Dear Sir,\\n\\nI am Barrister Tunde Dosumu (SAN)...</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>william2244drallo@maktoob.com&gt;</td>\n      <td>\"William Drallo\"</td>\n      <td>william2244drallo@maktoob</td>\n      <td>None</td>\n      <td>3 Nov 2002</td>\n      <td>URGENT BUSINESS PRPOSAL</td>\n      <td>FROM: WILLIAM DRALLO.\\nCONFIDENTIAL TEL: 233-2...</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>abdul_817@rediffmail.com&gt;</td>\n      <td>\"MR USMAN ABDUL\"</td>\n      <td>R@M</td>\n      <td>None</td>\n      <td>04 Nov 2002</td>\n      <td>THANK YOU</td>\n      <td>CHALLENGE SECURITIES LTD.\\nLAGOS, NIGERIA\\n\\n\\...</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>barrister_td@lycos.com&gt;</td>\n      <td>\"Tunde  Dosumu\"</td>\n      <td>barrister_td@lycos</td>\n      <td>None</td>\n      <td>None</td>\n      <td>Urgent Assistance</td>\n      <td>Dear Sir,\\n\\nI am Barrister Tunde Dosumu (SAN)...</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>temijohnson2@rediffmail.com&gt;</td>\n      <td>None</td>\n      <td>R@E</td>\n      <td>None</td>\n      <td>6 Nov 2001</td>\n      <td>IMPORTANT</td>\n      <td>FROM: MR TEMI JOHNSON\\nDEMOCRATIC REPUBLIC OF ...</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>sjordan@diplomats.com&gt;</td>\n      <td>\"Dr.Sam jordan\"</td>\n      <td>R@M</td>\n      <td>None</td>\n      <td>08 Nov 2002</td>\n      <td>URGENT ASSISTANCE.</td>\n      <td>FROM THE DESK OF DR.SAM JORDAN\\nTHE MANAGER\\nH...</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>p_brown2@lawyer.com</td>\n      <td>None</td>\n      <td>p_brown2@lawyer</td>\n      <td>None</td>\n      <td>8 Nov 2002</td>\n      <td>From: Barrister Peter Brown</td>\n      <td>\\nSENIOR ADVOCATE OF NIGERIA\\nBARR. PETER BROW...</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>mic_k1@post.com</td>\n      <td>None</td>\n      <td>R@M</td>\n      <td>None</td>\n      <td>11 Nov 2002</td>\n      <td>MICHAEL</td>\n      <td>From;Mr.Michael Kamah and Family,\\n          J...</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>mikebunduu1@rediffmail.com&gt;</td>\n      <td>\"COL. MICHAEL BUNDU\"</td>\n      <td>R@E</td>\n      <td>None</td>\n      <td>13 Nov 2001</td>\n      <td>*****SPAM***** IMPORTANT</td>\n      <td>FROM: COL. MICHAEL BUNDU. \\nDEMOCRATIC REPUBLI...</td>\n    </tr>\n    <tr>\n      <th>15</th>\n      <td>elixwilliam@usa.com&gt;</td>\n      <td>\"MRS MARIAM ABACHA\"</td>\n      <td>presidentwifeng@indiatimes</td>\n      <td>None</td>\n      <td>14 Nov 2002</td>\n      <td>TRUST TRANSACTION</td>\n      <td>Dear Sir,\\n\\nA Very Good day to you   \\n\\nI am...</td>\n    </tr>\n    <tr>\n      <th>16</th>\n      <td>anayoawka@hotmail.com&gt;</td>\n      <td>\" DR. ANAYO AWKA \"</td>\n      <td>webmaster@aclweb</td>\n      <td>None</td>\n      <td>15 Nov 2002</td>\n      <td>REQUEST FOR YOUR UNRESERVED ASSISTANCE</td>\n      <td>FROM: DR. ANAYO AWKA BANK MANAGER \\n(UNION BAN...</td>\n    </tr>\n    <tr>\n      <th>17</th>\n      <td>anayoawka@hotmail.com&gt;</td>\n      <td>\" DR. ANAYO AWKA \"</td>\n      <td>webmaster@aclweb</td>\n      <td>None</td>\n      <td>15 Nov 2002</td>\n      <td>REQUEST FOR YOUR UNRESERVED ASSISTANCE</td>\n      <td>FROM: DR. ANAYO AWKA BANK MANAGER \\n(UNION BAN...</td>\n    </tr>\n    <tr>\n      <th>18</th>\n      <td>victorloma@netscape.net&gt;</td>\n      <td>\"Victor Aloma\"</td>\n      <td>listmem@newns</td>\n      <td>None</td>\n      <td>16 Nov 2002</td>\n      <td>Urgent Assistance</td>\n      <td>The Director,\\n\\n\\n\\n            SEEKING FOR I...</td>\n    </tr>\n    <tr>\n      <th>19</th>\n      <td>victorloma@netscape.net&gt;</td>\n      <td>\"Victor Aloma\"</td>\n      <td>outgoing@mail</td>\n      <td>None</td>\n      <td>16 Nov 2002</td>\n      <td>Urgent Assistance</td>\n      <td>The Director,\\n\\n\\n\\n            SEEKING FOR I...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{},"execution_count":55}]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":[]}]}