pre_process_data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk import word_tokenize, sent_tokenize\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.stem import LancasterStemmer, WordNetLemmatizer\n",
    "file = open(r\"183586_MR20171019-0062_20171025132551.291\", \"r\") \n",
    "x =  file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"OPERATIVE REPORT\\n\\nSURGEON: DOC5, MD\\n1ST ASSISTANT: Dr. Doc6.\\n2ND ASSISTANT:\\nANESTHESIA: General endotracheal.\\nDATE OF SURGERY: 10/19/2017\\n\\n\\nPREOPERATIVE DIAGNOSES: Chronic cholecystitis and cholelithiasis.\\n\\nPOSTOPERATIVE DIAGNOSES: Chronic cholecystitis and cholelithiasis.\\n\\nOPERATION: Robotic assisted single site laparoscopic cholecystectomy with fluorescent\\ncholangiogram.\\n\\nANESTHESIOLOGIST: Doc7, MD\\n\\nDESCRIPTION OF PROCEDURE: The patient was placed supine on the operating table and satisfactory anesthesia was established via endotracheal tube. IV antibiotics were administered prophylactically. A nasogastric tube was inserted and sequential compression devices were applied to the lower extremities. The abdomen was then prepped and draped in the usual sterile fashion. The periumbilical region was infiltrated with 10 cc of 0.5% Marcaine.\\nA vertical incision was made directly in the umbilicus, extending slightly upward in the midline. This was deepened through the subcutaneous tissues and bleeders were controlled with cautery. The midline fascia and peritoneum were then opened for a length of about 5 cm.\\nWith careful palpation, we did not identify any adhesions around the umbilicus from the patient's prior laparoscopic hysterectomy. The single site umbilical port was positioned in the wound and secured in place with nylon sutures. This was connected to CO2 insufflation to an intraperitoneal pressure of 15 mmHg. The camera port and video laparoscope were then inserted.\\nThere was no evidence of any intra-abdominal trauma from the umbilical port insertion. Under laparoscopic vision, curved robotic cannulas were inserted without difficulty.\\nThe patient was placed in reverse Trendelenburg position and tilted leftward. The robotic arms were bend docked to the robotic cannulas, and the robotic arms were then burped together. The laparoscope was reinserted along with robotic instruments. The assist port and assist grasper were also inserted under laparoscopic vision. The gallbladder was easily identified.\\nThere were no adhesions to surrounding tissues. It was somewhat thick walled and chronically inflamed. The gallbladder was doubly grasped and gently elevated. Fluorescent cholangiography demonstrated normal extrahepatic biliary anatomy. Staying high against the gallbladder and well away from the porta hepatis and common bile duct, the cystic duct and the cystic artery were identified and dissected free. These 2 structures were then divided between sets of hemoclips, again staying high against the gallbladder. The gallbladder was then elevated further and was dissected free from the gallbladder fossa in retrograde fashion using a combination of blunt dissection and electrocautery. Any bleeders on the surface of the gallbladder fossa were controlled with cautery. Once the gallbladder was completely dissected free, the operative field was checked. There was no evidence of bleeding or bile leakage and the cystic duct and cystic artery clips appeared to be in proper position. The robotic instruments were removed and the robotic arms were undocked from the cannulas. The patient was then placed back in supine position. The umbilical port, assist port, assist grasper, and the attached gallbladder were then removed from the abdominal cavity. The gallbladder contained multiple palpable stones.\\nThis was submitted to Pathology in fixative. The pneumoperitoneum had been completely released.\\nThe midline fascia was then closed with running suture of 0 Maxon. The wound was irrigated and checked for hemostasis. An additional 10 cc of Marcaine was infiltrated around the fascial closure. The skin incision was then closed with running subcuticular suture of 3-0 plain catgut. Steri-Strips and a dry sterile dressing were applied. The patient was then awakened and extubated and she was moved to the recovery room in stable and satisfactory condition. She tolerated the procedure well. The case was considered clean. The estimated blood loss was less than 10 cc. At the end of the procedure, all counts were correct.\\nPostoperatively the patient and her family were given full wound care, diet, and activity instructions.\\nShe will use\\nPercocet as needed for pain and she will follow up in my office within the next 1 to 2 weeks.\""
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OPERATIVE REPORT  SURGEON: DOC5, MD 1ST ASSISTANT: Dr. Doc6. 2ND ASSISTANT: ANESTHESIA: General endotracheal. DATE OF SURGERY: 10/19/2017   PREOPERATIVE DIAGNOSES: Chronic cholecystitis and cholelithiasis.  POSTOPERATIVE DIAGNOSES: Chronic cholecystitis and cholelithiasis.  OPERATION: Robotic assisted single site laparoscopic cholecystectomy with fluorescent cholangiogram.  ANESTHESIOLOGIST: Doc7, MD  DESCRIPTION OF PROCEDURE: The patient was placed supine on the operating table and satisfactory anesthesia was established via endotracheal tube. IV antibiotics were administered prophylactically. A nasogastric tube was inserted and sequential compression devices were applied to the lower extremities. The abdomen was then prepped and draped in the usual sterile fashion. The periumbilical region was infiltrated with 10 cc of 0.5% Marcaine. A vertical incision was made directly in the umbilicus, extending slightly upward in the midline. This was deepened through the subcutaneous tissues and bleeders were controlled with cautery. The midline fascia and peritoneum were then opened for a length of about 5 cm. With careful palpation, we did not identify any adhesions around the umbilicus from the patient's prior laparoscopic hysterectomy. The single site umbilical port was positioned in the wound and secured in place with nylon sutures. This was connected to CO2 insufflation to an intraperitoneal pressure of 15 mmHg. The camera port and video laparoscope were then inserted. There was no evidence of any intra-abdominal trauma from the umbilical port insertion. Under laparoscopic vision, curved robotic cannulas were inserted without difficulty. The patient was placed in reverse Trendelenburg position and tilted leftward. The robotic arms were bend docked to the robotic cannulas, and the robotic arms were then burped together. The laparoscope was reinserted along with robotic instruments. The assist port and assist grasper were also inserted under laparoscopic vision. The gallbladder was easily identified. There were no adhesions to surrounding tissues. It was somewhat thick walled and chronically inflamed. The gallbladder was doubly grasped and gently elevated. Fluorescent cholangiography demonstrated normal extrahepatic biliary anatomy. Staying high against the gallbladder and well away from the porta hepatis and common bile duct, the cystic duct and the cystic artery were identified and dissected free. These 2 structures were then divided between sets of hemoclips, again staying high against the gallbladder. The gallbladder was then elevated further and was dissected free from the gallbladder fossa in retrograde fashion using a combination of blunt dissection and electrocautery. Any bleeders on the surface of the gallbladder fossa were controlled with cautery. Once the gallbladder was completely dissected free, the operative field was checked. There was no evidence of bleeding or bile leakage and the cystic duct and cystic artery clips appeared to be in proper position. The robotic instruments were removed and the robotic arms were undocked from the cannulas. The patient was then placed back in supine position. The umbilical port, assist port, assist grasper, and the attached gallbladder were then removed from the abdominal cavity. The gallbladder contained multiple palpable stones. This was submitted to Pathology in fixative. The pneumoperitoneum had been completely released. The midline fascia was then closed with running suture of 0 Maxon. The wound was irrigated and checked for hemostasis. An additional 10 cc of Marcaine was infiltrated around the fascial closure. The skin incision was then closed with running subcuticular suture of 3-0 plain catgut. Steri-Strips and a dry sterile dressing were applied. The patient was then awakened and extubated and she was moved to the recovery room in stable and satisfactory condition. She tolerated the procedure well. The case was considered clean. The estimated blood loss was less than 10 cc. At the end of the procedure, all counts were correct. Postoperatively the patient and her family were given full wound care, diet, and activity instructions. She will use Percocet as needed for pain and she will follow up in my office within the next 1 to 2 weeks.\n"
     ]
    }
   ],
   "source": [
    "newx = x.replace(\"\\n\", \" \")\n",
    "print(newx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['OPERATIVE', 'REPORT', 'SURGEON', 'DOC5', 'MD', '1ST', 'ASSISTANT', 'Dr.', 'Doc6', '2ND', 'ASSISTANT', 'ANESTHESIA', 'General', 'endotracheal', 'DATE', 'OF', 'SURGERY', '10/19/2017', 'PREOPERATIVE', 'DIAGNOSES', 'Chronic', 'cholecystitis', 'and', 'cholelithiasis', 'POSTOPERATIVE', 'DIAGNOSES', 'Chronic', 'cholecystitis', 'and', 'cholelithiasis', 'OPERATION', 'Robotic', 'assisted', 'single', 'site', 'laparoscopic', 'cholecystectomy', 'with', 'fluorescent', 'cholangiogram', 'ANESTHESIOLOGIST', 'Doc7', 'MD', 'DESCRIPTION', 'OF', 'PROCEDURE', 'The', 'patient', 'was', 'placed', 'supine', 'on', 'the', 'operating', 'table', 'and', 'satisfactory', 'anesthesia', 'was', 'established', 'via', 'endotracheal', 'tube', 'IV', 'antibiotics', 'were', 'administered', 'prophylactically', 'A', 'nasogastric', 'tube', 'was', 'inserted', 'and', 'sequential', 'compression', 'devices', 'were', 'applied', 'to', 'the', 'lower', 'extremities', 'The', 'abdomen', 'was', 'then', 'prepped', 'and', 'draped', 'in', 'the', 'usual', 'sterile', 'fashion', 'The', 'periumbilical', 'region', 'was', 'infiltrated', 'with', '10', 'cc', 'of', '0.5', 'Marcaine', 'A', 'vertical', 'incision', 'was', 'made', 'directly', 'in', 'the', 'umbilicus', 'extending', 'slightly', 'upward', 'in', 'the', 'midline', 'This', 'was', 'deepened', 'through', 'the', 'subcutaneous', 'tissues', 'and', 'bleeders', 'were', 'controlled', 'with', 'cautery', 'The', 'midline', 'fascia', 'and', 'peritoneum', 'were', 'then', 'opened', 'for', 'a', 'length', 'of', 'about', '5', 'cm', 'With', 'careful', 'palpation', 'we', 'did', 'not', 'identify', 'any', 'adhesions', 'around', 'the', 'umbilicus', 'from', 'the', 'patient', \"'s\", 'prior', 'laparoscopic', 'hysterectomy', 'The', 'single', 'site', 'umbilical', 'port', 'was', 'positioned', 'in', 'the', 'wound', 'and', 'secured', 'in', 'place', 'with', 'nylon', 'sutures', 'This', 'was', 'connected', 'to', 'CO2', 'insufflation', 'to', 'an', 'intraperitoneal', 'pressure', 'of', '15', 'mmHg', 'The', 'camera', 'port', 'and', 'video', 'laparoscope', 'were', 'then', 'inserted', 'There', 'was', 'no', 'evidence', 'of', 'any', 'intra-abdominal', 'trauma', 'from', 'the', 'umbilical', 'port', 'insertion', 'Under', 'laparoscopic', 'vision', 'curved', 'robotic', 'cannulas', 'were', 'inserted', 'without', 'difficulty', 'The', 'patient', 'was', 'placed', 'in', 'reverse', 'Trendelenburg', 'position', 'and', 'tilted', 'leftward', 'The', 'robotic', 'arms', 'were', 'bend', 'docked', 'to', 'the', 'robotic', 'cannulas', 'and', 'the', 'robotic', 'arms', 'were', 'then', 'burped', 'together', 'The', 'laparoscope', 'was', 'reinserted', 'along', 'with', 'robotic', 'instruments', 'The', 'assist', 'port', 'and', 'assist', 'grasper', 'were', 'also', 'inserted', 'under', 'laparoscopic', 'vision', 'The', 'gallbladder', 'was', 'easily', 'identified', 'There', 'were', 'no', 'adhesions', 'to', 'surrounding', 'tissues', 'It', 'was', 'somewhat', 'thick', 'walled', 'and', 'chronically', 'inflamed', 'The', 'gallbladder', 'was', 'doubly', 'grasped', 'and', 'gently', 'elevated', 'Fluorescent', 'cholangiography', 'demonstrated', 'normal', 'extrahepatic', 'biliary', 'anatomy', 'Staying', 'high', 'against', 'the', 'gallbladder', 'and', 'well', 'away', 'from', 'the', 'porta', 'hepatis', 'and', 'common', 'bile', 'duct', 'the', 'cystic', 'duct', 'and', 'the', 'cystic', 'artery', 'were', 'identified', 'and', 'dissected', 'free', 'These', '2', 'structures', 'were', 'then', 'divided', 'between', 'sets', 'of', 'hemoclips', 'again', 'staying', 'high', 'against', 'the', 'gallbladder', 'The', 'gallbladder', 'was', 'then', 'elevated', 'further', 'and', 'was', 'dissected', 'free', 'from', 'the', 'gallbladder', 'fossa', 'in', 'retrograde', 'fashion', 'using', 'a', 'combination', 'of', 'blunt', 'dissection', 'and', 'electrocautery', 'Any', 'bleeders', 'on', 'the', 'surface', 'of', 'the', 'gallbladder', 'fossa', 'were', 'controlled', 'with', 'cautery', 'Once', 'the', 'gallbladder', 'was', 'completely', 'dissected', 'free', 'the', 'operative', 'field', 'was', 'checked', 'There', 'was', 'no', 'evidence', 'of', 'bleeding', 'or', 'bile', 'leakage', 'and', 'the', 'cystic', 'duct', 'and', 'cystic', 'artery', 'clips', 'appeared', 'to', 'be', 'in', 'proper', 'position', 'The', 'robotic', 'instruments', 'were', 'removed', 'and', 'the', 'robotic', 'arms', 'were', 'undocked', 'from', 'the', 'cannulas', 'The', 'patient', 'was', 'then', 'placed', 'back', 'in', 'supine', 'position', 'The', 'umbilical', 'port', 'assist', 'port', 'assist', 'grasper', 'and', 'the', 'attached', 'gallbladder', 'were', 'then', 'removed', 'from', 'the', 'abdominal', 'cavity', 'The', 'gallbladder', 'contained', 'multiple', 'palpable', 'stones', 'This', 'was', 'submitted', 'to', 'Pathology', 'in', 'fixative', 'The', 'pneumoperitoneum', 'had', 'been', 'completely', 'released', 'The', 'midline', 'fascia', 'was', 'then', 'closed', 'with', 'running', 'suture', 'of', '0', 'Maxon', 'The', 'wound', 'was', 'irrigated', 'and', 'checked', 'for', 'hemostasis', 'An', 'additional', '10', 'cc', 'of', 'Marcaine', 'was', 'infiltrated', 'around', 'the', 'fascial', 'closure', 'The', 'skin', 'incision', 'was', 'then', 'closed', 'with', 'running', 'subcuticular', 'suture', 'of', '3-0', 'plain', 'catgut', 'Steri-Strips', 'and', 'a', 'dry', 'sterile', 'dressing', 'were', 'applied', 'The', 'patient', 'was', 'then', 'awakened', 'and', 'extubated', 'and', 'she', 'was', 'moved', 'to', 'the', 'recovery', 'room', 'in', 'stable', 'and', 'satisfactory', 'condition', 'She', 'tolerated', 'the', 'procedure', 'well', 'The', 'case', 'was', 'considered', 'clean', 'The', 'estimated', 'blood', 'loss', 'was', 'less', 'than', '10', 'cc', 'At', 'the', 'end', 'of', 'the', 'procedure', 'all', 'counts', 'were', 'correct', 'Postoperatively', 'the', 'patient', 'and', 'her', 'family', 'were', 'given', 'full', 'wound', 'care', 'diet', 'and', 'activity', 'instructions', 'She', 'will', 'use', 'Percocet', 'as', 'needed', 'for', 'pain', 'and', 'she', 'will', 'follow', 'up', 'in', 'my', 'office', 'within', 'the', 'next', '1', 'to', '2', 'weeks']\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "import string\n",
    "ret = re.sub(\"-\\s\",\"-\", newx)\n",
    "l = nltk.word_tokenize(ret)\n",
    "tokens = [x for x in l if not re.fullmatch('[' + string.punctuation + ']+', x)]\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['operative',\n",
       " 'report',\n",
       " 'surgeon',\n",
       " 'doc5',\n",
       " 'md',\n",
       " '1st',\n",
       " 'assistant',\n",
       " 'dr.',\n",
       " 'doc6',\n",
       " '2nd',\n",
       " 'assistant',\n",
       " 'anesthesia',\n",
       " 'general',\n",
       " 'endotracheal',\n",
       " 'date',\n",
       " 'of',\n",
       " 'surgery',\n",
       " '10/19/2017',\n",
       " 'preoperative',\n",
       " 'diagnoses',\n",
       " 'chronic',\n",
       " 'cholecystitis',\n",
       " 'and',\n",
       " 'cholelithiasis',\n",
       " 'postoperative',\n",
       " 'diagnoses',\n",
       " 'chronic',\n",
       " 'cholecystitis',\n",
       " 'and',\n",
       " 'cholelithiasis',\n",
       " 'operation',\n",
       " 'robotic',\n",
       " 'assisted',\n",
       " 'single',\n",
       " 'site',\n",
       " 'laparoscopic',\n",
       " 'cholecystectomy',\n",
       " 'with',\n",
       " 'fluorescent',\n",
       " 'cholangiogram',\n",
       " 'anesthesiologist',\n",
       " 'doc7',\n",
       " 'md',\n",
       " 'description',\n",
       " 'of',\n",
       " 'procedure',\n",
       " 'the',\n",
       " 'patient',\n",
       " 'was',\n",
       " 'placed',\n",
       " 'supine',\n",
       " 'on',\n",
       " 'the',\n",
       " 'operating',\n",
       " 'table',\n",
       " 'and',\n",
       " 'satisfactory',\n",
       " 'anesthesia',\n",
       " 'was',\n",
       " 'established',\n",
       " 'via',\n",
       " 'endotracheal',\n",
       " 'tube',\n",
       " 'iv',\n",
       " 'antibiotics',\n",
       " 'were',\n",
       " 'administered',\n",
       " 'prophylactically',\n",
       " 'a',\n",
       " 'nasogastric',\n",
       " 'tube',\n",
       " 'was',\n",
       " 'inserted',\n",
       " 'and',\n",
       " 'sequential',\n",
       " 'compression',\n",
       " 'devices',\n",
       " 'were',\n",
       " 'applied',\n",
       " 'to',\n",
       " 'the',\n",
       " 'lower',\n",
       " 'extremities',\n",
       " 'the',\n",
       " 'abdomen',\n",
       " 'was',\n",
       " 'then',\n",
       " 'prepped',\n",
       " 'and',\n",
       " 'draped',\n",
       " 'in',\n",
       " 'the',\n",
       " 'usual',\n",
       " 'sterile',\n",
       " 'fashion',\n",
       " 'the',\n",
       " 'periumbilical',\n",
       " 'region',\n",
       " 'was',\n",
       " 'infiltrated',\n",
       " 'with',\n",
       " '10',\n",
       " 'cc',\n",
       " 'of',\n",
       " '0.5',\n",
       " 'marcaine',\n",
       " 'a',\n",
       " 'vertical',\n",
       " 'incision',\n",
       " 'was',\n",
       " 'made',\n",
       " 'directly',\n",
       " 'in',\n",
       " 'the',\n",
       " 'umbilicus',\n",
       " 'extending',\n",
       " 'slightly',\n",
       " 'upward',\n",
       " 'in',\n",
       " 'the',\n",
       " 'midline',\n",
       " 'this',\n",
       " 'was',\n",
       " 'deepened',\n",
       " 'through',\n",
       " 'the',\n",
       " 'subcutaneous',\n",
       " 'tissues',\n",
       " 'and',\n",
       " 'bleeders',\n",
       " 'were',\n",
       " 'controlled',\n",
       " 'with',\n",
       " 'cautery',\n",
       " 'the',\n",
       " 'midline',\n",
       " 'fascia',\n",
       " 'and',\n",
       " 'peritoneum',\n",
       " 'were',\n",
       " 'then',\n",
       " 'opened',\n",
       " 'for',\n",
       " 'a',\n",
       " 'length',\n",
       " 'of',\n",
       " 'about',\n",
       " '5',\n",
       " 'cm',\n",
       " 'with',\n",
       " 'careful',\n",
       " 'palpation',\n",
       " 'we',\n",
       " 'did',\n",
       " 'not',\n",
       " 'identify',\n",
       " 'any',\n",
       " 'adhesions',\n",
       " 'around',\n",
       " 'the',\n",
       " 'umbilicus',\n",
       " 'from',\n",
       " 'the',\n",
       " 'patient',\n",
       " \"'s\",\n",
       " 'prior',\n",
       " 'laparoscopic',\n",
       " 'hysterectomy',\n",
       " 'the',\n",
       " 'single',\n",
       " 'site',\n",
       " 'umbilical',\n",
       " 'port',\n",
       " 'was',\n",
       " 'positioned',\n",
       " 'in',\n",
       " 'the',\n",
       " 'wound',\n",
       " 'and',\n",
       " 'secured',\n",
       " 'in',\n",
       " 'place',\n",
       " 'with',\n",
       " 'nylon',\n",
       " 'sutures',\n",
       " 'this',\n",
       " 'was',\n",
       " 'connected',\n",
       " 'to',\n",
       " 'co2',\n",
       " 'insufflation',\n",
       " 'to',\n",
       " 'an',\n",
       " 'intraperitoneal',\n",
       " 'pressure',\n",
       " 'of',\n",
       " '15',\n",
       " 'mmhg',\n",
       " 'the',\n",
       " 'camera',\n",
       " 'port',\n",
       " 'and',\n",
       " 'video',\n",
       " 'laparoscope',\n",
       " 'were',\n",
       " 'then',\n",
       " 'inserted',\n",
       " 'there',\n",
       " 'was',\n",
       " 'no',\n",
       " 'evidence',\n",
       " 'of',\n",
       " 'any',\n",
       " 'intra-abdominal',\n",
       " 'trauma',\n",
       " 'from',\n",
       " 'the',\n",
       " 'umbilical',\n",
       " 'port',\n",
       " 'insertion',\n",
       " 'under',\n",
       " 'laparoscopic',\n",
       " 'vision',\n",
       " 'curved',\n",
       " 'robotic',\n",
       " 'cannulas',\n",
       " 'were',\n",
       " 'inserted',\n",
       " 'without',\n",
       " 'difficulty',\n",
       " 'the',\n",
       " 'patient',\n",
       " 'was',\n",
       " 'placed',\n",
       " 'in',\n",
       " 'reverse',\n",
       " 'trendelenburg',\n",
       " 'position',\n",
       " 'and',\n",
       " 'tilted',\n",
       " 'leftward',\n",
       " 'the',\n",
       " 'robotic',\n",
       " 'arms',\n",
       " 'were',\n",
       " 'bend',\n",
       " 'docked',\n",
       " 'to',\n",
       " 'the',\n",
       " 'robotic',\n",
       " 'cannulas',\n",
       " 'and',\n",
       " 'the',\n",
       " 'robotic',\n",
       " 'arms',\n",
       " 'were',\n",
       " 'then',\n",
       " 'burped',\n",
       " 'together',\n",
       " 'the',\n",
       " 'laparoscope',\n",
       " 'was',\n",
       " 'reinserted',\n",
       " 'along',\n",
       " 'with',\n",
       " 'robotic',\n",
       " 'instruments',\n",
       " 'the',\n",
       " 'assist',\n",
       " 'port',\n",
       " 'and',\n",
       " 'assist',\n",
       " 'grasper',\n",
       " 'were',\n",
       " 'also',\n",
       " 'inserted',\n",
       " 'under',\n",
       " 'laparoscopic',\n",
       " 'vision',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'was',\n",
       " 'easily',\n",
       " 'identified',\n",
       " 'there',\n",
       " 'were',\n",
       " 'no',\n",
       " 'adhesions',\n",
       " 'to',\n",
       " 'surrounding',\n",
       " 'tissues',\n",
       " 'it',\n",
       " 'was',\n",
       " 'somewhat',\n",
       " 'thick',\n",
       " 'walled',\n",
       " 'and',\n",
       " 'chronically',\n",
       " 'inflamed',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'was',\n",
       " 'doubly',\n",
       " 'grasped',\n",
       " 'and',\n",
       " 'gently',\n",
       " 'elevated',\n",
       " 'fluorescent',\n",
       " 'cholangiography',\n",
       " 'demonstrated',\n",
       " 'normal',\n",
       " 'extrahepatic',\n",
       " 'biliary',\n",
       " 'anatomy',\n",
       " 'staying',\n",
       " 'high',\n",
       " 'against',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'and',\n",
       " 'well',\n",
       " 'away',\n",
       " 'from',\n",
       " 'the',\n",
       " 'porta',\n",
       " 'hepatis',\n",
       " 'and',\n",
       " 'common',\n",
       " 'bile',\n",
       " 'duct',\n",
       " 'the',\n",
       " 'cystic',\n",
       " 'duct',\n",
       " 'and',\n",
       " 'the',\n",
       " 'cystic',\n",
       " 'artery',\n",
       " 'were',\n",
       " 'identified',\n",
       " 'and',\n",
       " 'dissected',\n",
       " 'free',\n",
       " 'these',\n",
       " '2',\n",
       " 'structures',\n",
       " 'were',\n",
       " 'then',\n",
       " 'divided',\n",
       " 'between',\n",
       " 'sets',\n",
       " 'of',\n",
       " 'hemoclips',\n",
       " 'again',\n",
       " 'staying',\n",
       " 'high',\n",
       " 'against',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'was',\n",
       " 'then',\n",
       " 'elevated',\n",
       " 'further',\n",
       " 'and',\n",
       " 'was',\n",
       " 'dissected',\n",
       " 'free',\n",
       " 'from',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'fossa',\n",
       " 'in',\n",
       " 'retrograde',\n",
       " 'fashion',\n",
       " 'using',\n",
       " 'a',\n",
       " 'combination',\n",
       " 'of',\n",
       " 'blunt',\n",
       " 'dissection',\n",
       " 'and',\n",
       " 'electrocautery',\n",
       " 'any',\n",
       " 'bleeders',\n",
       " 'on',\n",
       " 'the',\n",
       " 'surface',\n",
       " 'of',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'fossa',\n",
       " 'were',\n",
       " 'controlled',\n",
       " 'with',\n",
       " 'cautery',\n",
       " 'once',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'was',\n",
       " 'completely',\n",
       " 'dissected',\n",
       " 'free',\n",
       " 'the',\n",
       " 'operative',\n",
       " 'field',\n",
       " 'was',\n",
       " 'checked',\n",
       " 'there',\n",
       " 'was',\n",
       " 'no',\n",
       " 'evidence',\n",
       " 'of',\n",
       " 'bleeding',\n",
       " 'or',\n",
       " 'bile',\n",
       " 'leakage',\n",
       " 'and',\n",
       " 'the',\n",
       " 'cystic',\n",
       " 'duct',\n",
       " 'and',\n",
       " 'cystic',\n",
       " 'artery',\n",
       " 'clips',\n",
       " 'appeared',\n",
       " 'to',\n",
       " 'be',\n",
       " 'in',\n",
       " 'proper',\n",
       " 'position',\n",
       " 'the',\n",
       " 'robotic',\n",
       " 'instruments',\n",
       " 'were',\n",
       " 'removed',\n",
       " 'and',\n",
       " 'the',\n",
       " 'robotic',\n",
       " 'arms',\n",
       " 'were',\n",
       " 'undocked',\n",
       " 'from',\n",
       " 'the',\n",
       " 'cannulas',\n",
       " 'the',\n",
       " 'patient',\n",
       " 'was',\n",
       " 'then',\n",
       " 'placed',\n",
       " 'back',\n",
       " 'in',\n",
       " 'supine',\n",
       " 'position',\n",
       " 'the',\n",
       " 'umbilical',\n",
       " 'port',\n",
       " 'assist',\n",
       " 'port',\n",
       " 'assist',\n",
       " 'grasper',\n",
       " 'and',\n",
       " 'the',\n",
       " 'attached',\n",
       " 'gallbladder',\n",
       " 'were',\n",
       " 'then',\n",
       " 'removed',\n",
       " 'from',\n",
       " 'the',\n",
       " 'abdominal',\n",
       " 'cavity',\n",
       " 'the',\n",
       " 'gallbladder',\n",
       " 'contained',\n",
       " 'multiple',\n",
       " 'palpable',\n",
       " 'stones',\n",
       " 'this',\n",
       " 'was',\n",
       " 'submitted',\n",
       " 'to',\n",
       " 'pathology',\n",
       " 'in',\n",
       " 'fixative',\n",
       " 'the',\n",
       " 'pneumoperitoneum',\n",
       " 'had',\n",
       " 'been',\n",
       " 'completely',\n",
       " 'released',\n",
       " 'the',\n",
       " 'midline',\n",
       " 'fascia',\n",
       " 'was',\n",
       " 'then',\n",
       " 'closed',\n",
       " 'with',\n",
       " 'running',\n",
       " 'suture',\n",
       " 'of',\n",
       " '0',\n",
       " 'maxon',\n",
       " 'the',\n",
       " 'wound',\n",
       " 'was',\n",
       " 'irrigated',\n",
       " 'and',\n",
       " 'checked',\n",
       " 'for',\n",
       " 'hemostasis',\n",
       " 'an',\n",
       " 'additional',\n",
       " '10',\n",
       " 'cc',\n",
       " 'of',\n",
       " 'marcaine',\n",
       " 'was',\n",
       " 'infiltrated',\n",
       " 'around',\n",
       " 'the',\n",
       " 'fascial',\n",
       " 'closure',\n",
       " 'the',\n",
       " 'skin',\n",
       " 'incision',\n",
       " 'was',\n",
       " 'then',\n",
       " 'closed',\n",
       " 'with',\n",
       " 'running',\n",
       " 'subcuticular',\n",
       " 'suture',\n",
       " 'of',\n",
       " '3-0',\n",
       " 'plain',\n",
       " 'catgut',\n",
       " 'steri-strips',\n",
       " 'and',\n",
       " 'a',\n",
       " 'dry',\n",
       " 'sterile',\n",
       " 'dressing',\n",
       " 'were',\n",
       " 'applied',\n",
       " 'the',\n",
       " 'patient',\n",
       " 'was',\n",
       " 'then',\n",
       " 'awakened',\n",
       " 'and',\n",
       " 'extubated',\n",
       " 'and',\n",
       " 'she',\n",
       " 'was',\n",
       " 'moved',\n",
       " 'to',\n",
       " 'the',\n",
       " 'recovery',\n",
       " 'room',\n",
       " 'in',\n",
       " 'stable',\n",
       " 'and',\n",
       " 'satisfactory',\n",
       " 'condition',\n",
       " 'she',\n",
       " 'tolerated',\n",
       " 'the',\n",
       " 'procedure',\n",
       " 'well',\n",
       " 'the',\n",
       " 'case',\n",
       " 'was',\n",
       " 'considered',\n",
       " 'clean',\n",
       " 'the',\n",
       " 'estimated',\n",
       " 'blood',\n",
       " 'loss',\n",
       " 'was',\n",
       " 'less',\n",
       " 'than',\n",
       " '10',\n",
       " 'cc',\n",
       " 'at',\n",
       " 'the',\n",
       " 'end',\n",
       " 'of',\n",
       " 'the',\n",
       " 'procedure',\n",
       " 'all',\n",
       " 'counts',\n",
       " 'were',\n",
       " 'correct',\n",
       " 'postoperatively',\n",
       " 'the',\n",
       " 'patient',\n",
       " 'and',\n",
       " 'her',\n",
       " 'family',\n",
       " 'were',\n",
       " 'given',\n",
       " 'full',\n",
       " 'wound',\n",
       " 'care',\n",
       " 'diet',\n",
       " 'and',\n",
       " 'activity',\n",
       " 'instructions',\n",
       " 'she',\n",
       " 'will',\n",
       " 'use',\n",
       " 'percocet',\n",
       " 'as',\n",
       " 'needed',\n",
       " 'for',\n",
       " 'pain',\n",
       " 'and',\n",
       " 'she',\n",
       " 'will',\n",
       " 'follow',\n",
       " 'up',\n",
       " 'in',\n",
       " 'my',\n",
       " 'office',\n",
       " 'within',\n",
       " 'the',\n",
       " 'next',\n",
       " '1',\n",
       " 'to',\n",
       " '2',\n",
       " 'weeks']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = [word.lower() for word in tokens]\n",
    "words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "med_remove =['a', 'about', 'all', 'almost', 'also', 'although', 'always', 'among', 'an', 'and', 'another', 'any', 'are', 'as', 'at',\n",
    "         'be', 'because', 'been', 'being', 'between', 'both', 'but', 'by','can', 'could','did', 'do', 'does', 'done', 'due',\n",
    "          'each', 'either', 'enough', 'especially', 'etc','for', 'found', 'from', 'further','had', 'has', 'have', 'having', 'here', 'how', 'however',\n",
    "         'i', 'if', 'in', 'into', 'is', 'it', 'its', 'itself','just','kg','km','made', 'mainly', 'make', 'may', 'mg', 'might', 'ml', 'mm', 'most', 'mostly', 'must',\n",
    "         'nearly', 'neither', 'nor','obtained', 'of', 'often', 'on', 'our', 'overall','perhaps', 'pmid','quite','rather', 'really', 'regarding',\n",
    "         'seem', 'seen', 'several', 'should', 'show', 'showed', 'shown', 'shows', 'significantly', 'since', 'so', 'some', 'such',\n",
    "         'than', 'that', 'the', 'their', 'theirs', 'them', 'then', 'there', 'therefore', 'these', 'they', 'this', 'those', 'through', 'thus', 'to',\n",
    "         'upon', 'use', 'used', 'using','various', 'very','was', 'we', 'were', 'what', 'when', 'which', 'while', 'with', 'within', 'without', 'would']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_stopwords = set(stopwords.words('english')+med_remove) - {'after','again','before','no','during','not'} \n",
    "words = [w for w in words if not w in new_stopwords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "operative report surgeon doc5 md 1st assistant dr. doc6 2nd assistant anesthesia general endotracheal date surgery 10/19/2017 preoperative diagnosis chronic cholecystitis cholelithiasis postoperative diagnosis chronic cholecystitis cholelithiasis operation robotic assisted single site laparoscopic cholecystectomy fluorescent cholangiogram anesthesiologist doc7 md description procedure patient placed supine operating table satisfactory anesthesia established via endotracheal tube iv antibiotic administered prophylactically nasogastric tube inserted sequential compression device applied lower extremity abdomen prepped draped usual sterile fashion periumbilical region infiltrated 10 cc 0.5 marcaine vertical incision directly umbilicus extending slightly upward midline deepened subcutaneous tissue bleeder controlled cautery midline fascia peritoneum opened length 5 cm careful palpation not identify adhesion around umbilicus patient 's prior laparoscopic hysterectomy single site umbilical port positioned wound secured place nylon suture connected co2 insufflation intraperitoneal pressure 15 mmhg camera port video laparoscope inserted no evidence intra-abdominal trauma umbilical port insertion laparoscopic vision curved robotic cannula inserted difficulty patient placed reverse trendelenburg position tilted leftward robotic arm bend docked robotic cannula robotic arm burped together laparoscope reinserted along robotic instrument assist port assist grasper inserted laparoscopic vision gallbladder easily identified no adhesion surrounding tissue somewhat thick walled chronically inflamed gallbladder doubly grasped gently elevated fluorescent cholangiography demonstrated normal extrahepatic biliary anatomy staying high gallbladder well away porta hepatis common bile duct cystic duct cystic artery identified dissected free 2 structure divided set hemoclips again staying high gallbladder gallbladder elevated dissected free gallbladder fossa retrograde fashion combination blunt dissection electrocautery bleeder surface gallbladder fossa controlled cautery gallbladder completely dissected free operative field checked no evidence bleeding bile leakage cystic duct cystic artery clip appeared proper position robotic instrument removed robotic arm undocked cannula patient placed back supine position umbilical port assist port assist grasper attached gallbladder removed abdominal cavity gallbladder contained multiple palpable stone submitted pathology fixative pneumoperitoneum completely released midline fascia closed running suture 0 maxon wound irrigated checked hemostasis additional 10 cc marcaine infiltrated around fascial closure skin incision closed running subcuticular suture 3-0 plain catgut steri-strips dry sterile dressing applied patient awakened extubated moved recovery room stable satisfactory condition tolerated procedure well case considered clean estimated blood loss le 10 cc end procedure count correct postoperatively patient family given full wound care diet activity instruction percocet needed pain follow office next 1 2 week\n"
     ]
    }
   ],
   "source": [
    "lemmatizer = WordNetLemmatizer()\n",
    "lemmatized_output = ' '.join([lemmatizer.lemmatize(w) for w in words])\n",
    "print(lemmatized_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "ret = re.sub(\"\\s'\",\"'\", lemmatized_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"operative report surgeon doc5 md 1st assistant dr. doc6 2nd assistant anesthesia general endotracheal date surgery 10/19/2017 preoperative diagnosis chronic cholecystitis cholelithiasis postoperative diagnosis chronic cholecystitis cholelithiasis operation robotic assisted single site laparoscopic cholecystectomy fluorescent cholangiogram anesthesiologist doc7 md description procedure patient placed supine operating table satisfactory anesthesia established via endotracheal tube iv antibiotic administered prophylactically nasogastric tube inserted sequential compression device applied lower extremity abdomen prepped draped usual sterile fashion periumbilical region infiltrated 10 cc 0.5 marcaine vertical incision directly umbilicus extending slightly upward midline deepened subcutaneous tissue bleeder controlled cautery midline fascia peritoneum opened length 5 cm careful palpation not identify adhesion around umbilicus patient's prior laparoscopic hysterectomy single site umbilical port positioned wound secured place nylon suture connected co2 insufflation intraperitoneal pressure 15 mmhg camera port video laparoscope inserted no evidence intra-abdominal trauma umbilical port insertion laparoscopic vision curved robotic cannula inserted difficulty patient placed reverse trendelenburg position tilted leftward robotic arm bend docked robotic cannula robotic arm burped together laparoscope reinserted along robotic instrument assist port assist grasper inserted laparoscopic vision gallbladder easily identified no adhesion surrounding tissue somewhat thick walled chronically inflamed gallbladder doubly grasped gently elevated fluorescent cholangiography demonstrated normal extrahepatic biliary anatomy staying high gallbladder well away porta hepatis common bile duct cystic duct cystic artery identified dissected free 2 structure divided set hemoclips again staying high gallbladder gallbladder elevated dissected free gallbladder fossa retrograde fashion combination blunt dissection electrocautery bleeder surface gallbladder fossa controlled cautery gallbladder completely dissected free operative field checked no evidence bleeding bile leakage cystic duct cystic artery clip appeared proper position robotic instrument removed robotic arm undocked cannula patient placed back supine position umbilical port assist port assist grasper attached gallbladder removed abdominal cavity gallbladder contained multiple palpable stone submitted pathology fixative pneumoperitoneum completely released midline fascia closed running suture 0 maxon wound irrigated checked hemostasis additional 10 cc marcaine infiltrated around fascial closure skin incision closed running subcuticular suture 3-0 plain catgut steri-strips dry sterile dressing applied patient awakened extubated moved recovery room stable satisfactory condition tolerated procedure well case considered clean estimated blood loss le 10 cc end procedure count correct postoperatively patient family given full wound care diet activity instruction percocet needed pain follow office next 1 2 week\""
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ret"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}