predicting-interview-attendence.ipynb

{"cells":[{"metadata":{"_cell_guid":"ab740dc8-fe3c-4e67-be35-4283f8c5d259","_uuid":"c5d5e60b2df203d561ef6e3f810a039c013fd6c1","collapsed":true},"cell_type":"markdown","source":"# Can We Predict Who Will Show for Interviews"},{"metadata":{"_cell_guid":"99a6cc1c-1f7c-40aa-967d-9e9551ddefb9","_uuid":"154c17483652d914742e913447fa5c2d2128885d"},"cell_type":"markdown","source":"I've been looking for a small data set to practice some machine learning techniques I've picked up in the fastai machine learning [course](http://course.fast.ai/). Specifically, the machine learning course goes into a lot of detail on random forest methods and how to go about interpreting and tuning them. So without further ado, let's dive into this new data set and see what we can do."},{"metadata":{"_cell_guid":"b9035f0e-f34d-41b9-b187-268f3c3a6cf5","_uuid":"89ac58a469b77b3b616b3e060a49d3b3fce41969"},"cell_type":"markdown","source":"#### Package loading"},{"metadata":{"_cell_guid":"e4de7889-3d31-40de-a021-4c3d9c2dab2e","_uuid":"7d26901eaccd666a6c30d3673362aab72e223a40","collapsed":true,"trusted":true},"cell_type":"code","source":"%load_ext autoreload\n%autoreload 2\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"66d59846-325f-4739-8e29-b9d2d416981c","_uuid":"e72a5f2a02b3e0fee3e6946f21e4f9bd38905f4b","collapsed":true,"trusted":true},"cell_type":"code","source":"from fastai.imports import *\nfrom fastai.structured import *\n\nfrom pandas_summary import DataFrameSummary\nfrom sklearn.ensemble import RandomForestClassifier\nfrom IPython.display import display\n\nfrom sklearn import metrics","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9dcc0211-149e-4dfc-a643-73e2b94240d4","_uuid":"c1d5c2241b1cdb30b5bb789a78c1b1b94bb0ae1b"},"cell_type":"markdown","source":"## Data Import and Cleaning"},{"metadata":{"_cell_guid":"fc175002-3e42-42f3-b290-b5fcc95a27cd","_uuid":"be183f720900b92213ec75d27f6f7a3e3c3b139e"},"cell_type":"markdown","source":"My data for this comes from kaggle. It's a user submitted [data set](https://www.kaggle.com/vishnusraghavan/the-interview-attendance-problem/data), not a competition set, so there's no prize money, but there's a lot of opportunity for exploration."},{"metadata":{"_cell_guid":"f501fb9d-8069-431e-a29d-2fa195e865dc","_uuid":"66535790b08d5a1cd8244619ae51bd11f4626903","collapsed":true,"trusted":true},"cell_type":"code","source":"#PATH = \"data/\"","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8eaa4d02-a220-4c75-8732-d6172616a090","_uuid":"5a1c28a6f15b1792d13307a9e57dc9e6be245507","collapsed":true,"trusted":true},"cell_type":"code","source":"#!ls {PATH}","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4246bd8d-affa-445a-a84d-14b833a0a165","_uuid":"90842dc5bada2e544b0ff10226d84810c3632757"},"cell_type":"markdown","source":"I can read it in with pandas and take a look at the first couple rows to get a sense of what I am working with."},{"metadata":{"_cell_guid":"c75ed910-bb11-436b-a0f7-56802f8ee593","_uuid":"6841fdcdb4a14caf0b092a9e000ef8a8a3d5e590","collapsed":true,"trusted":true},"cell_type":"code","source":"# Read in the data\n#df_raw = pd.read_csv(f'{PATH}Interview.csv', parse_dates = ['Date of Interview'])\n\n#Alt read in for Kaggle kernal\ndf_raw = pd.read_csv('../input/Interview.csv')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f980b208-612e-412e-b9a6-5ecf6b024e00","_uuid":"b02f1bcadf0fca172b22aa4b5b188c6334265597","collapsed":true,"trusted":true},"cell_type":"code","source":"df_raw.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9ec3cfee-bdc2-4c50-838b-e596bfe22d03","_uuid":"89a16dc2ef051ce847a3b37593b38f74cff833ca"},"cell_type":"markdown","source":"First thing I notice is that we have 5 unnamed, nan filled columns. I'm guessing this is related to how the csv file was created, but regardless we aren't going to want those empty columns, so let's get rid of them straight away.\n\nSecond, the column names are kind of absurdly long. Let's go ahead and clean those up just to make subsequent data manipulation have a little less typing."},{"metadata":{"_cell_guid":"3c3ab439-c2ed-4875-a93d-5ac6d095c6df","_uuid":"fe9c897a26e252c41c23ccad58532dd521deaf7e","collapsed":true,"trusted":true},"cell_type":"code","source":"# Removing empty variables\n# I'll go ahead and put all this work in a new df so I have an original copy if I need to go back for any reason.\ninterview_df = df_raw.drop(['Unnamed: 23', 'Unnamed: 24', 'Unnamed: 25', 'Unnamed: 26', 'Unnamed: 27'], axis = 1)\n\n# Renaming variables to strings that are a little easier to work with.\ninterview_df.columns = ['Date', 'Client', 'Industry', 'Location', 'Position', 'Skillset',\n                        'Interview_Type', 'ID', 'Gender', 'Cand_Loc', 'Job_Loc', 'Venue',\n                        'Native_Loc', 'Permission', 'Hope', 'Three_hour_call', 'Alt_phone',\n                        'Resume_Printout', 'Clarify_Venue', 'Shared_Letter', 'Expected', \n                        'Attended', 'Martial_Status']\ninterview_df.shape","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"41e79968-7af0-4245-b397-fa1393fb6fe3","_uuid":"9698e2a9ef29597b91eda293659c581f46b4255f"},"cell_type":"markdown","source":"This leaves us with a relatively small data frame, so initially I assumed this was going to be pretty straightforward. Maybe a little cleaning of the data and then straight into building some random forests. For good measure though I decided to loop through all the variables and just take a quick look at the unique values in each variable. This can give you a good idea of what sort of cleaning is going to be necessary. It's not a great thing to do on an enormous data set, perhaps, but for something this small it's very quick to do."},{"metadata":{"_cell_guid":"4271bce2-594a-4a44-ae5b-e1dad60433ef","_uuid":"d5bf745dd4ea9ad22d17408f9572fe71be1f2875","collapsed":true,"trusted":true},"cell_type":"code","source":"for c in interview_df.columns:\n    print(c)\n    print(interview_df[c].unique())","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"65894755-ed3f-4d25-a3ef-a49802135509","_uuid":"3f96ea44cd90215e6e9418ebb96196138c0eead2"},"cell_type":"markdown","source":"Oh boy! Maybe this isn't going to be so quick after all. Pretty much every variable is encoded in an inconsistent manner. Dates are separated by /, ., - just to name a few. Sometimes there is a time tacked onto the date as well. Skillset is similarly a mess. It looks like some rows have multiple skills (with various separators, of course) and some rows have times in them for some reason.\n\nThere are also going to be a lot of judgment calls on how to clean this up. For instance, in the Interview_Type column, is 'scheduled walk in' actually different from a 'walk in'? Similar decisions are going to have to be made on Shared_Letter, Resume_Printout, etc. Basically, though this data set is tiny, it's going to require a ton of cleaning.  I guess I should dive in."},{"metadata":{"_cell_guid":"6f0a3166-b11b-421d-bb2f-7cd80a13f684","_uuid":"bce2cb2eac1f6f40247eaac7ada8369500aeaea3"},"cell_type":"markdown","source":"I'm starting with the Date variable, so I wanted to print out it's unique values to see what I was working with a little more clearly. It looks like I will need to get rid of a bunch of whitespace, decide on a separator for the date and convert all the inconsistent ones and then remove some random times that made it into the fray.  This actually took a little bit of experimenting to get right, but I think the below does a good--though very hackish--job of cleaning up the dates."},{"metadata":{"_cell_guid":"1bd7e4e8-76eb-408f-bfaa-49807bda368b","_uuid":"8ff10d23adf3fd07ed5a996fc6b88d70a8d5f6d9","collapsed":true,"trusted":true},"cell_type":"code","source":"print(interview_df['Date'].unique())","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"86e154a2-915f-46d7-a2e4-defe17eb31d8","_uuid":"54cf081719907f9239786ebbccf232a2f8d554d0","collapsed":true,"trusted":true},"cell_type":"code","source":"def clean_date(date):\n    date = date.str.strip()\n    date = date.str.split(\"&\").str[0]\n    date = date.str.replace('–', '/')\n    date = date.str.replace('.', '/')\n    date = date.str.replace('Apr', '04')\n    date = date.str.replace('-', '/')\n    date = date.str.replace(' ', '/')\n    date = date.str.replace('//+', '/')\n    return date","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f81eb258-fe29-4c3e-b6a1-dc383081e8b7","_uuid":"bb3f9ba4b0e82a908c9764025acd82d005bb20af","collapsed":true,"trusted":true},"cell_type":"code","source":"interview_df['Date'] = clean_date(interview_df['Date'])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"476dd472-d86a-4664-b8ea-ef8506b810b0","_uuid":"2a4c4e33519ffda0f7c7bc9022e16600af260bdc"},"cell_type":"markdown","source":"So this little function took care of most of the cleaning, but I still had a stubborn issue where some years where formatted in 2 digit format and some were in 4 digit. As far as I'm aware, pandas can't deal with that, so I had to fix this. In another hackish fix, I decided to split the new Date column into a month, day and year column. From there I could fix the year formats and then I can group them back together in a new Date column. I'm 100% sure this was not the best way to handle this, so I'd love to hear how you would do it. But at the end of the day, it worked and I didn't want to waste more time figuring out the 'correct' solution."},{"metadata":{"_cell_guid":"ac11d15e-90c7-4068-8964-3bf602f1afd8","_uuid":"9b5ac5faefae0af3c1c48e7c392a8a9be9288c6d","collapsed":true,"trusted":true},"cell_type":"code","source":"# Through exploration I discovered row 1233 had a ton of missign values, including the date. Since I can't parse \n# a missing date (and I'm going to create a bunch of new features from that date variable) I'm just removing the row\ninterview_df.drop(interview_df.index[[1233]], inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"9c96f82e-43a3-4464-b865-0706f15a7097","_uuid":"ee446d37d642386515ddf8e2178ffc0b85c9e84f","collapsed":true,"trusted":true},"cell_type":"code","source":"# Create my new date variables\ninterview_df['year'] = interview_df['Date'].str.split(\"/\").str[2]\ninterview_df['day'] = interview_df['Date'].str.split(\"/\").str[0]\ninterview_df['month'] = interview_df['Date'].str.split(\"/\").str[1]\n\n# This will find the short years and replace with long years\ninterview_df['year'].replace(['16', '15'], ['2016', '2015'], inplace = True)\n\n# Finally I create the new Date column\ninterview_df['date'] = pd.to_datetime(pd.DataFrame({'year': interview_df['year'],\n                                            'month': interview_df['month'],\n                                            'day': interview_df['day']}), format = '%Y-%m-%d')\ninterview_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8c786501-ec33-4ceb-b402-07965fa0eab8","_uuid":"a8946e0c4ea034088c69513a3f2d73b6b096e918"},"cell_type":"markdown","source":"That seems to have worked out just fine, so I can drop the old Date column and my temporary year, month, day columns."},{"metadata":{"_cell_guid":"9704013f-1007-4e89-b17c-005502574bde","_uuid":"19d075e93acfa05dcd302118ec5b8925372213d6","collapsed":true,"trusted":true},"cell_type":"code","source":"interview_df.drop(['Date', 'year', 'month', 'day'], axis = 1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"12837424-94fa-4d45-8905-8a44d5661edb","_uuid":"94d59715a90c17bab5b4167d65e4b966ae6765a8"},"cell_type":"markdown","source":"Onward to the next messy variable. Personally I find lower case a bit easy to read, so first I will lowercase all the data. It's probably a good idea to strip whitespace from everything up front too."},{"metadata":{"_cell_guid":"f6398a2f-fe36-45b0-a035-0d97316cba61","_uuid":"e0078dbbb83b67ff85bbf7898ba4d46da36aa000","collapsed":true,"trusted":true},"cell_type":"code","source":"interview_df = pd.concat([interview_df[c].astype(str).str.lower() for c in interview_df.columns], axis = 1)\ninterview_df = pd.concat([interview_df[c].astype(str).str.strip() for c in interview_df.columns], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"cef0001a-0de6-4428-82f7-dc02b58c22e8","_uuid":"4d6b3d03f743da0a9cb3a0bbf86ff762b69f55d9"},"cell_type":"markdown","source":"Looking at the next variable, Client, it's pretty unclear whether entries 'standard chartered bank' and 'standard chartered bank chennai' are the same client. Several entries are like that. Just thinking ahead, I took a look at how often each of these clients occurs in the data. "},{"metadata":{"_cell_guid":"48b0b970-a3ee-4bae-99d2-4502f4f2f78b","_uuid":"4a75e4e8162bf37b64fb5f7b02476dbd66e943bf","collapsed":true,"trusted":true},"cell_type":"code","source":"interview_df['Client'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0dc92f61-81d8-4a20-aa47-c02b9800cc58","_uuid":"0d8fbcea6eaf8fcfc3adb5cfcc5ff030c9a3054c"},"cell_type":"markdown","source":"So this is really unbalanced by Client. I have a strong suspicion the similar clients are just different encodings of the same Client name. It's easy enough to find and replace these alternate values with the 'default' value. It's certainly possible that these values are actually different clients, but for now I'll go with grouping up the ones that seem like the same company."},{"metadata":{"_cell_guid":"a5ef6852-0f65-4fb6-b82d-f1ef4251980c","_uuid":"155a2c280bba1dbe4462c3ec717357fdca4accf8","collapsed":true,"trusted":true},"cell_type":"code","source":"interview_df['Client'].replace(['standard chartered bank chennai', 'aon hewitt gurgaon', 'hewitt'], \n                              ['standard chartered bank', 'aon hewitt', 'aon hewitt'], inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"adb48836-3285-497c-b1b9-74b98bcbed38","_uuid":"baeea9bdf8843c970e9afbb9fc0ac3d8dd3705dc"},"cell_type":"markdown","source":"There's quite a few variables with similar issues, so I'll just do all their cleaning here."},{"metadata":{"_cell_guid":"b6020a95-1a46-4b13-a926-7b2d955386dc","_uuid":"a0369752a3ac5c2fc96ddad21ae32e27676ca803","collapsed":true,"trusted":true},"cell_type":"code","source":"interview_df['Industry'].replace(['it products and services', 'it services'], \n                              ['it', 'it'], inplace = True)\n\ninterview_df['Location'].replace(['- cochin-'], \n                              ['cochin'], inplace = True)\n\n# I'm really not sure about this Interview_Type variable. I'd ask if given the chance. For now I'm just going\n# to use 'scheduled walkin', 'walkin' and 'scheduled'\ninterview_df['Interview_Type'].replace(['scheduled walk in', 'sceduled walkin'],\n                                       ['scheduled walkin', 'scheduled walkin'], inplace = True)\n# I wonder why  cochin is always messed up?\ninterview_df['Cand_Loc'].replace(['- cochin-'], \n                              ['cochin'], inplace = True)\ninterview_df['Job_Loc'].replace(['- cochin-'], \n                              ['cochin'], inplace = True)\ninterview_df['Venue'].replace(['- cochin-'], \n                              ['cochin'], inplace = True)\n# I'm assuming all these native locations are actual places. I didn't check them all\ninterview_df['Native_Loc'].replace(['- cochin-'], \n                              ['cochin'], inplace = True)\n# I don't know if nan's and 'not yet' are actually different, but I'm treating them like they are\ninterview_df['Permission'].replace(['na', 'not yet', 'yet to confirm'], \n                              ['nan', 'tbd', 'tbd'], inplace = True)\ninterview_df['Hope'].replace(['na', 'not sure', 'cant say'], \n                              ['nan', 'unsure', 'unsure'], inplace = True)\ninterview_df['Three_hour_call'].replace(['na', 'no dont'], \n                              ['nan', 'no'], inplace = True)\ninterview_df['Hope'].replace(['na', 'not sure', 'cant say'], \n                              ['nan', 'unsure', 'unsure'], inplace = True)\ninterview_df['Alt_phone'].replace(['na', 'no i have only thi number'], \n                              ['nan', 'no'], inplace = True)\ninterview_df['Resume_Printout'].replace(['na', 'not yet', 'no- will take it soon'], \n                              ['nan', 'ny', 'ny'], inplace = True)\ninterview_df['Clarify_Venue'].replace(['na', 'no- i need to check'], \n                              ['nan', 'no'], inplace = True)\ninterview_df['Shared_Letter'].replace(['na', 'not sure', 'need to check', 'not yet', 'yet to check',\n                                       'havent checked'],\n                                      ['nan', 'unsure', 'unsure', 'unsure', 'unsure', 'unsure'], inplace = True)\ninterview_df['Expected'].replace(['na', '11:00 am', '10.30 am'], \n                              ['nan', 'nan', 'nan'], inplace = True)\ninterview_df['Attended'].replace(['yes', 'no'],\n                                 [1, 0], inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8ec2a034-a334-4a2d-a0ee-6c96c063e9d8","_uuid":"23e1a1be2dd36892ca1b300833eef396de23c3ee"},"cell_type":"markdown","source":"Now the last column, which I think is probably the most difficult- Skillset. There's all sorts of variation in terms of what's even in the column, and after we figure out how to standardize that, we need to spread the column out into multiple skill columns. On top of that, it's not clear from the original data set whether this is the applicants skillset or the skillset that the employer is looking for. I have my doubts as to whether skillset is \ngoing to be that useful in the model. Since there are ~100 unique values of Skillset and cleaning will involve working through all of them, I'm going to see how well I can do without Skillset first."},{"metadata":{"_cell_guid":"13735437-5f5f-4f7a-aa96-6f4cfb6b217f","_uuid":"8e4ed3be0b8f44a975ceeaff5a334518ee4dbb4d","collapsed":true,"trusted":true},"cell_type":"code","source":"modeling_df = interview_df.drop(['Skillset'], axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2ccf2d68-46e4-44ad-8ae5-5dcd94f1012a","_uuid":"d6d893dab2642ec9686e8119bd906e812ec2cf3c"},"cell_type":"markdown","source":"Looking through this data set, I noticed that there were a handful of dates that were from the future (think >2019). I'm assuming these are miscoded and I'm going to remove them."},{"metadata":{"_cell_guid":"1c2f7696-ce6b-4227-a655-978e780ca77d","_uuid":"fcdd0178bc9b9af46ca0aef635ab651408b04a00","collapsed":true,"trusted":true},"cell_type":"code","source":"modeling_df = modeling_df[modeling_df['date'] < '2018-01-01']","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6d2c2032-eb2f-4698-8a2d-bd8967eb3b54","_uuid":"e8691ff020ac4b81a3cd03bf66f3711f2e5ebd7e","collapsed":true},"cell_type":"markdown","source":"Just a couple more tweaks before we can actually start to build some models. First, I want to extract date information from my date variable. Date contains a ton of useful information -- day of week, weekday/weekend, month, year, start of month, etc-- but much of it is hidden. Fastai actually has a method for extracting this information from a datetime variable, so we'll use it here."},{"metadata":{"_cell_guid":"0e30fc6c-f934-487e-8790-b2697243cf38","_uuid":"bff8a5f99b2197a3eaf2e2c7010da002d637324a","collapsed":true,"trusted":true},"cell_type":"code","source":"add_datepart(modeling_df, 'date')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"682ceb25-a60a-4f13-ad27-d83399ddaa11","_uuid":"d1e07bc2f2894ae940c49cfd745e7be996d47cf3","collapsed":true,"trusted":true},"cell_type":"code","source":"modeling_df.columns","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"94fbff66-6b56-4a04-b902-3c1186a97b00","_uuid":"bdd3f4af3dcc3cac95dcafe90890be98d990daf8"},"cell_type":"markdown","source":"So I now have 13 date related variables that I was able to create from the date function. The second thing I need to do is take all the categorical variables and convert them from their string format into something the random forest model can use. This is really easy with the fastai library."},{"metadata":{"_cell_guid":"6c6498ec-58a8-416f-973e-810e655d0a84","_uuid":"a7d69a19b66fcc54937b199ae9525995c598e2e0","collapsed":true,"trusted":true},"cell_type":"code","source":"# Handling categorical variables\ntrain_cats(modeling_df)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c5397188-32df-4087-9f86-2396cf20fdbf","_uuid":"cedf2ce77951d617fd46eceef504e83569f27082"},"cell_type":"markdown","source":"I want to take a quick look at the data now that I've done all my cleaning. I need to change the amount of data frame pandas  will display, but that's a simple fix with the function below."},{"metadata":{"_cell_guid":"d284ddda-dd6b-4429-87e0-8cae76217b51","_uuid":"e0dea0b51b44e72911828c64886e0fd3e55ff002","collapsed":true,"trusted":true},"cell_type":"code","source":"def display_all(df):\n    with pd.option_context(\"display.max_rows\", 1000, \"display.max_columns\", 1000):\n        display(df)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a5d91b97-496f-4a1c-a2da-a5555313626d","_uuid":"671da483e26b3eaac25f4f5bd2356faab4275533","collapsed":true,"trusted":true},"cell_type":"code","source":"display_all(modeling_df.tail().T)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"7610d66c-01ea-4d72-85ce-a10b56a38abd","_uuid":"9390ae76d913dd53b87083ef3b3f597eaf8f1648"},"cell_type":"markdown","source":"Congrats! Our data is basically in a usable form now. Note that in 'Attended' 1 is representing Yes and 0 No. The rest is pretty clear--at least in my opinion (after looking at it while doing all the cleaning)."},{"metadata":{"_cell_guid":"5fd49c48-3a71-4e25-b271-45da7021ec65","_uuid":"75e37a20c4538c6c270c291c15b5be5bbaf76f7f"},"cell_type":"markdown","source":"## Pre-processing "},{"metadata":{"_cell_guid":"9189733d-14af-4859-8926-1fe176c8bf01","_uuid":"96fafc50a9d5d6f5c5ea6d0f7b53ced650d73cee"},"cell_type":"markdown","source":"Since this data set didn't have a test set included, I need to create my own. It would be nice to split into a train, validation and test set, but since the data set is so small to begin with --and the fact that this is just exploration--I'm just going to split into a training and validation set. I'll set aside 20% of the data to be used for validation. I'm also going to split the dependent variable into it's own data frame just for convenience. Note that proc_df is also going to convert all the categorical variables into numeric values to use in the model."},{"metadata":{"_cell_guid":"da12a66c-6f7c-44f0-8c55-ad18e4f11455","_uuid":"6aec9b88f54b1b75ca5949d59a8dd9261b8e8042","collapsed":true,"trusted":true},"cell_type":"code","source":"df, y, nas = proc_df(modeling_df, \"Attended\")","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d2d064f0-cae8-445d-ac60-983f7f1774d8","_uuid":"7e99d6030299a6706e5a0823254fd7231e941a49","collapsed":true,"trusted":true},"cell_type":"code","source":"display_all(df.tail().T)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d2c07ef8-20bb-4ef9-b88c-8389376f8a43","_uuid":"508fca0196bcf234b975fefca2899829f30306f0","collapsed":true,"trusted":true},"cell_type":"code","source":"display_all(y.T)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"5235eea4-ee53-4369-9c47-f5ab7587168c","_uuid":"132e219f996bb1a0ca174926e30b5c1693b0adc9"},"cell_type":"markdown","source":"This will give me about 1000 rows of data to train with and ~250 for validation. With such a small data set, I expect there could be some slight variation between different versions of the same model. If you think about it, a change in the assignment of just 1 or 2 rows will make a relatively big difference when we only have 1000 data points versus when we are building models on data sets with millions of rows.  Once all my modeling is finished, I will rerun the model with all the same parameters using all the data. Obviously I won't be able to test it on anything, but using the out-of-bag score should give me some idea of how the model does with a little bit more data.\n\nRegardless, even with a small data set like this, it's critical to have a validation setup. As we'll see in just a minute, it's incredibly easy to overfit and think your model is performing great only to be bitten when you put the model into production and it doesn't generalize to new data at all."},{"metadata":{"_cell_guid":"1f6f29b0-7ee2-4bcf-a9ad-1873663277d4","_uuid":"913fd4d94096080fcf7df3ebd90570d1fdafa70c","collapsed":true,"trusted":true},"cell_type":"code","source":"def split_vals(a, n):\n    return a[:n].copy(), a[n:].copy()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ccc99867-469b-425d-85db-c58d70888585","_uuid":"8accdb8d65ce578726ad1088de2ee51fb1f3e347","collapsed":true,"trusted":true},"cell_type":"code","source":"n_valid = int(len(modeling_df) * .2)\nn_trn = len(modeling_df) - n_valid\nraw_train, raw_valid = split_vals(modeling_df, n_trn)\nX_train, X_valid = split_vals(df, n_trn)\ny_train, y_valid = split_vals(y, n_trn)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"270c0b7e-ebb5-4797-917b-90117aa7d829","_uuid":"96e293134934e5ea9a6ca38eb6e3fecb8ede190b"},"cell_type":"markdown","source":"## Modeling"},{"metadata":{"_cell_guid":"83c6297f-7696-43e2-af92-ccf0105bc23f","_uuid":"abe51eb18b23581b522250aa6a439bc13e2240d9"},"cell_type":"markdown","source":"### Base Model"},{"metadata":{"_cell_guid":"7bfa6a5a-0473-477a-81c8-1db0cd7f8fc1","_uuid":"5ae13da6d6067a5378062fb56b21ff54fb28acfb"},"cell_type":"markdown","source":"Now that I have clean data and a validation setup, I can actually start to explore models. First, I want to build a random forest with all default parameters."},{"metadata":{"_cell_guid":"ae7c2387-e966-40ae-9b48-18fa35a10938","_uuid":"3be70d72c3dc0d6deedcbb689e30bf3779f075cb","collapsed":true,"trusted":true},"cell_type":"code","source":"m_base = RandomForestClassifier(n_jobs = -1, oob_score=True)\n%time m_base.fit(X_train, y_train)\nprint(\"Training Acc:\", round(m_base.score(X_train, y_train),5)),\nprint(\"Validation Acc:\", round(m_base.score(X_valid, y_valid), 5)),\nprint(\"Out-of-Bag Acc:\", round(m_base.oob_score_, 5))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"d549277b-63c9-4d2a-805e-7143f72aa631","_uuid":"e34884c552348e7a177ecc9181460a232462f1c3"},"cell_type":"markdown","source":"This first model is just using all the defaults in sklearn's random forest classifier method. That means it will grow the tree till each node is as pure as possible. If we take a look at the resulting tree, we can see it's huge. "},{"metadata":{"_cell_guid":"90a84a8f-0f0b-46df-b341-901854d63532","_uuid":"2e1398e343c2b34767555178ae20e49eefc7db0c","collapsed":true,"trusted":true},"cell_type":"code","source":"probs = m_base.predict_proba(X_valid)\nprobs = [p[1] for p in probs]\nfpr, tpr, thresholds = metrics.roc_curve(y_valid, probs)\nroc_auc = metrics.roc_auc_score(y_valid, probs)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"6fb848a9-d804-4e35-b562-50f631a524bd","_uuid":"486ff57e3e2e183cf4196b212c0d86d983075dd8","collapsed":true,"trusted":true},"cell_type":"code","source":"plt.plot(fpr, tpr, color = 'darkorange',\n        label='ROC curve (area = %0.2f)' % roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', linestyle='--')\nplt.legend(loc=\"lower right\")\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"63f51134-2fea-4d58-8e2d-f0492d3fa678","_uuid":"255a7f3bea08eb425a925457e7c522b3010bb259","collapsed":true,"trusted":true},"cell_type":"code","source":"draw_tree(m_base.estimators_[0], df, precision = 3)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"4c4048bf-3c94-478b-9f6a-5b8412b12d25","_uuid":"99337f36dc8d41f3f333b488d92944101bac2381"},"cell_type":"markdown","source":"There is simply no way this tree didn't overfit, and when I look at the validation set accuracy I see that's the case. The model was 97% accurate on the training set, but only 64% on the validation set. That's a sure sign of overfitting. The first thing I should test is limiting the depth of the tree. "},{"metadata":{"_cell_guid":"e9cfb70b-ccee-423f-b861-449dc30f3204","_uuid":"100a2600f71acad19782a75edb74106e0406881b","collapsed":true,"trusted":true},"cell_type":"code","source":"m = RandomForestClassifier(min_samples_leaf=5, n_jobs=-1, oob_score=True)\nm.fit(X_train, y_train)\nprint(\"Training Acc:\", round(m.score(X_train, y_train),5)),\nprint(\"Validation Acc:\", round(m.score(X_valid, y_valid), 5))\nprint(\"Out-of-Bag Acc:\", round(m.oob_score_, 5))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"c6bad696-5f56-472f-b82b-7849729d240b","_uuid":"920d37d46076d1490553a9b63f02d04953e2a540","collapsed":true,"trusted":true},"cell_type":"code","source":"draw_tree(m.estimators_[0], df, precision = 3)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3c641832-476f-4d6b-b147-e8093bca815e","_uuid":"46a9fb8c3cad68260cc185e61866c12c6f26cbad"},"cell_type":"markdown","source":"So while my training set accuracy went down dramatically, this model actually generalized better to the validation set. Setting min_samples_leaf tells the model that each node has to have at least 5 rows in it. This results in a simpler model, but one that should generalize better to new data.\n\nAnother lever to pull when tuning is max_features. This specifies what proportion of features(aka columns) to randomly select for each tree in the random forest. This has the effect of increasing the amount of variation per tree--after all, you trees are now being built with different features-- which will result in better generalization. The sklearn documentation has a couple suggestions for starting values and after tinkering with a bunch, I found 'log2' gave me the best performance and the most stable results. "},{"metadata":{"_cell_guid":"0d8e5de7-6b37-4f1a-a754-83e88e0cfba0","_uuid":"d15f5ca90323e58eac69a0b02325fef6e83cf458","collapsed":true,"trusted":true},"cell_type":"code","source":"m = RandomForestClassifier(min_samples_leaf=5, max_features = 'log2',n_jobs=-1, oob_score=True)\nm.fit(X_train, y_train)\nprint(\"Training Acc:\", round(m.score(X_train, y_train),5)),\nprint(\"Validation Acc:\", round(m.score(X_valid, y_valid), 5))\nprint(\"Out-of-Bag Acc:\", round(m.oob_score_, 5))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"0b6665fd-7bb7-4107-83ae-a09174690c91","_uuid":"36cce48722c7282733f05ed8e2b49b42c7f8f3d1"},"cell_type":"markdown","source":"I could also try subsampling to reduce over-fitting, but here the results were marginally worse, so I will stick with my adjusted tree parameters. Subsampling **is** really useful when we are working with a larger data set, however. You get the benefit of speeding up your computations *and* you reduce over-fitting at the same time by using a different random sample of data per tree.\n\nSo far, I've just used the tree building parameters available in the RandomForestClassifier method from sklearn to tune my tree and reduce over-fitting. I've looked at the number of trees in the forest, the depth of those trees, the size of the nodes of those trees and the number of features to random choose while building each tree.  Of those parameters, only the size of the nodes and the number of features in each tree had much effect in reducing over-fitting. With such a small data set, additional trees didn't seem to help much after about 8-10 trees--though that parameter is especially important to keep in your toolkit for larger data sets."},{"metadata":{"_cell_guid":"29c25f7a-cbdf-46bf-9375-d11da704048b","_uuid":"a665a412339e44bd95ade3afb75d0a386b314cab"},"cell_type":"markdown","source":"### Feature Importance"},{"metadata":{"_cell_guid":"a874154e-0f98-4aef-a065-b6d335b4f2d2","_uuid":"6ae6077ea2552b8d4481fc6bbcdd073273b46d93"},"cell_type":"markdown","source":"Tree depth and node size are important parameters, but they aren't the whole story. I can also adjust my model by looking at feature importance and removing unnecessary or redundant features. Perhaps more importantly, I can use these to gain an understanding of my problem. In this case, maybe there is a feature that is a clear signal someone will miss an interview. Identifying this feature could be more important that your full model in some cases.\n\nTo examine the importance of my features, I can use the rf_feat_importance function from the fastai library. Determining a features importance is actually pretty straightforward. Say I wanted to determine how important 'year' was to my model. I can take the values of the year column and randomly reshuffle them. In doing this, I create a variable that has no relationship to the dependent variable.  Now, if I run my data through the model again, I can see how much the performance decreases. This performance difference is the feature importance of the year variable. For a full list of feature importances, I just run through each feature in the model in the same manner."},{"metadata":{"_cell_guid":"194eece5-0c00-4a6e-8333-68eada1ac866","_uuid":"4c73f88653b0a19e1a65c7b13a7f944b2740db12","collapsed":true,"trusted":true},"cell_type":"code","source":"feature_imp = rf_feat_importance(m, df)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"af7b8481-f7ac-43f6-9f22-f59e58d84d43","_uuid":"020fd8622c5a2321b1f3475a1e4557a9d7dbd749","collapsed":true,"trusted":true},"cell_type":"code","source":"feature_imp.plot('cols', 'imp', figsize = (10, 6), legend = False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2894c65a-4f96-45a8-9d7a-02ec642820e6","_uuid":"aefe6583753d510d6fbfc00caff9561cea01e0a1"},"cell_type":"markdown","source":"It's clear from the plot above that the first two features explain a ton of the variance in the model. I can use a different view to see exactly what these features are."},{"metadata":{"_cell_guid":"29de200d-6b08-4373-a12e-9cba83b20dcd","_uuid":"0076484e2f6a81c94f1082ee1359d017916aa3f0","collapsed":true,"trusted":true},"cell_type":"code","source":"def plot_fi(fi):\n    return fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend = False)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"18d1979a-77b7-4d1e-8ce1-a9d844522d14","_uuid":"78b2a018021e3d3746bde2311772066937fa014e","collapsed":true,"scrolled":true,"trusted":true},"cell_type":"code","source":"plot_fi(feature_imp)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"225c8cff-e937-4daf-a750-8520d0bb2cd9","_uuid":"209d736b5373e5786397f55c180f538c2947c528"},"cell_type":"markdown","source":"So 'Expected' and 'ID' are the most important features by far. This is pretty worrying. The ID variable appears to just be a numeric ID of the individuals interviewing for jobs. Looking back at the original data set, the ID's are just assigned 1-1232 with no regard for date, so it's probably something that was assigned after the fact to anonymize the data. But, if this was assigned after the fact and is this important to the model, it makes me think we have some information leakage. Like perhaps the data was ordered in some way before this ID was assigned and that ordering had some relation to who actually attended interviews.  I'm not sure what exactly happened, but it's definitely concerning and is something I'd ask about if I were actually working with this data.\n\nExpected could fall into the same trap. When was the expected variable assigned? Was it something after the fact? It could be that the screeners for these applicants did a good job in predicting who would attended, so the correlation between the expected and actual attendance was high. I can't really be sure for ID or Expected what is happening, but in this instance, I'm just going to leave them as is.\n\nLooking at the other end of the importance plot, you can see that all the date variables related to starting or ending periods are essentially useless. I can remove all the variables below a certain threshold to tidy up our model."},{"metadata":{"_cell_guid":"df560e1b-04a8-4113-af81-463c33c48e38","_uuid":"b7158f49c846248e15402f577712690c21dbfd96","collapsed":true,"trusted":true},"cell_type":"code","source":"keep = feature_imp[feature_imp.imp > 0.0075].cols","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"de810cc0-417e-43ad-b44d-7148d505a0b2","_uuid":"1e0c7b50693108b5cee30f0a0e091967d3f65f8d","collapsed":true,"trusted":true},"cell_type":"code","source":"len(keep)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"a20f922c-f46e-42cc-bba0-82d07eb23fb6","_uuid":"3899e7c5ade7d7580c51def21929487f45b65e96","collapsed":true,"trusted":true},"cell_type":"code","source":"df_keep = df[keep].copy()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"04463914-4ed0-4770-8303-c06f238e8a1f","_uuid":"6412a83e632b1809172d4299346928ec38e16dda"},"cell_type":"markdown","source":"When I retrain the model with these features removed, I actually see a tiny bit of improvement. Simplifying your model not only makes it more interpretable, but it can improve it's generalization."},{"metadata":{"_cell_guid":"9d4fee12-6e12-4899-8dfa-d6746f10be87","_uuid":"d0822f5efc9d1b6640f8641e83800577a2cc67f4","collapsed":true,"trusted":true},"cell_type":"code","source":"X_train, X_valid = split_vals(df_keep, n_trn)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ef618cf2-526f-49a5-a2f8-5c301b9766ef","_uuid":"d021c738b6421593fad22cbdd0305d464b6d653e","collapsed":true,"trusted":true},"cell_type":"code","source":"m = RandomForestClassifier(min_samples_leaf = 5,\n                           max_features='log2', n_jobs=-1, oob_score=True)\nm.fit(X_train, y_train)\nprint(\"Training Acc:\", round(m.score(X_train, y_train),5)),\nprint(\"Validation Acc:\", round(m.score(X_valid, y_valid), 5))\nprint(\"Out-of-Bag Acc:\", round(m.oob_score_, 5))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"35d94a95-3c03-4533-8f37-e7cd9bb7dfbc","_uuid":"adaa70b51499ded945f8dca0c5ec3b7ee37e0a11","collapsed":true,"trusted":true},"cell_type":"code","source":"fi = rf_feat_importance(m, df_keep)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"ce6ce78b-d86d-46e6-91e3-75b1b3d70c8c","_uuid":"1e03e5e2766cb6ebd0deea439962be9d36653983","collapsed":true,"trusted":true},"cell_type":"code","source":"plot_fi(fi)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"00d0980a-aa04-42dc-8056-181c08bf4783","_uuid":"36831b03c43212b91247ab468f0da6105efc6544"},"cell_type":"markdown","source":"If you look at these features, they make a lot of sense. Whether the screener expected the applicant is quite predictive, as are whether the applicant shared their cover letter, an alternative phone number and their resume. These are all things that I would consider a sign of interest in a position and so doing them probably means you are interested in the job and more likely to show up for an interview."},{"metadata":{"_cell_guid":"6e6da493-cd7f-4f91-804e-2cdc01f32e4f","_uuid":"3819bd27fec1475d70e8c116fffd4aad9e594212","heading_collapsed":true},"cell_type":"markdown","source":"### One-Hot Encoding"},{"metadata":{"_cell_guid":"7fc44d01-4b9c-404d-bd1c-3c119f9d47c7","_uuid":"bf7b9f1f7993a04c19fb09bf4067928e33f12e71","collapsed":true,"hidden":true,"trusted":true},"cell_type":"code","source":"df_trn, y_trn, nas = proc_df(modeling_df, 'Attended', max_n_cat = 7)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"69e7e6d1-2bd1-4317-a4a9-e2b7cdd5532b","_uuid":"380b17d9b2e799a37345d75d07b32f4a367a4f09","collapsed":true,"hidden":true,"trusted":true},"cell_type":"code","source":"X_train, X_valid = split_vals(df_trn, n_trn)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3b8e6d1d-b6f6-4ab3-b8b0-24cc496115cc","_uuid":"18e2e4fa38154cbf36d54231ea6054b078881127","collapsed":true,"hidden":true,"trusted":true},"cell_type":"code","source":"m = RandomForestClassifier(min_samples_leaf = 5,\n                           max_features='log2', n_jobs=-1, oob_score=True)\nm.fit(X_train, y_train)\nprint(\"Training Acc:\", round(m.score(X_train, y_train),5)),\nprint(\"Validation Acc:\", round(m.score(X_valid, y_valid), 5))\nprint(\"Out-of-Bag Acc:\", round(m.oob_score_, 5))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2cbfbf6d-e0a5-49fc-83cd-d7b7c4ab6475","_uuid":"69cc93ea6aea167b7a0166cd6b99014d20df8447","collapsed":true,"hidden":true,"trusted":true},"cell_type":"code","source":"fi = rf_feat_importance(m, df_trn)\nplot_fi(fi[:25])","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"f286859f-1b59-43d3-95ff-7c93a3295eca","_uuid":"a44e5c73c7435aab8f88599c17956a104188deaf"},"cell_type":"markdown","source":"### Removing Redundant features"},{"metadata":{"_cell_guid":"3edef4d3-773f-4da7-ad2f-3e0f307bd991","_uuid":"75cd44706e1b874146094e93c6d0a306d82f7263"},"cell_type":"markdown","source":"Another useful technique for examining your model is dendrogram plots. These plots show the relation between features. The closer the split is to the right side of the plot, the more correlated the features are. In this case, features like Dayofyear and Week are highly correlated while features such as Client and Day are very uncorrelated."},{"metadata":{"_cell_guid":"d23003e3-c642-4ee6-9788-1574f54efb90","_uuid":"a7509653d6a8135abea0a867a213e404eedf545c","collapsed":true,"trusted":true},"cell_type":"code","source":"from scipy.cluster import hierarchy as hc","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"27226473-af6d-4fe5-a74a-f196373de6b0","_uuid":"5402bfd341a60f4af1e13c41b9b6c7c373d3f3e5","collapsed":true,"trusted":true},"cell_type":"code","source":"corr = np.round(scipy.stats.spearmanr(df_keep).correlation,4)\ncorr_condensed = hc.distance.squareform(1-corr)\nz = hc.linkage(corr_condensed, method = 'average')\nfig = plt.figure(figsize=(16,10))\ndendrogram = hc.dendrogram(z, labels = df_keep.columns, orientation='left',\n                          leaf_font_size =16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"943449f9-df11-4d02-9003-7786aa8cde6b","_uuid":"c3f139cf198401b120c3929bec9dd9be712a731a"},"cell_type":"markdown","source":"You can use this information to improve your model. Since I see that Dayofyear and Week are highly correlated, why not look and see how my model performs if I remove one of the variables. That's precisely what I do below."},{"metadata":{"_cell_guid":"14eb4217-b5cf-4243-9809-3270ecce9639","_uuid":"b484f01e03cebb1035f68be3344e596bee361f33","collapsed":true,"trusted":true},"cell_type":"code","source":"def get_oob(df):\n    m = RandomForestClassifier(n_estimators=40, min_samples_leaf=8, max_features='sqrt', n_jobs=-1, oob_score=True)\n    x, _ = split_vals(df, n_trn)\n    m.fit(x, y_train)\n    return m.oob_score_","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"8d452273-a862-46fd-bb8c-e84653a08832","_uuid":"7bfd914684ec1118f3b0468a5f4708ba30bbdaa6","collapsed":true,"trusted":true},"cell_type":"code","source":"get_oob(df_keep)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"13afb0a4-0755-414e-a710-4859c17ab309","_uuid":"c5d2fdb9c7e39abe1523c60e228013901fe6f72d","collapsed":true,"trusted":true},"cell_type":"code","source":"for c in ('Week', 'Dayofyear'):\n    print(c, get_oob(df_keep.drop(c, axis = 1)))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"48be607f-fffe-4c4c-9b2b-a5cfd6f4cf59","_uuid":"69314af45ec2a4462ec3fdff5f1a74567f9c0467"},"cell_type":"markdown","source":"While removing Dayofyear doesn't improve the model much, it does simplify it. Going along with what I've discussed so far, this will generally lead to better generalization of the model. When I remove the Dayofyear feature and rebuild my model, the model accuracy also improves slightly. "},{"metadata":{"_cell_guid":"3b54c281-0939-467a-9241-9480d3daa91b","_uuid":"87756255d85a845907d0017be729ca1e231f6a6e","collapsed":true,"trusted":true},"cell_type":"code","source":"modeling_df.drop(['Dayofyear'], axis = 1, inplace = True)\ndf_trn, y_trn, nas = proc_df(modeling_df, 'Attended', max_n_cat = 7)\nX_train, X_valid = split_vals(df_trn, n_trn)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"2f620c56-3a7c-464c-8d96-327213b9882a","_uuid":"c2fad5b71003fbf21c482e763ca6371d4482c995","collapsed":true,"trusted":true},"cell_type":"code","source":"m = RandomForestClassifier(min_samples_leaf = 5,\n                           max_features='log2', n_jobs=-1, oob_score=True)\nm.fit(X_train, y_train)\nprint(\"Training Acc:\", round(m.score(X_train, y_train),5)),\nprint(\"Validation Acc:\", round(m.score(X_valid, y_valid), 5))\nprint(\"Out-of-Bag Acc:\", round(m.oob_score_, 5))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1a62237-9483-40f1-bbcf-67fcbdecd4e1","_uuid":"a6438ed885823592c1841e816214cc7e722c00fd","collapsed":true,"trusted":true},"cell_type":"code","source":"probs = m.predict_proba(X_valid)\nprobs = [p[1] for p in probs]\nfpr, tpr, thresholds = metrics.roc_curve(y_valid, probs)\nroc_auc = metrics.roc_auc_score(y_valid, probs)","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"dd773b4a-ab65-4ea7-ae38-1bf618f0e937","_uuid":"62b92f08644e89fae93fd7a1ebe0decfda648c86","collapsed":true,"trusted":true},"cell_type":"code","source":"plt.plot(fpr, tpr, color = 'darkorange',\n        label='ROC curve (area = %0.2f)' % roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', linestyle='--')\nplt.legend(loc=\"lower right\")\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"819f1d97-542c-4ea8-8c71-d573bc0ae4f3","_uuid":"a3b8903668ed98f22c3eb0fe310ba570d36969d0"},"cell_type":"markdown","source":"So after a lot of tinkering, I have a model that's about 70% accurate. That's about 6% more accurate than the initial model, so that's nothing to sneeze at. I explored a number of other avenues (such as upsampling the data to improve predictive power--this did not work well) that didn't end up making it into this final post. Sometimes the technique just flat didn't work and other times it simply didn't make much of a difference.  That is the nature of data science, however. There are a lot of techniques out there, and you have to know when to use each one. More importantly, you have to be willing to scrape something you put a lot of work into because it's just not useful. It's a frustrating, but important lesson to learn."},{"metadata":{"_cell_guid":"a64f841b-b07d-46b0-879d-465f39c5aedc","_uuid":"e3cc8ed0c6e856d452acffb8312c0f9d7c5f4556"},"cell_type":"markdown","source":"A couple final points of note. You may have noticed that while my accuracy improved, the AUROC(Area Under Receiver Operating Characteristic) did not. This is a more robust scoring metric that uses False Positive Rate and True Positive Rate to produce a more balanced view of your model performance. While accuracy flat out fails on data sets that are severely imbalanced (for instance, in medicine when dealing with rare diseases), AUC takes into consideration how many positive data points(in our case how a 'yes') were correctly identified as positive (also called TPR or Recall) and also how many negative data points('no's in our case) were incorrectly identified as positive ones. It's a little disappointing that AUROC didn't really improve, but I have a theory as to why that might be.\n\nWhen training my models, there was quite a bit of variability when training the same model multiple times. This is really bad. It probably has to do with the amount of data I have, but basically it means I can't trust how good my model is. I could retrain it and get a result that's significantly different. I think given more data this problem would go away, but it's also possible that I'd need to go back to the drawing board and reassess how I'm modeling this data.\n\nJust as quick, dirty test, I built the same model but using the full data set. I only have the oob score to evaluate with now, but the results seem pretty similar to the training/validation split. I can't be too unhappy with this quick analysis overall though. ~70% accuracy is decent for a couple hours of work. I'd love to hear how others are approaching this or similar problems. \n\nThanks for reading!"},{"metadata":{"_cell_guid":"8d1188c0-f099-44ee-9fca-d659cbc8e1ea","_uuid":"ad7ab1f11a24f8aea731a8a684077f4f9b7fe3ca","collapsed":true,"trusted":true},"cell_type":"code","source":"df, y, nas = proc_df(modeling_df, \"Attended\")\nm = RandomForestClassifier(min_samples_leaf = 5,\n                           max_features='log2', n_jobs=-1, oob_score=True)\nm.fit(df, y)\nprint(\"Training Acc:\", round(m.score(df, y),5)),\nprint(\"Out-of-Bag Acc:\", round(m.oob_score_, 5))","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"3f637a3a-0a2d-4daa-bee7-f18501c5f75d","_uuid":"403847d68e5989b40dac5d7ae377aa0444a4c78f"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}