3_exercise-shap-values.ipynb

{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Machine Learning Explainability](https://www.kaggle.com/learn/machine-learning-explainability) course.  You can reference the tutorial at [this link](https://www.kaggle.com/dansbecker/shap-values).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"## Set Up\n\nAt this point, you have enough tools to put together compelling solutions to real-world problems. You will ned to pick the right techniques for each part of the following data science scenario. Along the way, you'll use SHAP values along with your other insights tools.\n\n**The questions below give you feedback on your work by using some checking code. Run the following cell to set up our feedback system.**","metadata":{}},{"cell_type":"code","source":"from learntools.ml_explainability.ex4 import *\nprint(\"Setup Complete\")","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:01:09.088819Z","iopub.execute_input":"2021-12-13T12:01:09.089138Z","iopub.status.idle":"2021-12-13T12:01:09.147174Z","shell.execute_reply.started":"2021-12-13T12:01:09.089057Z","shell.execute_reply":"2021-12-13T12:01:09.146368Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"## The Scenario\nA hospital has struggled with \"readmissions,\" where they release a patient before the patient has recovered enough, and the patient returns with health complications. \n\nThe hospital wants your help identifying patients at highest risk of being readmitted. Doctors (rather than your model) will make the final decision about when to release each patient; but they hope your model will highlight issues the doctors should consider when releasing a patient.\n\nThe hospital has given you relevant patient medical information.  Here is a list of columns in the data:\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndata = pd.read_csv('../input/hospital-readmissions/train.csv')\ndata.columns","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:01:15.935218Z","iopub.execute_input":"2021-12-13T12:01:15.935518Z","iopub.status.idle":"2021-12-13T12:01:16.270334Z","shell.execute_reply.started":"2021-12-13T12:01:15.935487Z","shell.execute_reply":"2021-12-13T12:01:16.269325Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"markdown","source":"## Your Code Library\nAs you write code to work through this scenario, these code snippets from previous tutorials may be useful. You'll still need to modify them, but we've copied them here to save you from having to look them up.\n\n**Calculate and show permutation importance:**\n```\nimport eli5\nfrom eli5.sklearn import PermutationImportance\n\nperm = PermutationImportance(my_model, random_state=1).fit(val_X, val_y)\neli5.show_weights(perm, feature_names = val_X.columns.tolist())\n```\n\n**Calculate and show partial dependence plot:**\n```\nfrom matplotlib import pyplot as plt\nfrom pdpbox import pdp, get_dataset, info_plots\n\n# Create the data that we will plot\npdp_goals = pdp.pdp_isolate(model=my_model, dataset=val_X, model_features=feature_names, feature='Goal Scored')\n\n# plot it\npdp.pdp_plot(pdp_goals, 'Goal Scored')\nplt.show()\n```\n\n**Calculate and show Shap Values for One Prediction:**\n```\nimport shap  # package used to calculate Shap values\n\ndata_for_prediction = val_X.iloc[0,:]  # use 1 row of data here. Could use multiple rows if desired\n\n# Create object that can calculate shap values\nexplainer = shap.TreeExplainer(my_model)\nshap_values = explainer.shap_values(data_for_prediction)\nshap.initjs()\nshap.force_plot(explainer.expected_value[0], shap_values[0], data_for_prediction)\n```\n\n## Step 1\nYou have built a simple model, but the doctors say they don't know how to evaluate a model, and they'd like you to show them some evidence the model is doing something in line with their medical intuition. Create any graphics or tables that will show them a quick overview of what the model is doing?\n\nThey are very busy. So they want you to condense your model overview into just 1 or 2 graphics, rather than a long string of graphics.\n\nWe'll start after the point where you've built a basic model. Just run the following cell to build the model called `my_model`.","metadata":{}},{"cell_type":"markdown","source":"Here are some quick hints at interpreting the field names:\n\n- Your prediction target is `readmitted`\n- Columns with the word `diag` indicate the diagnostic code of the illness or illnesses the patient was admitted with. For example, `diag_1_428` means the doctor said their first illness diagnosis is number \"428\".  What illness does 428 correspond to? You could look it up in a codebook, but without more medical background it wouldn't mean anything to you anyway.\n- A column names like `glimepiride_No` mean the patient did not have the medicine `glimepiride`. If this feature had a value of False, then the patient did take the drug `glimepiride`\n- Features whose names begin with `medical_specialty` describe the specialty of the doctor seeing the patient. The values in these fields are all `True` or `False`.\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\n\ndata = pd.read_csv('../input/hospital-readmissions/train.csv')\n\ny = data.readmitted\n\nbase_features = [c for c in data.columns if c != \"readmitted\"]\n\nX = data[base_features]\n\ntrain_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)\nmy_model = RandomForestClassifier(n_estimators=30, random_state=1).fit(train_X, train_y)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:01:39.665078Z","iopub.execute_input":"2021-12-13T12:01:39.665374Z","iopub.status.idle":"2021-12-13T12:01:42.175901Z","shell.execute_reply.started":"2021-12-13T12:01:39.665345Z","shell.execute_reply":"2021-12-13T12:01:42.174862Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"Now use the following cell to create the materials for the doctors.","metadata":{}},{"cell_type":"code","source":"# Your code here\nimport eli5\nfrom eli5.sklearn import PermutationImportance\n\nperm = PermutationImportance(my_model, random_state=1).fit(val_X, val_y)\neli5.show_weights(perm, feature_names = val_X.columns.tolist())","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:01:45.653240Z","iopub.execute_input":"2021-12-13T12:01:45.653560Z","iopub.status.idle":"2021-12-13T12:02:11.345012Z","shell.execute_reply.started":"2021-12-13T12:01:45.653525Z","shell.execute_reply":"2021-12-13T12:02:11.344006Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"markdown","source":"For our idea of what to show, run the cell below.  ","metadata":{}},{"cell_type":"code","source":"# Run this code cell to receive credit!\nq_1.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you'd like to discuss your approach or see what others have done, we have a discussion forum [here](https://www.kaggle.com/learn-forum/66267#latest-390149).\n\n","metadata":{}},{"cell_type":"markdown","source":"## Step 2\n\nIt appears `number_inpatient` is a really important feature. The doctors would like to know more about that. Create a graph for them that shows how `num_inpatient` affects the model's predictions.","metadata":{}},{"cell_type":"code","source":"# Your Code Here\nfrom matplotlib import pyplot as plt\nfrom pdpbox import pdp, get_dataset, info_plots\n\nfeature_name = 'number_inpatient'\n# Create the data that we will plot\nmy_pdp = pdp.pdp_isolate(model=my_model, dataset=val_X, model_features=val_X.columns, feature=feature_name)\n\n# plot it\npdp.pdp_plot(my_pdp, feature_name)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:02:20.313910Z","iopub.execute_input":"2021-12-13T12:02:20.314172Z","iopub.status.idle":"2021-12-13T12:02:21.260385Z","shell.execute_reply.started":"2021-12-13T12:02:20.314144Z","shell.execute_reply":"2021-12-13T12:02:21.259527Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"markdown","source":"For our solution, run the line below.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_2.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 3\n\nThe doctors think it's a good sign that increasing the number of inpatient procedures leads to increased predictions.  But they can't tell from this plot whether that change in the plot is big or small. They'd like you to create something similar for `time_in_hospital` to see how that compares.","metadata":{}},{"cell_type":"code","source":"# Your Code Here\nfrom matplotlib import pyplot as plt\nfrom pdpbox import pdp, get_dataset, info_plots\n\nfeature_name = 'time_in_hospital'\n# Create the data that we will plot\nmy_pdp = pdp.pdp_isolate(model=my_model, dataset=val_X, model_features=val_X.columns, feature=feature_name)\n\n# plot it\npdp.pdp_plot(my_pdp, feature_name)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:02:32.441488Z","iopub.execute_input":"2021-12-13T12:02:32.441770Z","iopub.status.idle":"2021-12-13T12:02:33.619528Z","shell.execute_reply.started":"2021-12-13T12:02:32.441741Z","shell.execute_reply":"2021-12-13T12:02:33.618652Z"},"trusted":true},"execution_count":6,"outputs":[]},{"cell_type":"markdown","source":"Uncomment the relevant line below to see one solution.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_3.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 4\n\nWoah!  It seems like `time_in_hospital` doesn't matter at all.  The difference between the lowest value on the partial dependence plot and the highest value is about 5%.\n\nIf that is what your model concluded, the doctors will believe it. But it seems so low. Could  the data be wrong, or is your model doing something more complex than they expect?  \n\nThey'd like you to show them the raw readmission rate for each value of `time_in_hospital` to see how it compares to the partial dependence plot.\n\n- Make that plot. \n- Are the results similar or different?","metadata":{}},{"cell_type":"code","source":"# Your Code Here\nall_train = pd.concat([train_X, train_y], axis=1)\n\nall_train.groupby(['time_in_hospital']).mean().readmitted.plot()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:02:35.961112Z","iopub.execute_input":"2021-12-13T12:02:35.961683Z","iopub.status.idle":"2021-12-13T12:02:36.206543Z","shell.execute_reply.started":"2021-12-13T12:02:35.961645Z","shell.execute_reply":"2021-12-13T12:02:36.205678Z"},"trusted":true},"execution_count":7,"outputs":[]},{"cell_type":"markdown","source":"For a hint, uncomment the line below.","metadata":{}},{"cell_type":"code","source":"# q_4.hint()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_4.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Step 5\nNow the doctors are convinced you have the right data, and the model overview looked reasonable.  It's time to turn this into a finished product they can use. Specifically, the hospital wants you to create a function `patient_risk_factors` that does the following\n- Takes a single row with patient data (of the same format you as your raw data)\n- Creates a visualization showing what features of that patient increased their risk of readmission, what features decreased it, and how much those features mattered.\n\nIt's not important to show every feature with every miniscule impact on the readmission risk.  It's fine to focus on only the most important features for that patient.","metadata":{}},{"cell_type":"code","source":"# Your Code Here\nimport shap  \n\nsample_data_for_prediction = val_X.iloc[0].astype(float)  # to test function\n\ndef patient_risk_factors(model, patient_data):\n    # Create object that can calculate shap values\n    explainer = shap.TreeExplainer(model)\n    shap_values = explainer.shap_values(patient_data)\n    shap.initjs()\n    return shap.force_plot(explainer.expected_value[1], shap_values[1], patient_data)","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:02:39.806779Z","iopub.execute_input":"2021-12-13T12:02:39.807183Z","iopub.status.idle":"2021-12-13T12:02:41.161693Z","shell.execute_reply.started":"2021-12-13T12:02:39.807153Z","shell.execute_reply":"2021-12-13T12:02:41.160901Z"},"trusted":true},"execution_count":8,"outputs":[]},{"cell_type":"markdown","source":"For a hint, uncomment the line below.","metadata":{}},{"cell_type":"code","source":"# q_5.hint()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\n#q_5.solution()","metadata":{"execution":{"iopub.status.busy":"2021-12-13T12:02:58.641289Z","iopub.execute_input":"2021-12-13T12:02:58.641938Z","iopub.status.idle":"2021-12-13T12:02:58.645612Z","shell.execute_reply.started":"2021-12-13T12:02:58.641898Z","shell.execute_reply":"2021-12-13T12:02:58.644549Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"markdown","source":"## Keep Going\nYou have some powerful tools to get insights about both models and individual predictions. Next you will look at **[aggregations of SHAP values](https://www.kaggle.com/dansbecker/advanced-uses-of-shap-values)** to link the model-level and prediction-level insights.\n","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/machine-learning-explainability/discussion) to chat with other learners.*","metadata":{}}]}