introduction-to-ensembling-stacking-in-python.ipynb

{"cells":[{"metadata":{"_uuid":"e919d1161f20999e599ba1fd66a5a45b9c82f229","_cell_guid":"bc64948f-5d6a-078d-085d-1beb58687bd3"},"cell_type":"markdown","source":"# Introduction\n\nThis notebook is a very basic and simple introductory primer to the method of ensembling (combining) base learning models, in particular the variant of ensembling known as Stacking. In a nutshell stacking uses as a first-level (base), the predictions of a few basic classifiers and then uses another model at the second-level to predict the output from the earlier first-level predictions.\n\nThe Titanic dataset is a prime candidate for introducing this concept as many newcomers to Kaggle start out here. Furthermore even though stacking has been responsible for many a team winning Kaggle competitions there seems to be a dearth of kernels on this topic so I hope this notebook can fill somewhat of that void.\n\nI myself am quite a newcomer to the Kaggle scene as well and the first proper ensembling/stacking script that I managed to chance upon and study was one written in the AllState Severity Claims competition by the great Faron. The material in this notebook borrows heavily from Faron's script although ported to factor in ensembles of classifiers whilst his was ensembles of regressors. Anyway please check out his script here:\n\n[Stacking Starter][1] : by Faron \n\n\nNow onto the notebook at hand and I hope that it manages to do justice and convey the concept of ensembling in an intuitive and concise manner.  My other standalone Kaggle [script][2] which implements exactly the same ensembling steps (albeit with different parameters) discussed below gives a Public LB score of 0.808 which is good enough to get to the top 9% and runs just under 4 minutes. Therefore I am pretty sure there is a lot of room to improve and add on to that script. Anyways please feel free to leave me any comments with regards to how I can improve\n\n\n  [1]: https://www.kaggle.com/mmueller/allstate-claims-severity/stacking-starter/run/390867\n  [2]: https://www.kaggle.com/arthurtok/titanic/simple-stacking-with-xgboost-0-808"},{"metadata":{"_execution_state":"idle","_cell_guid":"14630296-b1aa-759e-bafa-b6a73f3896ed","_uuid":"2e37a274400cfeb472b6405d524325245588dd66","trusted":true},"cell_type":"code","source":"# Load in our libraries\nimport pandas as pd\nimport numpy as np\nimport re\nimport sklearn\nimport xgboost as xgb\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Going to use these 5 base models for the stacking\nfrom sklearn.ensemble import (RandomForestClassifier, AdaBoostClassifier, \n                              GradientBoostingClassifier, ExtraTreesClassifier)\nfrom sklearn.svm import SVC\nfrom sklearn.cross_validation import KFold","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"8b590aafe06a2ac55daae9d2456155e457914f5f","_cell_guid":"d647b74c-099b-851a-dcd2-3a58c9e8f10c"},"cell_type":"markdown","source":"# Feature Exploration, Engineering and Cleaning \n\nNow we will proceed much like how most kernels in general are structured, and that is to first explore the data on hand, identify possible feature engineering opportunities as well as numerically encode any categorical features."},{"metadata":{"_execution_state":"idle","_cell_guid":"5937fd72-d1ad-f678-cc82-f08a96e4cad0","_uuid":"b2ad78041b69ce13d1f41bd9bc8c93cafaf7b8ac","trusted":true},"cell_type":"code","source":"# Load in the train and test datasets\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\n\n# Store our passenger ID for easy access\nPassengerId = test['PassengerId']\n\ntrain.head(3)","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"81378834770e55c76751347588322fe32acf5737","_cell_guid":"dff1c1dd-1e59-3907-88fa-1a1d699122be"},"cell_type":"markdown","source":"Well it is no surprise that our task is to somehow extract the information out of the categorical variables \n\n**Feature Engineering**\n\nHere, credit must be extended to Sina's very comprehensive and well-thought out notebook for the feature engineering ideas so please check out his work \n\n[Titanic Best Working Classfier][1] : by Sina\n\n\n  [1]: https://www.kaggle.com/sinakhorami/titanic/titanic-best-working-classifier"},{"metadata":{"collapsed":true,"_execution_state":"idle","_cell_guid":"e85b2a80-88a4-928f-f2b2-24895dea38f3","_uuid":"b1c67249f91768ce8e5e2751364d32c87446cf55","trusted":true},"cell_type":"code","source":"full_data = [train, test]\n\n# Some features of my own that I have added in\n# Gives the length of the name\ntrain['Name_length'] = train['Name'].apply(len)\ntest['Name_length'] = test['Name'].apply(len)\n# Feature that tells whether a passenger had a cabin on the Titanic\ntrain['Has_Cabin'] = train[\"Cabin\"].apply(lambda x: 0 if type(x) == float else 1)\ntest['Has_Cabin'] = test[\"Cabin\"].apply(lambda x: 0 if type(x) == float else 1)\n\n# Feature engineering steps taken from Sina\n# Create new feature FamilySize as a combination of SibSp and Parch\nfor dataset in full_data:\n    dataset['FamilySize'] = dataset['SibSp'] + dataset['Parch'] + 1\n# Create new feature IsAlone from FamilySize\nfor dataset in full_data:\n    dataset['IsAlone'] = 0\n    dataset.loc[dataset['FamilySize'] == 1, 'IsAlone'] = 1\n# Remove all NULLS in the Embarked column\nfor dataset in full_data:\n    dataset['Embarked'] = dataset['Embarked'].fillna('S')\n# Remove all NULLS in the Fare column and create a new feature CategoricalFare\nfor dataset in full_data:\n    dataset['Fare'] = dataset['Fare'].fillna(train['Fare'].median())\ntrain['CategoricalFare'] = pd.qcut(train['Fare'], 4)\n# Create a New feature CategoricalAge\nfor dataset in full_data:\n    age_avg = dataset['Age'].mean()\n    age_std = dataset['Age'].std()\n    age_null_count = dataset['Age'].isnull().sum()\n    age_null_random_list = np.random.randint(age_avg - age_std, age_avg + age_std, size=age_null_count)\n    dataset['Age'][np.isnan(dataset['Age'])] = age_null_random_list\n    dataset['Age'] = dataset['Age'].astype(int)\ntrain['CategoricalAge'] = pd.cut(train['Age'], 5)\n# Define function to extract titles from passenger names\ndef get_title(name):\n    title_search = re.search(' ([A-Za-z]+)\\.', name)\n    # If the title exists, extract and return it.\n    if title_search:\n        return title_search.group(1)\n    return \"\"\n# Create a new feature Title, containing the titles of passenger names\nfor dataset in full_data:\n    dataset['Title'] = dataset['Name'].apply(get_title)\n# Group all non-common titles into one single grouping \"Rare\"\nfor dataset in full_data:\n    dataset['Title'] = dataset['Title'].replace(['Lady', 'Countess','Capt', 'Col','Don', 'Dr', 'Major', 'Rev', 'Sir', 'Jonkheer', 'Dona'], 'Rare')\n\n    dataset['Title'] = dataset['Title'].replace('Mlle', 'Miss')\n    dataset['Title'] = dataset['Title'].replace('Ms', 'Miss')\n    dataset['Title'] = dataset['Title'].replace('Mme', 'Mrs')\n\nfor dataset in full_data:\n    # Mapping Sex\n    dataset['Sex'] = dataset['Sex'].map( {'female': 0, 'male': 1} ).astype(int)\n    \n    # Mapping titles\n    title_mapping = {\"Mr\": 1, \"Miss\": 2, \"Mrs\": 3, \"Master\": 4, \"Rare\": 5}\n    dataset['Title'] = dataset['Title'].map(title_mapping)\n    dataset['Title'] = dataset['Title'].fillna(0)\n    \n    # Mapping Embarked\n    dataset['Embarked'] = dataset['Embarked'].map( {'S': 0, 'C': 1, 'Q': 2} ).astype(int)\n    \n    # Mapping Fare\n    dataset.loc[ dataset['Fare'] <= 7.91, 'Fare'] \t\t\t\t\t\t        = 0\n    dataset.loc[(dataset['Fare'] > 7.91) & (dataset['Fare'] <= 14.454), 'Fare'] = 1\n    dataset.loc[(dataset['Fare'] > 14.454) & (dataset['Fare'] <= 31), 'Fare']   = 2\n    dataset.loc[ dataset['Fare'] > 31, 'Fare'] \t\t\t\t\t\t\t        = 3\n    dataset['Fare'] = dataset['Fare'].astype(int)\n    \n    # Mapping Age\n    dataset.loc[ dataset['Age'] <= 16, 'Age'] \t\t\t\t\t       = 0\n    dataset.loc[(dataset['Age'] > 16) & (dataset['Age'] <= 32), 'Age'] = 1\n    dataset.loc[(dataset['Age'] > 32) & (dataset['Age'] <= 48), 'Age'] = 2\n    dataset.loc[(dataset['Age'] > 48) & (dataset['Age'] <= 64), 'Age'] = 3\n    dataset.loc[ dataset['Age'] > 64, 'Age'] = 4 ;","execution_count":3,"outputs":[]},{"metadata":{"collapsed":true,"_execution_state":"idle","_cell_guid":"8fc645ba-4b38-cea7-17b3-02175cb103d9","_uuid":"ca2d48b03d45f914db2ee9ae3ee95aad8fb20431","trusted":true},"cell_type":"code","source":"# Feature selection\ndrop_elements = ['PassengerId', 'Name', 'Ticket', 'Cabin', 'SibSp']\ntrain = train.drop(drop_elements, axis = 1)\ntrain = train.drop(['CategoricalAge', 'CategoricalFare'], axis = 1)\ntest  = test.drop(drop_elements, axis = 1)","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"dd288776321804d99e4e4a7e88594c1d631e4409","_cell_guid":"e9814168-b7cd-d4e4-1b1d-e21c6637a663"},"cell_type":"markdown","source":"All right so now having cleaned the features and extracted relevant information and dropped the categorical columns our features should now all be numeric, a format suitable to feed into our Machine Learning models. However before we proceed let us generate some simple correlation and distribution plots of our transformed dataset to observe ho\n\n## Visualisations "},{"metadata":{"_execution_state":"idle","_cell_guid":"fc426b8f-873d-6f23-4299-99f174956cca","_uuid":"1f280a1c11dc35a93b57af494938998e6d0b4544","trusted":true},"cell_type":"code","source":"train.head(3)","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"3544b03040a1d691f6c48433c84d1e57f3c15e3d","_cell_guid":"41102927-8218-415a-0b73-5129c8f5dd0c"},"cell_type":"markdown","source":"**Pearson Correlation Heatmap**\n\nlet us generate some correlation plots of the features to see how related one feature is to the next. To do so, we will utilise the Seaborn plotting package which allows us to plot heatmaps very conveniently as follows"},{"metadata":{"_uuid":"b6142da58d9515979930abee19549aacf7f62c9f","_cell_guid":"af2eba12-b836-42a1-9ff3-d7a55bec8f9d","trusted":true},"cell_type":"code","source":"colormap = plt.cm.RdBu\nplt.figure(figsize=(14,12))\nplt.title('Pearson Correlation of Features', y=1.05, size=15)\nsns.heatmap(train.astype(float).corr(),linewidths=0.1,vmax=1.0, \n            square=True, cmap=colormap, linecolor='white', annot=True)","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"b6c7a1ea3866d177016a1e13c5d9e7341c49c147","_cell_guid":"ccc92745-0680-df4d-d709-10003475d8e3"},"cell_type":"markdown","source":"**Takeaway from the Plots**\n\nOne thing that that the Pearson Correlation plot can tell us is that there are not too many features strongly correlated with one another. This is good from a point of view of feeding these features into your learning model because this means that there isn't much redundant or superfluous data in our training set and we are happy that each feature carries with it some unique information. Here are two most correlated features are that of Family size and Parch (Parents and Children). I'll still leave both features in for the purposes of this exercise.\n\n**Pairplots**\n\nFinally let us generate some pairplots to observe the distribution of data from one feature to the other. Once again we use Seaborn to help us."},{"metadata":{"_execution_state":"idle","_cell_guid":"ea6b0a8f-5a33-666f-8057-c0d689f370f5","_uuid":"624446543aafd518025fd3f5346d32ee1aab6f9a","trusted":true},"cell_type":"code","source":"g = sns.pairplot(train[[u'Survived', u'Pclass', u'Sex', u'Age', u'Parch', u'Fare', u'Embarked',\n       u'FamilySize', u'Title']], hue='Survived', palette = 'seismic',size=1.2,diag_kind = 'kde',diag_kws=dict(shade=True),plot_kws=dict(s=10) )\ng.set(xticklabels=[])","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"b74f445f5d9fb9622c4609c00679254abb3c91b1","_cell_guid":"32ac3ce1-42bc-9a7d-44f3-4b7c025a921c"},"cell_type":"markdown","source":"# Ensembling & Stacking models\n\nFinally after that brief whirlwind detour with regards to feature engineering and formatting, we finally arrive at the meat and gist of the this notebook.\n\nCreating a Stacking ensemble!"},{"metadata":{"_uuid":"4e5fda8c6c92fc3cdf8e906089494f80e7c37245","_cell_guid":"3e922821-5a10-040b-305a-c2d47d633c49"},"cell_type":"markdown","source":"### Helpers via Python Classes\n\nHere we invoke the use of Python's classes to help make it more convenient for us. For any newcomers to programming, one normally hears Classes being used in conjunction with Object-Oriented Programming (OOP). In short, a class helps to extend some code/program for creating objects (variables for old-school peeps) as well as to implement functions and methods specific to that class.\n\nIn the section of code below, we essentially write a class *SklearnHelper* that allows one to extend the inbuilt methods (such as train, predict and fit) common to all the Sklearn classifiers. Therefore this cuts out redundancy as  won't need to write the same methods five times if we wanted to invoke five different classifiers."},{"metadata":{"collapsed":true,"_uuid":"04d921ea89a0560cf010e956e4065bb2eaf21619","_cell_guid":"c017c078-172d-16e9-65f2-4a01c6e0626f","trusted":true},"cell_type":"code","source":"# Some useful parameters which will come in handy later on\nntrain = train.shape[0]\nntest = test.shape[0]\nSEED = 0 # for reproducibility\nNFOLDS = 5 # set folds for out-of-fold prediction\nkf = KFold(ntrain, n_folds= NFOLDS, random_state=SEED)\n\n# Class to extend the Sklearn classifier\nclass SklearnHelper(object):\n    def __init__(self, clf, seed=0, params=None):\n        params['random_state'] = seed\n        self.clf = clf(**params)\n\n    def train(self, x_train, y_train):\n        self.clf.fit(x_train, y_train)\n\n    def predict(self, x):\n        return self.clf.predict(x)\n    \n    def fit(self,x,y):\n        return self.clf.fit(x,y)\n    \n    def feature_importances(self,x,y):\n        print(self.clf.fit(x,y).feature_importances_)\n    \n# Class to extend XGboost classifer","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"634199a5b36cba6b4d85db2c0401426a58b95279","_cell_guid":"75f59fa7-65a4-e53e-95df-4c747a608408"},"cell_type":"markdown","source":"Bear with me for those who already know this but for people who have not created classes or objects in Python before, let me explain what the code given above does. In creating my base classifiers, I will only use the models already present in the Sklearn library and therefore only extend the class for that.\n\n**def init** : Python standard for invoking the default constructor for the class. This means that when you want to create an object (classifier), you have to give it the parameters of clf (what sklearn classifier you want), seed (random seed) and params (parameters for the classifiers).\n\nThe rest of the code are simply methods of the class which simply call the corresponding methods already existing within the sklearn classifiers. Essentially, we have created a wrapper class to extend the various Sklearn classifiers so that this should help us reduce having to write the same code over and over when we implement multiple learners to our stacker."},{"metadata":{"_uuid":"4d193c581df258e823aff2796bf015cf906aac99","_cell_guid":"6f67620d-b531-a2fa-c297-e951970c3c28"},"cell_type":"markdown","source":"### Out-of-Fold Predictions\n\nNow as alluded to above in the introductory section, stacking uses predictions of base classifiers as input for training to a second-level model. However one cannot simply train the base models on the full training data, generate predictions on the full test set and then output these for the second-level training. This runs the risk of your base model predictions already having \"seen\" the test set and therefore overfitting when feeding these predictions."},{"metadata":{"collapsed":true,"_uuid":"46a93dc062e973832cecd50246d0d7581aafb02b","_cell_guid":"406d0494-1d0c-3126-19d9-bc53127c4249","trusted":true},"cell_type":"code","source":"def get_oof(clf, x_train, y_train, x_test):\n    oof_train = np.zeros((ntrain,))\n    oof_test = np.zeros((ntest,))\n    oof_test_skf = np.empty((NFOLDS, ntest))\n\n    for i, (train_index, test_index) in enumerate(kf):\n        x_tr = x_train[train_index]\n        y_tr = y_train[train_index]\n        x_te = x_train[test_index]\n\n        clf.train(x_tr, y_tr)\n\n        oof_train[test_index] = clf.predict(x_te)\n        oof_test_skf[i, :] = clf.predict(x_test)\n\n    oof_test[:] = oof_test_skf.mean(axis=0)\n    return oof_train.reshape(-1, 1), oof_test.reshape(-1, 1)","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"7b1a7767ae61b6b217a3311e89190b05ab0a4891","_cell_guid":"3cd92196-f7ba-4f14-0fc4-36520fbcb2ca"},"cell_type":"markdown","source":"# Generating our Base First-Level Models \n\nSo now let us prepare five learning models as our first level classification. These models can all be conveniently invoked via the Sklearn library and are listed as follows:\n\n 1. Random Forest classifier\n 2. Extra Trees classifier\n 3. AdaBoost classifer\n 4. Gradient Boosting classifer\n 5. Support Vector Machine"},{"metadata":{"_uuid":"12e3a5f76fb118ff6906431fc60e7010e33106ad","_cell_guid":"0ef6862a-b5cc-6829-f040-d2b2b2c817f3"},"cell_type":"markdown","source":"**Parameters**\n\nJust a quick summary of the parameters that we will be listing here for completeness,\n\n**n_jobs** : Number of cores used for the training process. If set to -1, all cores are used.\n\n**n_estimators** : Number of classification trees in your learning model ( set to 10 per default)\n\n**max_depth** : Maximum depth of tree, or how much a node should be expanded. Beware if set to too high  a number would run the risk of overfitting as one would be growing the tree too deep\n\n**verbose** : Controls whether you want to output any text during the learning process. A value of 0 suppresses all text while a value of 3 outputs the tree learning process at every iteration.\n\n Please check out the full description via the official Sklearn website. There you will find that there are a whole host of other useful parameters that you can play around with. "},{"metadata":{"collapsed":true,"_uuid":"d77772886c0125e022d1fbb39cd484c95121d74d","_cell_guid":"6e634aba-90b6-0620-eceb-3e1a39fbfedc","trusted":true},"cell_type":"code","source":"# Put in our parameters for said classifiers\n# Random Forest parameters\nrf_params = {\n    'n_jobs': -1,\n    'n_estimators': 500,\n     'warm_start': True, \n     #'max_features': 0.2,\n    'max_depth': 6,\n    'min_samples_leaf': 2,\n    'max_features' : 'sqrt',\n    'verbose': 0\n}\n\n# Extra Trees Parameters\net_params = {\n    'n_jobs': -1,\n    'n_estimators':500,\n    #'max_features': 0.5,\n    'max_depth': 8,\n    'min_samples_leaf': 2,\n    'verbose': 0\n}\n\n# AdaBoost parameters\nada_params = {\n    'n_estimators': 500,\n    'learning_rate' : 0.75\n}\n\n# Gradient Boosting parameters\ngb_params = {\n    'n_estimators': 500,\n     #'max_features': 0.2,\n    'max_depth': 5,\n    'min_samples_leaf': 2,\n    'verbose': 0\n}\n\n# Support Vector Classifier parameters \nsvc_params = {\n    'kernel' : 'linear',\n    'C' : 0.025\n    }","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"4a49a6c0cbe7028035b6efb7f9374084f3fa152a","_cell_guid":"ee6325d7-a7c1-c767-fcba-4c59eaa83787"},"cell_type":"markdown","source":"Furthermore, since having mentioned about Objects and classes within the OOP framework, let us now create 5 objects that represent our 5 learning models via our Helper Sklearn Class we defined earlier."},{"metadata":{"collapsed":true,"_uuid":"3bd54b4dfbbedc86d6c415cf83936bb8cd6c0973","_cell_guid":"6798243f-5ff9-527b-01b1-09cfe62284bb","trusted":true},"cell_type":"code","source":"# Create 5 objects that represent our 4 models\nrf = SklearnHelper(clf=RandomForestClassifier, seed=SEED, params=rf_params)\net = SklearnHelper(clf=ExtraTreesClassifier, seed=SEED, params=et_params)\nada = SklearnHelper(clf=AdaBoostClassifier, seed=SEED, params=ada_params)\ngb = SklearnHelper(clf=GradientBoostingClassifier, seed=SEED, params=gb_params)\nsvc = SklearnHelper(clf=SVC, seed=SEED, params=svc_params)","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"cd3f527543e61b3841f242b744c3c5d9af608941","_cell_guid":"46e1e6fc-92e2-e7e2-1ab9-470e6c7039ce"},"cell_type":"markdown","source":"**Creating NumPy arrays out of our train and test sets**\n\nGreat. Having prepared our first layer base models as such, we can now ready the training and test test data for input into our classifiers by generating NumPy arrays out of their original dataframes as follows:"},{"metadata":{"collapsed":true,"_uuid":"95fdd1e8cd9f23cfcef3bed92511da084a323c55","_cell_guid":"968cbd7f-80b2-7f8d-2ad6-b68b3aeae671","trusted":true},"cell_type":"code","source":"# Create Numpy arrays of train, test and target ( Survived) dataframes to feed into our models\ny_train = train['Survived'].ravel()\ntrain = train.drop(['Survived'], axis=1)\nx_train = train.values # Creates an array of the train data\nx_test = test.values # Creats an array of the test data","execution_count":12,"outputs":[]},{"metadata":{"_uuid":"d30ac9528b69cdaec565b647fee98a92391112c9","_cell_guid":"606b619c-2301-8aaa-a5de-781d981c4a6f"},"cell_type":"markdown","source":"**Output of the First level Predictions** \n\nWe now feed the training and test data into our 5 base classifiers and use the Out-of-Fold prediction function we defined earlier to generate our first level predictions. Allow a handful of minutes for the chunk of code below to run."},{"metadata":{"_uuid":"114750e2d5e4fdd234ccd8647fc349463a56fa09","_cell_guid":"79bd2a86-82e2-648a-e816-9660e89794ad","trusted":true},"cell_type":"code","source":"# Create our OOF train and test predictions. These base results will be used as new features\net_oof_train, et_oof_test = get_oof(et, x_train, y_train, x_test) # Extra Trees\nrf_oof_train, rf_oof_test = get_oof(rf,x_train, y_train, x_test) # Random Forest\nada_oof_train, ada_oof_test = get_oof(ada, x_train, y_train, x_test) # AdaBoost \ngb_oof_train, gb_oof_test = get_oof(gb,x_train, y_train, x_test) # Gradient Boost\nsvc_oof_train, svc_oof_test = get_oof(svc,x_train, y_train, x_test) # Support Vector Classifier\n\nprint(\"Training is complete\")","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"4cf41b3d9a541c9d39b645a66c8f1116eaf76861","_cell_guid":"3f292e65-fe8a-d662-6ace-41a19866d671"},"cell_type":"markdown","source":"**Feature importances generated from the different classifiers**\n\nNow having learned our the first-level classifiers, we can utilise a very nifty feature of the Sklearn models and that is to output the importances of the various features in the training and test sets with one very simple line of code.\n\nAs per the Sklearn documentation, most of the classifiers are built in with an attribute which returns feature importances by simply typing in **.feature_importances_**. Therefore we will invoke this very useful attribute via our function earliand plot the feature importances as such"},{"metadata":{"_uuid":"b3b0356c8bef0dceb5fcfa7fb7a11359010b2098","_cell_guid":"ed9cf8b5-95a4-d974-fb11-592214949d1f","trusted":true},"cell_type":"code","source":"rf_feature = rf.feature_importances(x_train,y_train)\net_feature = et.feature_importances(x_train, y_train)\nada_feature = ada.feature_importances(x_train, y_train)\ngb_feature = gb.feature_importances(x_train,y_train)","execution_count":14,"outputs":[]},{"metadata":{"_uuid":"b0c66aa58b45b917157f47c811e7d222513ddb0d","_cell_guid":"f1be8fbb-34c0-3f92-c7d2-757fceced507"},"cell_type":"markdown","source":"So I have not yet figured out how to assign and store the feature importances outright. Therefore I'll print out the values from the code above and then simply copy and paste into Python lists as below (sorry for the lousy hack)"},{"metadata":{"collapsed":true,"_uuid":"527425374ec8cd66edc015842f352a95041821c6","_cell_guid":"bde9b5fd-3100-8f21-053e-45015b99cf65","trusted":true},"cell_type":"code","source":"rf_features = [0.10474135,  0.21837029,  0.04432652,  0.02249159,  0.05432591,  0.02854371\n  ,0.07570305,  0.01088129 , 0.24247496,  0.13685733 , 0.06128402]\net_features = [ 0.12165657,  0.37098307  ,0.03129623 , 0.01591611 , 0.05525811 , 0.028157\n  ,0.04589793 , 0.02030357 , 0.17289562 , 0.04853517,  0.08910063]\nada_features = [0.028 ,   0.008  ,      0.012   ,     0.05866667,   0.032 ,       0.008\n  ,0.04666667 ,  0.     ,      0.05733333,   0.73866667,   0.01066667]\ngb_features = [ 0.06796144 , 0.03889349 , 0.07237845 , 0.02628645 , 0.11194395,  0.04778854\n  ,0.05965792 , 0.02774745,  0.07462718,  0.4593142 ,  0.01340093]","execution_count":15,"outputs":[]},{"metadata":{"_uuid":"d59169f89c5541177f537200a90805420b449001","_cell_guid":"2a03ad8f-0ea1-5afa-a6e8-56284482c646"},"cell_type":"markdown","source":"Create a dataframe from the lists containing the feature importance data for easy plotting via the Plotly package."},{"metadata":{"collapsed":true,"_uuid":"6f68b3033a8f185f61d83e80323c2486024f5d4d","_cell_guid":"635a063f-281d-66d4-6572-587ebecd6b4b","trusted":true},"cell_type":"code","source":"cols = train.columns.values\n# Create a dataframe with features\nfeature_dataframe = pd.DataFrame( {'features': cols,\n     'Random Forest feature importances': rf_features,\n     'Extra Trees  feature importances': et_features,\n      'AdaBoost feature importances': ada_features,\n    'Gradient Boost feature importances': gb_features\n    })","execution_count":16,"outputs":[]},{"metadata":{"_uuid":"7e25675f239b0ab008e0264917abff497795681a","_cell_guid":"06b9f410-f93e-0206-b029-24df035eea2b"},"cell_type":"markdown","source":"**Interactive feature importances via Plotly scatterplots**\n\nI'll use the interactive Plotly package at this juncture to visualise the feature importances values of the different classifiers  via a plotly scatter plot by calling \"Scatter\" as follows:"},{"metadata":{"_uuid":"d8ee9114cd391433835f1272ef81d0a729c78b71","_cell_guid":"1ac351c6-83c6-c35b-9d66-64f16b5d073f","trusted":true},"cell_type":"code","source":"# Scatter plot \ntrace = go.Scatter(\n    y = feature_dataframe['Random Forest feature importances'].values,\n    x = feature_dataframe['features'].values,\n    mode='markers',\n    marker=dict(\n        sizemode = 'diameter',\n        sizeref = 1,\n        size = 25,\n#       size= feature_dataframe['AdaBoost feature importances'].values,\n        #color = np.random.randn(500), #set color equal to a variable\n        color = feature_dataframe['Random Forest feature importances'].values,\n        colorscale='Portland',\n        showscale=True\n    ),\n    text = feature_dataframe['features'].values\n)\ndata = [trace]\n\nlayout= go.Layout(\n    autosize= True,\n    title= 'Random Forest Feature Importance',\n    hovermode= 'closest',\n#     xaxis= dict(\n#         title= 'Pop',\n#         ticklen= 5,\n#         zeroline= False,\n#         gridwidth= 2,\n#     ),\n    yaxis=dict(\n        title= 'Feature Importance',\n        ticklen= 5,\n        gridwidth= 2\n    ),\n    showlegend= False\n)\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig,filename='scatter2010')\n\n# Scatter plot \ntrace = go.Scatter(\n    y = feature_dataframe['Extra Trees  feature importances'].values,\n    x = feature_dataframe['features'].values,\n    mode='markers',\n    marker=dict(\n        sizemode = 'diameter',\n        sizeref = 1,\n        size = 25,\n#       size= feature_dataframe['AdaBoost feature importances'].values,\n        #color = np.random.randn(500), #set color equal to a variable\n        color = feature_dataframe['Extra Trees  feature importances'].values,\n        colorscale='Portland',\n        showscale=True\n    ),\n    text = feature_dataframe['features'].values\n)\ndata = [trace]\n\nlayout= go.Layout(\n    autosize= True,\n    title= 'Extra Trees Feature Importance',\n    hovermode= 'closest',\n#     xaxis= dict(\n#         title= 'Pop',\n#         ticklen= 5,\n#         zeroline= False,\n#         gridwidth= 2,\n#     ),\n    yaxis=dict(\n        title= 'Feature Importance',\n        ticklen= 5,\n        gridwidth= 2\n    ),\n    showlegend= False\n)\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig,filename='scatter2010')\n\n# Scatter plot \ntrace = go.Scatter(\n    y = feature_dataframe['AdaBoost feature importances'].values,\n    x = feature_dataframe['features'].values,\n    mode='markers',\n    marker=dict(\n        sizemode = 'diameter',\n        sizeref = 1,\n        size = 25,\n#       size= feature_dataframe['AdaBoost feature importances'].values,\n        #color = np.random.randn(500), #set color equal to a variable\n        color = feature_dataframe['AdaBoost feature importances'].values,\n        colorscale='Portland',\n        showscale=True\n    ),\n    text = feature_dataframe['features'].values\n)\ndata = [trace]\n\nlayout= go.Layout(\n    autosize= True,\n    title= 'AdaBoost Feature Importance',\n    hovermode= 'closest',\n#     xaxis= dict(\n#         title= 'Pop',\n#         ticklen= 5,\n#         zeroline= False,\n#         gridwidth= 2,\n#     ),\n    yaxis=dict(\n        title= 'Feature Importance',\n        ticklen= 5,\n        gridwidth= 2\n    ),\n    showlegend= False\n)\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig,filename='scatter2010')\n\n# Scatter plot \ntrace = go.Scatter(\n    y = feature_dataframe['Gradient Boost feature importances'].values,\n    x = feature_dataframe['features'].values,\n    mode='markers',\n    marker=dict(\n        sizemode = 'diameter',\n        sizeref = 1,\n        size = 25,\n#       size= feature_dataframe['AdaBoost feature importances'].values,\n        #color = np.random.randn(500), #set color equal to a variable\n        color = feature_dataframe['Gradient Boost feature importances'].values,\n        colorscale='Portland',\n        showscale=True\n    ),\n    text = feature_dataframe['features'].values\n)\ndata = [trace]\n\nlayout= go.Layout(\n    autosize= True,\n    title= 'Gradient Boosting Feature Importance',\n    hovermode= 'closest',\n#     xaxis= dict(\n#         title= 'Pop',\n#         ticklen= 5,\n#         zeroline= False,\n#         gridwidth= 2,\n#     ),\n    yaxis=dict(\n        title= 'Feature Importance',\n        ticklen= 5,\n        gridwidth= 2\n    ),\n    showlegend= False\n)\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig,filename='scatter2010')","execution_count":17,"outputs":[]},{"metadata":{"_uuid":"9d7b8fdd0c3102d7e3ddcffaf26ce19b02e5ad74","_cell_guid":"553828f0-c994-5ee1-695f-9373f11a1a7b"},"cell_type":"markdown","source":"Now let us calculate the mean of all the feature importances and store it as a new column in the feature importance dataframe."},{"metadata":{"_uuid":"f611812e2c9de3773df2264dfb2b13c0995807ac","_cell_guid":"06847850-a829-0858-b12c-7b66e53e030a","trusted":true},"cell_type":"code","source":"# Create the new column containing the average of values\n\nfeature_dataframe['mean'] = feature_dataframe.mean(axis= 1) # axis = 1 computes the mean row-wise\nfeature_dataframe.head(3)","execution_count":18,"outputs":[]},{"metadata":{"_uuid":"1f0aff0896fef90b326fff7816393fef0e0cb992","_cell_guid":"5645e647-c517-7822-f881-b8d7e38ef5da"},"cell_type":"markdown","source":"**Plotly Barplot of Average Feature Importances**\n\nHaving obtained the mean feature importance across all our classifiers, we can plot them into a Plotly bar plot as follows:"},{"metadata":{"_uuid":"0bd069388b419fe45306c01825aa3e6f5466ba2b","_cell_guid":"63d86121-8c29-4b7f-b2ad-12b0a593f1d6","trusted":true},"cell_type":"code","source":"y = feature_dataframe['mean'].values\nx = feature_dataframe['features'].values\ndata = [go.Bar(\n            x= x,\n             y= y,\n            width = 0.5,\n            marker=dict(\n               color = feature_dataframe['mean'].values,\n            colorscale='Portland',\n            showscale=True,\n            reversescale = False\n            ),\n            opacity=0.6\n        )]\n\nlayout= go.Layout(\n    autosize= True,\n    title= 'Barplots of Mean Feature Importance',\n    hovermode= 'closest',\n#     xaxis= dict(\n#         title= 'Pop',\n#         ticklen= 5,\n#         zeroline= False,\n#         gridwidth= 2,\n#     ),\n    yaxis=dict(\n        title= 'Feature Importance',\n        ticklen= 5,\n        gridwidth= 2\n    ),\n    showlegend= False\n)\nfig = go.Figure(data=data, layout=layout)\npy.iplot(fig, filename='bar-direct-labels')","execution_count":19,"outputs":[]},{"metadata":{"_uuid":"bbb76d189e8d03921caaacfa9545cef894348c7d","_cell_guid":"c1201ecc-b07d-f8a1-0870-b8d78c89ebc0"},"cell_type":"markdown","source":"# Second-Level Predictions from the First-level Output"},{"metadata":{"_uuid":"fed132782b73dda8d265065867e7f57c0aed7f50","_cell_guid":"6b901750-ccdd-38ca-d8ea-1c361121ec4f"},"cell_type":"markdown","source":"**First-level output as new features**\n\nHaving now obtained our first-level predictions, one can think of it as essentially building a new set of features to be used as training data for the next classifier. As per the code below, we are therefore having as our new columns the first-level predictions from our earlier classifiers and we train the next classifier on this."},{"metadata":{"_uuid":"a5945e93337b87a1a8ee5580856768bbb14c07cd","_cell_guid":"7330a71c-0b71-87c2-1f4d-dd0f6d6fa586","trusted":true},"cell_type":"code","source":"base_predictions_train = pd.DataFrame( {'RandomForest': rf_oof_train.ravel(),\n     'ExtraTrees': et_oof_train.ravel(),\n     'AdaBoost': ada_oof_train.ravel(),\n      'GradientBoost': gb_oof_train.ravel()\n    })\nbase_predictions_train.head()","execution_count":20,"outputs":[]},{"metadata":{"_uuid":"2a6f987bd9b8ffc32a72e21cb8c43a6bc43ba200","_cell_guid":"f69c11db-d84e-8536-4c7e-382fbe67483e"},"cell_type":"markdown","source":"**Correlation Heatmap of the Second Level Training set**"},{"metadata":{"_uuid":"9714ecaedf7385c5b8ad346ab909215eb9f2abc6","_cell_guid":"4cf590ee-133f-6487-cf5a-53f346893d1c","trusted":true},"cell_type":"code","source":"data = [\n    go.Heatmap(\n        z= base_predictions_train.astype(float).corr().values ,\n        x=base_predictions_train.columns.values,\n        y= base_predictions_train.columns.values,\n          colorscale='Viridis',\n            showscale=True,\n            reversescale = True\n    )\n]\npy.iplot(data, filename='labelled-heatmap')","execution_count":21,"outputs":[]},{"metadata":{"_uuid":"4d9ef0298b568e43da6925f385403e0d77bd6e33","_cell_guid":"4d6b61c0-5d72-b02a-3b37-cbf6518d71b6"},"cell_type":"markdown","source":"There have been quite a few articles and Kaggle competition winner stories about the merits of having trained models that are more uncorrelated with one another producing better scores."},{"metadata":{"collapsed":true,"_uuid":"fef365199854ca3fff754399b4699d941b7e43b8","_cell_guid":"6685fa11-497f-3fc2-ab1f-97f92d6eca61","trusted":true},"cell_type":"code","source":"x_train = np.concatenate(( et_oof_train, rf_oof_train, ada_oof_train, gb_oof_train, svc_oof_train), axis=1)\nx_test = np.concatenate(( et_oof_test, rf_oof_test, ada_oof_test, gb_oof_test, svc_oof_test), axis=1)","execution_count":22,"outputs":[]},{"metadata":{"_uuid":"65727ae393d3f7118215fde76e4fd5a9d0e9dd6c","_cell_guid":"a02a94ab-3c9c-a824-7168-e964c5a0f5d5"},"cell_type":"markdown","source":"Having now concatenated and joined both the first-level train and test predictions as x_train and x_test, we can now fit a second-level learning model."},{"metadata":{"_uuid":"dc4a32e9a8e7c9e611124cba676e5d28240b38be","_cell_guid":"628a03ea-933c-7075-a589-0ff7af237dfd"},"cell_type":"markdown","source":"### Second level learning model via XGBoost\n\nHere we choose the eXtremely famous library for boosted tree learning model, XGBoost. It was built to optimize large-scale boosted tree algorithms. For further information about the algorithm, check out the [official documentation][1].\n\n  [1]: https://xgboost.readthedocs.io/en/latest/\n\nAnyways, we call an XGBClassifier and fit it to the first-level train and target data and use the learned model to predict the test data as follows:"},{"metadata":{"collapsed":true,"_uuid":"5155d370069fe6de0fe5105309342ce55130dae8","_cell_guid":"3a7c7517-b9a3-3a21-3a7b-299ca37c6843","trusted":true},"cell_type":"code","source":"gbm = xgb.XGBClassifier(\n    #learning_rate = 0.02,\n n_estimators= 2000,\n max_depth= 4,\n min_child_weight= 2,\n #gamma=1,\n gamma=0.9,                        \n subsample=0.8,\n colsample_bytree=0.8,\n objective= 'binary:logistic',\n nthread= -1,\n scale_pos_weight=1).fit(x_train, y_train)\npredictions = gbm.predict(x_test)","execution_count":23,"outputs":[]},{"metadata":{"_uuid":"0101e6b843f6378838874ccfb844ed464b81d627","_cell_guid":"0a8152d8-6842-ed00-6bc5-47a511adce1c"},"cell_type":"markdown","source":"Just a quick run down of the XGBoost parameters used in the model:\n\n**max_depth** : How deep you want to grow your tree. Beware if set to too high a number might run the risk of overfitting.\n\n**gamma** : minimum loss reduction required to make a further partition on a leaf node of the tree. The larger, the more conservative the algorithm will be.\n\n**eta** : step size shrinkage used in each boosting step to prevent overfitting"},{"metadata":{"_uuid":"52ac0cd99cee0099d86a180127da42ff7fff960a","_cell_guid":"6b4a5c81-e968-d41e-27e4-871481019867"},"cell_type":"markdown","source":"**Producing the Submission file**\n\nFinally having trained and fit all our first-level and second-level models, we can now output the predictions into the proper format for submission to the Titanic competition as follows:"},{"metadata":{"collapsed":true,"_uuid":"9d607d829dbadd6c72ee01c9735a642435eb53e6","_cell_guid":"f5a31787-5fe1-a559-bee9-ad6b6d83ae14","trusted":true},"cell_type":"code","source":"# Generate Submission File \nStackingSubmission = pd.DataFrame({ 'PassengerId': PassengerId,\n                            'Survived': predictions })\nStackingSubmission.to_csv(\"StackingSubmission.csv\", index=False)","execution_count":24,"outputs":[]},{"metadata":{"_uuid":"9db5fd7cbb0d406ab0ef9aa08cf56532c51ec8b5","_cell_guid":"1e56c738-b8f3-95e4-d642-c483f9757ed8"},"cell_type":"markdown","source":"**Steps for Further Improvement**\n\nAs a closing remark it must be noted that the steps taken above just show a very simple way of producing an ensemble stacker. You hear of ensembles created at the highest level of Kaggle competitions which involves monstrous combinations of stacked classifiers as well as levels of stacking which go to more than 2 levels. \n\nSome additional steps that may be taken to improve one's score could be:\n\n 1. Implementing a good cross-validation strategy in training the models to find optimal parameter values\n 2. Introduce a greater variety of base models for learning. The more uncorrelated the results, the better the final score."},{"metadata":{"_uuid":"c32d1d64e1a5f8fbe5f51a0a7afd952ccfdec57e","_cell_guid":"9a8f83fd-d0e8-035a-cf7f-25c9012e9373"},"cell_type":"markdown","source":"### Conclusion\n\nI have this notebook has been helpful somewhat in introducing a working script for stacking learning models. Again credit must be extended to Faron and Sina. \n\nFor other excellent material on stacking or ensembling in general, refer to the de-facto Must read article on the website MLWave: [Kaggle Ensembling Guide][1]. \n\nTill next time, Peace Out\n\n  [1]: http://mlwave.com/kaggle-ensembling-guide/"},{"metadata":{"collapsed":true,"_uuid":"8549710f41f2ca55f11400e7914ed2f15c380964","_cell_guid":"74641376-e124-e46f-0e7c-8cbee2a1321b","trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"_change_revision":0,"_is_fork":false,"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}