NY Taxi Fare - Comprehensive and Simple Analysis.ipynb

{"cells":[{"metadata":{"_uuid":"2dad935f3cf633c9e8ac9f7ebf7199284d5195a1"},"cell_type":"markdown","source":"# Google New York Taxi Fare Prediction"},{"metadata":{"_uuid":"e486dfd0811be4048a180f5553d2fb671e5c8158"},"cell_type":"markdown","source":"This is a short kernel I made for myself and others as a reference guide/tutorial whilst I'm working out the best way to make accurate predictions. It's probably the most useful to beginners and I did try to describe/explain almost everything. I hope you find this useful, feel free to comment or contact me with any suggestions or questions."},{"metadata":{"_uuid":"ece45b62b3957331853a41b26aae4542336e287c"},"cell_type":"markdown","source":"![](https://kaggle2.blob.core.windows.net/competitions/kaggle/10170/logos/header.png?t=2018-07-12-22-07-30)"},{"metadata":{"_uuid":"77f705b587b373d53161b4b4aa8f41061d0a7482"},"cell_type":"markdown","source":"# Import Libraries and Data"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom geopy.distance import great_circle #calculate distances\nfrom sklearn import metrics #evaluating models\n#from sklearn.preprocessing import StandardScaler, RobustScaler\nfrom sklearn.model_selection import train_test_split, cross_val_score #set splitting and validation\nfrom sklearn.linear_model import LinearRegression \nimport xgboost as xgb #XGBoost classifier\nimport matplotlib.pyplot as plt #plotting\nimport seaborn as sns #plotting\nfrom math import sin, cos, sqrt, atan2, radians\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"collapsed":true},"cell_type":"code","source":"#Check for the comptetition files\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48966199c35e8b571c0a27865e6db5b3c5c2b914","collapsed":true},"cell_type":"code","source":"#Load test dataset\ntest = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0740164a65aca0ed7540cb00470343d7f27c61b3","collapsed":true},"cell_type":"code","source":"#Get types of values\ntest.dtypes","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"688ffaef23b6bf802ff99012d036701955e37765"},"cell_type":"markdown","source":"As pointed out in this kernel https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows there is no need to use float64 for GPS coordinates, and float32 will provide just as accurate results and allow you to import and process data much quicker."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"318fae3a54c8577dd518a4581b1e84d102b3583e"},"cell_type":"code","source":"#Set lighter types to a dictionary to speed up train dataset (float64 is an overkill for GPS coordinates)\ntypes = {'fare_amount': 'float32',\n         'pickup_longitude': 'float32',\n         'pickup_latitude': 'float32',\n         'dropoff_longitude': 'float32',\n         'dropoff_latitude': 'float32',\n         'passenger_count': 'uint8'}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2da835a57027e504aa5cdd10d3743f5385ed4100","collapsed":true},"cell_type":"code","source":"#Load portion of the dataset in the defined types\ntrain = pd.read_csv('../input/train.csv',nrows=100000,dtype=types)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2622cf2aca88d5184c756604e071e99d1e74127"},"cell_type":"markdown","source":"# Data Exploration and Cleanup"},{"metadata":{"_uuid":"bfcfa15c386bd66dc13a58130dfdd0a9f3a372e7"},"cell_type":"markdown","source":"Now let's look at what the data look like and some general statistics about it."},{"metadata":{"trusted":true,"_uuid":"898f3e955a00c2c624ca82a973ccad644c4765c5","collapsed":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"93d22a102770e74a76f3c175d7261a4fde8534ca","collapsed":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cdd68e8d269c09c5ce88868fb3c3bcd259533947"},"cell_type":"markdown","source":"Let's plot distribution of couple of the values to get an easier insight into the data and the way it's distributed. You can see that there are a few values which are 0 dollars or less on fare_amount or 0 passengers on passenger_count. Feel free to check distributions of other values."},{"metadata":{"trusted":true,"_uuid":"6c78cde46c12cdf907c6c719f2e9274db3c80307","collapsed":true},"cell_type":"code","source":"#Distribution plot of fares\nsns.distplot(train['fare_amount'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"14e5b15850a19959df77fe8dccecb86eec212412","collapsed":true},"cell_type":"code","source":"#Distribution plot of passanger count\nsns.distplot(train['passenger_count'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44d6faa31ad1ddf0c9fc65ebe0cc28fe962a9c84"},"cell_type":"markdown","source":"Based on just a quick look at the data, we can see that it's not 100% clean and some entries will contribute to higher error rates. As there are more than enough entries, we can easily sacrifice any rows that have null data. Note that depending on how large of a slice of training data you took, you might not see any null values, but there are some in the 6 million rows provided. Just in case let's drop them. Additionally we can drop all entries where fare_amount or passanger_count is less or equal to 0. We can do a similar cleanup with coordinates, as the dataset is on New York taxi fares, and some of the values go beyond New York coordinates, and some of them are clearly errors."},{"metadata":{"trusted":true,"_uuid":"bec00c004b5d22254064a5363818d4db5bb0d8c0","collapsed":true},"cell_type":"code","source":"#Check how many rows have null values\ntrain.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"431f55ded9d8b9e52df2e2e00b57ae0eb41b5a64","collapsed":true},"cell_type":"code","source":"#Drop nulls if exist\ntrain.dropna(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"11cc664af06d9ccb6c614a834e568aa84603d777","collapsed":true},"cell_type":"code","source":"#Clean up the trian dataset to eliminate out of range values\ntrain = train[train['fare_amount'] > 0]\ntrain = train[train['pickup_longitude'] < -72]\ntrain = train[(train['pickup_latitude'] > 40) & (train['pickup_latitude'] < 44)]\ntrain = train[train['dropoff_longitude'] < -72]\ntrain = train[(train['dropoff_latitude'] > 40) & (train['dropoff_latitude'] < 44)]\ntrain = train[(train['passenger_count'] > 0) & (train['passenger_count'] < 10)]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87b3dc548574836fb8af6ed60d493582390fb566"},"cell_type":"markdown","source":"Now we can see there are no obvious inconsitentcies with the data."},{"metadata":{"trusted":true,"_uuid":"e6063be2bb8b13865322cbcc6500d6a95a72ebfc","collapsed":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c17549ac5b186af7159bcfb4790b955f023748e"},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{"_uuid":"8397c44d0b4f09ffbbb9799b744aafea9eb1a288"},"cell_type":"markdown","source":"We're only provided with a 8 columns of directly useable data, however we can extract much more information from it by engineering features from those columns or combinations of them. To start with we can get the distanct between the pickup and dropoff points, which should be a strong predictor for the fare. "},{"metadata":{"trusted":true,"_uuid":"291eb371721ffa6809846f503eaf9e2c45dafbb1","collapsed":true},"cell_type":"code","source":"#Define function to calculate distance in km from coordinates\ndef dist_calc(df):\n    for i,row in df.iterrows():\n        df.at[i,'distance'] = great_circle((row['pickup_latitude'],row['pickup_longitude']),(row['dropoff_latitude'],row['dropoff_longitude'])).km","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d59593feeb0433b5c3b494c6e0ad19d1de292a7b"},"cell_type":"code","source":"#Quicker but slightly less accurate\ndef quick_dist_calc(df):\n    R = 6373.0\n    for i,row in df.iterrows():\n\n        lat1 = radians(row['pickup_latitude'])\n        lon1 = radians(row['pickup_longitude'])\n        lat2 = radians(row['dropoff_latitude'])\n        lon2 = radians(row['dropoff_longitude'])\n\n        dlon = lon2 - lon1\n        dlat = lat2 - lat1\n\n        a = sin(dlat / 2)**2 + cos(lat1) * cos(lat2) * sin(dlon / 2)**2\n        c = 2 * atan2(sqrt(a), sqrt(1 - a))\n\n        distance = R * c\n        df.at[i,'distance'] = distance","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75adc7fc22c18148fa57f50961695e1682871f8c","collapsed":true},"cell_type":"code","source":"#Get distance for both sets\nquick_dist_calc(train)\nquick_dist_calc(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"60b143599a2df87b005e5173fbacbfac0b1a68d5"},"cell_type":"markdown","source":"Then if you look at the pickup_datetime column, we can extract quite a few datetime related features from it. First we need to format the string column into datetime format, after which it will be easy getting atomic features like hour, day, year, etc. of the pickup. "},{"metadata":{"trusted":true,"_uuid":"9a86889e8073b1a258c884c8b66b03557137407c","collapsed":true},"cell_type":"code","source":"#Get useable date for feature engineering\ntrain['pickup_datetime'] = train['pickup_datetime'].str.replace(\" UTC\", \"\")\ntrain['pickup_datetime'] = pd.to_datetime(train['pickup_datetime'], format='%Y-%m-%d %H:%M:%S')\n\ntest['pickup_datetime'] = test['pickup_datetime'].str.replace(\" UTC\", \"\")\ntest['pickup_datetime'] = pd.to_datetime(test['pickup_datetime'], format='%Y-%m-%d %H:%M:%S')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"347fdfef7c06e5552d3893b5ef4747abc8eb7a16","collapsed":true},"cell_type":"code","source":"#Getting interger numbers from the pickup_datetime\ntrain[\"hour\"] = train.pickup_datetime.dt.hour\ntrain[\"weekday\"] = train.pickup_datetime.dt.weekday\ntrain[\"month\"] = train.pickup_datetime.dt.month\ntrain[\"year\"] = train.pickup_datetime.dt.year\n\ntest[\"hour\"] = test.pickup_datetime.dt.hour\ntest[\"weekday\"] = test.pickup_datetime.dt.weekday\ntest[\"month\"] = test.pickup_datetime.dt.month\ntest[\"year\"] = test.pickup_datetime.dt.year","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6f373b52361bae81f0f292767c108fe8b0c01381"},"cell_type":"markdown","source":"Last but not least, we can look at fare correlation with the distance from certain hotspots around New York where prices will be higher or lower than usual. You can have a look at this great kernel https://www.kaggle.com/shaz13/simple-exploration-notebook-map-plots-v2 where the fare data is plotted geographically and gives an nice visual representation of such places. The most obvious ones would be airports and the center of Manhattan. Check out this kernel https://www.kaggle.com/gunbl4d3/xgboost-ing-taxi-fares for getting distance to and from higher fare hotspots. "},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c7f2d821bcc322fee0d87456f25c4a720fec958b"},"cell_type":"code","source":"#Function for distance calculation between coordinates as mapped variables\ndef sphere_dist(pickup_lat, pickup_lon, dropoff_lat, dropoff_lon):\n    #Define earth radius (km)\n    R_earth = 6371\n    #Convert degrees to radians\n    pickup_lat, pickup_lon, dropoff_lat, dropoff_lon = map(np.radians,\n                                                             [pickup_lat, pickup_lon, \n                                                              dropoff_lat, dropoff_lon])\n    #Compute distances along lat, lon dimensions\n    dlat = dropoff_lat - pickup_lat\n    dlon = dropoff_lon - pickup_lon\n    \n    #Compute haversine distance\n    a = np.sin(dlat/2.0)**2 + np.cos(pickup_lat) * np.cos(dropoff_lat) * np.sin(dlon/2.0)**2\n    \n    return 2 * R_earth * np.arcsin(np.sqrt(a))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"037e5cc188bb2d97b0ccc376c3df1ff586db4816"},"cell_type":"code","source":"#Function for calculating distance between newly obtained distances from the hotspots.\ndef add_airport_dist(dataset):\n    jfk_coord = (40.639722, -73.778889)\n    ewr_coord = (40.6925, -74.168611)\n    lga_coord = (40.77725, -73.872611)\n    \n    pickup_lat = dataset['pickup_latitude']\n    dropoff_lat = dataset['dropoff_latitude']\n    pickup_lon = dataset['pickup_longitude']\n    dropoff_lon = dataset['dropoff_longitude']\n    \n    pickup_jfk = sphere_dist(pickup_lat, pickup_lon, jfk_coord[0], jfk_coord[1]) \n    dropoff_jfk = sphere_dist(jfk_coord[0], jfk_coord[1], dropoff_lat, dropoff_lon) \n    pickup_ewr = sphere_dist(pickup_lat, pickup_lon, ewr_coord[0], ewr_coord[1])\n    dropoff_ewr = sphere_dist(ewr_coord[0], ewr_coord[1], dropoff_lat, dropoff_lon) \n    pickup_lga = sphere_dist(pickup_lat, pickup_lon, lga_coord[0], lga_coord[1]) \n    dropoff_lga = sphere_dist(lga_coord[0], lga_coord[1], dropoff_lat, dropoff_lon) \n    \n    dataset['jfk_dist'] = pd.concat([pickup_jfk, dropoff_jfk], axis=1).min(axis=1)\n    dataset['ewr_dist'] = pd.concat([pickup_ewr, dropoff_ewr], axis=1).min(axis=1)\n    dataset['lga_dist'] = pd.concat([pickup_lga, dropoff_lga], axis=1).min(axis=1)\n    \n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b65dd583c3ae27a7daa395839ddde5271abfda9d","collapsed":true},"cell_type":"code","source":"#Run the functions to add the features to the dataset\ntrain = add_airport_dist(train)\ntest = add_airport_dist(test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5055621fd66d6775c1613a59848b3466b698e805"},"cell_type":"markdown","source":"Check the data after addition of all the new variables and make sure train and test datasets match."},{"metadata":{"trusted":true,"_uuid":"ff9018e11374b447f7126bd17172c574c8e7975e","collapsed":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7184164b7d6b72e65f223e644ad07b7571ea6fd7","collapsed":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9400cdcf56453de9e9b7defdff1ad1ad2f94a87d"},"cell_type":"markdown","source":"You can plot a correlation between all the features using a heatmap. The further away the values from 0, the more of an impact they play on the final fare prediciton."},{"metadata":{"trusted":true,"_uuid":"68c2abd8c2db88aace531c80ba58521e2961a4a4","collapsed":true},"cell_type":"code","source":"#Plot heatmap of value correlations\nplt.figure(figsize=(15,8))\nsns.heatmap(train.drop(['key','pickup_datetime'],axis=1).corr(),annot=True,fmt='.4f')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f7a9521c1fd9effa7d733dc82f80daa43be4ee51"},"cell_type":"markdown","source":"# Model Training"},{"metadata":{"_uuid":"f9ca276b45880efb0c35c44b27d06eaaf5b58d44"},"cell_type":"markdown","source":"As the coordinates columns seem to directly correlate with the fare_amount, I decided to leave them in the model fitting and prediction, along with all the newly generated features. Now we need to drop non-predicting columns and split the data into train and test sets for training the model. "},{"metadata":{"trusted":true,"_uuid":"bbfdd80f2d3c656d7758e4d725e62143374d28c9","collapsed":true},"cell_type":"code","source":"X = train.drop(['key','fare_amount','pickup_datetime'],axis=1)\ny = train['fare_amount']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"63e6f9826087523d57bd4544f23a1cf81898c533","collapsed":true},"cell_type":"code","source":"X.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f0dabaffc1c5bd8a92474a2b18b92b922d2c9d18","collapsed":true},"cell_type":"code","source":"y.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e448a8e930bc758728d3214e76d7acff4dcdcf4a","collapsed":true},"cell_type":"code","source":"#Split train set into test and train subsets\nX_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"7d451a0683d4a58a532946cc72fdb71b2d24813e"},"cell_type":"code","source":"#Drop columns from test dataset we're not going to use\ntest_pred = test.drop(['key','pickup_datetime'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b9549c65a163f38b66ab0b914fbc3d181014d9da","collapsed":true},"cell_type":"code","source":"#Scale values if needed for a particular model\n#scaler = RobustScaler()\n#X_train_scaled = scaler.fit_transform(X_train)\n#X_test_scaled = scaler.fit_transform(X_test)\n#X_scaled = scaler.fit_transform(X)\n#test_scaled = scaler.fit_transform(test_pred)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a5ee73e95ed384c9355bd313cd44db51882cb19"},"cell_type":"markdown","source":"## Linear Regression"},{"metadata":{"_uuid":"ca1a56daba9842c2613252bfdb81240363c7925a"},"cell_type":"markdown","source":"Let's start with a simple linear regression model and see how well it does for predicting the fare_amount. "},{"metadata":{"trusted":true,"_uuid":"eca71a4dbd0dcd72f9bd743af7b0ba2c87ccc435","collapsed":true},"cell_type":"code","source":"#Initilise a linear regression model, fit the data and get scores\nlm = LinearRegression()\nlm.fit(X_train,y_train)\nprint(lm.score(X_train,y_train))\nprint(lm.score(X_test,y_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"44b97a9085e9634f985d8fc81f950a2676bfb17e","collapsed":true},"cell_type":"code","source":"#Predict fares and get a rmse for them\ny_pred = lm.predict(X)\nlrmse = np.sqrt(metrics.mean_squared_error(y_pred, y))\nlrmse","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"967b1ebd5fabb53d71011256911d0c1cadbe7ae2","collapsed":true},"cell_type":"code","source":"#Predict final fares for submission\nLinearPredictions = lm.predict(test_pred)\nLinearPredictions = np.round(LinearPredictions, decimals=2)\nLinearPredictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7bfcb6a630b67bc0663ee323f2e39c6d5e679c31","collapsed":true},"cell_type":"code","source":"#Check predictions have the correct dimensions\nLinearPredictions.size","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d08f009fad913525c3b1816ab278cecd438e651e","collapsed":true},"cell_type":"code","source":"#Set up predictions for a submittable dataframe\nlinear_submission = pd.DataFrame({\"key\": test['key'],\"fare_amount\": LinearPredictions},columns = ['key','fare_amount'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a62507a82b6c9111fd469a2b387adb10cc995d0e","collapsed":true},"cell_type":"code","source":"#Check the submissions look reasonable\nlinear_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e5331d775090e908f3d299d2dc2e9b3b7f9b19e1"},"cell_type":"markdown","source":"## Gradient Boosting"},{"metadata":{"_uuid":"f564addcfd090565bb1a8fdd9c00b06d713edf60"},"cell_type":"markdown","source":"Linear regression seemed to do reasonably well, but you can get much more accurate predictions using a more finely tuned model such as gradient boosting optimization. First we define the model and some basic parameters for it (feel free to play around to see if you can get better results). "},{"metadata":{"trusted":true,"_uuid":"c741b4c8be0d9c90bf878cb039a1c310865cff7e","collapsed":true},"cell_type":"code","source":"#Define a XGB model and parameters\ndef XGBoost(X_train,X_test,y_train,y_test):\n    dtrain = xgb.DMatrix(X_train,label=y_train)\n    dtest = xgb.DMatrix(X_test,label=y_test)\n\n    return xgb.train(params={'objective':'reg:linear','eval_metric':'rmse'}\n                    ,dtrain=dtrain,num_boost_round=400, \n                    early_stopping_rounds=30,evals=[(dtest,'test')])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0bc0f9df843d774ba1d9310912bd748d25a72fc4","scrolled":false,"collapsed":true},"cell_type":"code","source":"#Fit data and optimise the model, generate predictions\nxgbm = XGBoost(X_train,X_test,y_train,y_test)\nXGBPredictions = xgbm.predict(xgb.DMatrix(test_pred), ntree_limit = xgbm.best_ntree_limit)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5465e82627d9672ee0512d5795118938ca605d6","collapsed":true},"cell_type":"code","source":"#Check if predictions seem to be realistic looking\nXGBPredictions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1da3d4151ce24a5bc9b494bb114b8cb0a1a2bf66","collapsed":true},"cell_type":"code","source":"#Round predictions to 2 decimal numbers\nXGBPredictions = np.round(XGBPredictions, decimals=2)\nXGBPredictions","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82e8c46b5864d58cfb59ce2f58546793c385a84b"},"cell_type":"markdown","source":"# Submission"},{"metadata":{"_uuid":"0ac3b63fddaadaa8359f4e12292323e8718e272e"},"cell_type":"markdown","source":"Now seeing that the XGB model has done much better than linear regression, let's prepare the data for submission by putting it in the correct format and match each prediction fare with the corresponding key. After which generate a csv file which will be ready to be uploaded for final submission."},{"metadata":{"trusted":true,"_uuid":"9e12ee8440a19942b2de401d2db6b1a33f8d8a0d","collapsed":true},"cell_type":"code","source":"#Prepare predictions for submission\nXGB_submission = pd.DataFrame({\"key\": test['key'],\"fare_amount\": XGBPredictions},columns = ['key','fare_amount'])\nXGB_submission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"adca2aceb5db7c49bd113f03c57a9fc13ec8eb5e","collapsed":true},"cell_type":"code","source":"#submission = linear_submission\nsubmission = XGB_submission","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c7997718e0db0972b5d2a196c2b25e878e2b9a4","collapsed":true},"cell_type":"code","source":"#Generate the final submission csv file\nsubmission.to_csv('XGBSubmission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"233b44ff58b8ee4eb3f1aa87d81e70b954231033"},"cell_type":"markdown","source":"That is it for this quick guide on some basic data exploration, analysis and prediction. Feel free to ask me any questions or comment any suggestions/corrections if you have any. I hope you've found this useful."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}