VenkataSowjanyaKoka-Autompg-LinearRegression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# For structured data we use Spark SQL, SparkSession acts a pipeline between data and sql statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sparksession is like a class and we need to create an instance of a class to utilize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName(\"AutoMPG_Linear_Regression\").getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading the CSV File as a dataframe\n",
    "# if infer schema is given false it expects to give schema\n",
    "# if header is given false it takes its own column names like c_0,c_1...\n",
    "AutoMPG_DF = spark.read.csv(\"/Users/sowjanyakoka/Desktop/Spring2020/MachineLearning/AutoMPG.csv\", inferSchema = True, header = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(1).What is the shape of the data contained in AutoMPG.csv?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape : (392, 9)\n"
     ]
    }
   ],
   "source": [
    "#(Answer):The shape of the data is the number of rows and columns (m = rows, n = columns) present in the dataset\n",
    "print(\"Shape :\",(AutoMPG_DF.count(),len(AutoMPG_DF.columns)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question(2).What features (or attributes) are recorded for each automobile?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The features recorded for each automobile are : ['mpg', 'cylinders', 'displacement', 'hp', 'weight', 'acceleration', 'model_year', 'origin', 'car_name']\n"
     ]
    }
   ],
   "source": [
    "#(Answer):The features recorded for each automobile can be known by the column names in the dataframe\n",
    "AutoMPG_Features = AutoMPG_DF.columns\n",
    "print(\"The features recorded for each automobile are :\", AutoMPG_Features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(3).Provide a schema of the AutoMPG data set to verify that all relevant features contain numeric data type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The schema of the AutoMPG data set is:\n",
      "root\n",
      " |-- mpg: double (nullable = true)\n",
      " |-- cylinders: integer (nullable = true)\n",
      " |-- displacement: double (nullable = true)\n",
      " |-- hp: double (nullable = true)\n",
      " |-- weight: double (nullable = true)\n",
      " |-- acceleration: double (nullable = true)\n",
      " |-- model_year: integer (nullable = true)\n",
      " |-- origin: integer (nullable = true)\n",
      " |-- car_name: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#(Answer):Schema is the outline of the dataframe which gives us an outline(column_name, datatype, possibility of null values) of each column in the dataset\n",
    "print(\"The schema of the AutoMPG data set is:\")\n",
    "#Displaying the schema of the dataset\n",
    "AutoMPG_DF.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The schema of the AutoMPG_LinearRegression_DF data set is:\n",
      "root\n",
      " |-- mpg: double (nullable = true)\n",
      " |-- cylinders: integer (nullable = true)\n",
      " |-- displacement: double (nullable = true)\n",
      " |-- hp: double (nullable = true)\n",
      " |-- weight: double (nullable = true)\n",
      " |-- acceleration: double (nullable = true)\n",
      " |-- model_year: integer (nullable = true)\n",
      " |-- origin: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## Question(3).Are there any columns/features that is not applicable in developing a Linear Regression algorithm?That is, does not meet the requirements/assumptions to use a Linear Regression model.\n",
    "# (Answer):Yes, the car_name column/feature is not applicable in developing a Linear Regression algorithm because it does not meet the following requirements of Linear Regression,\n",
    "# -We use numeric input variables to predict a numeric output variable.(car_name is a categorical variable)\n",
    "# -All data columns must be integer or float data type.(car_name is of string datatype) \n",
    "#===================================================================================================================\n",
    "# Question(3).If so, eliminate those columns from further analysis and regenerate the schema to ensure that the ‘offending’ column is removed from further analysis. Remember, it should not be permanently removed from the dataset.\n",
    "# (Answer):\n",
    "AutoMPG_LinearRegression_DF = AutoMPG_DF['mpg','cylinders','displacement','hp','weight','acceleration','model_year','origin']\n",
    "print(\"The schema of the AutoMPG_LinearRegression_DF data set is:\")\n",
    "#Displaying the schema of the dataset for further analysis\n",
    "AutoMPG_LinearRegression_DF.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(4).Evaluate the correlation between mpg and each of the independent variables (pairwise mpg and cylinders, mpg and displacement, etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "##(Answer):\n",
    "#Loading the libraries for correlation function\n",
    "from pyspark.sql.functions import corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation value between mpg and cylinders is:\n",
      "+--------------------+\n",
      "|corr(mpg, cylinders)|\n",
      "+--------------------+\n",
      "| -0.7776175081260227|\n",
      "+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##(Answer):4{1}-Performing correlation between mpg and cylinders\n",
    "print(\"The correlation value between mpg and cylinders is:\")\n",
    "AutoMPG_LinearRegression_DF.select(corr('mpg','cylinders')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation value between mpg and displacement is:\n",
      "+-----------------------+\n",
      "|corr(mpg, displacement)|\n",
      "+-----------------------+\n",
      "|    -0.8051269467104577|\n",
      "+-----------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##(Answer):4{2}-Performing correlation between mpg and displacement\n",
    "print(\"The correlation value between mpg and displacement is:\")\n",
    "AutoMPG_LinearRegression_DF.select(corr('mpg','displacement')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation value between mpg and hp is:\n",
      "+-------------------+\n",
      "|      corr(mpg, hp)|\n",
      "+-------------------+\n",
      "|-0.7784267838977761|\n",
      "+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##(Answer):4{3}-Performing correlation between mpg and hp\n",
    "print(\"The correlation value between mpg and hp is:\")\n",
    "AutoMPG_LinearRegression_DF.select(corr('mpg','hp')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation value between mpg and weight is:\n",
      "+------------------+\n",
      "| corr(mpg, weight)|\n",
      "+------------------+\n",
      "|-0.832244214831575|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##(Answer):4{4}-Performing correlation between mpg and weight\n",
    "print(\"The correlation value between mpg and weight is:\")\n",
    "AutoMPG_LinearRegression_DF.select(corr('mpg','weight')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation value between mpg and acceleration is:\n",
      "+-----------------------+\n",
      "|corr(mpg, acceleration)|\n",
      "+-----------------------+\n",
      "|    0.42332853690278693|\n",
      "+-----------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##(Answer):4{5}-Performing correlation between mpg and acceleration\n",
    "print(\"The correlation value between mpg and acceleration is:\")\n",
    "AutoMPG_LinearRegression_DF.select(corr('mpg','acceleration')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation value between mpg and model_year is:\n",
      "+---------------------+\n",
      "|corr(mpg, model_year)|\n",
      "+---------------------+\n",
      "|   0.5805409660907859|\n",
      "+---------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##(Answer):4{6}-Performing correlation between mpg and model_year\n",
    "print(\"The correlation value between mpg and model_year is:\")\n",
    "AutoMPG_LinearRegression_DF.select(corr('mpg','model_year')).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The correlation value between mpg and origin is:\n",
      "+------------------+\n",
      "| corr(mpg, origin)|\n",
      "+------------------+\n",
      "|0.5652087567164604|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##(Answer):4{7}-Performing correlation between mpg and origin\n",
    "print(\"The correlation value between mpg and origin is:\")\n",
    "AutoMPG_LinearRegression_DF.select(corr('mpg','origin')).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question(4).On the basis of individual correlation coefficients, can you determine which independent variables are useful in predicting mpg?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#Write response to your analysis as comments in your source code/notebook.\n",
    "#Remember, Correlation coefficient value ranges from -1 to +1; closer to 1, stronger the relationship.\n",
    "#====================================================================================================================\n",
    "##(Answer):\n",
    "#Based on individual correlation coefficients we can observe that MPG is,\n",
    "#-first it is highly correlated with weight(negative correlation):(-0.83)\n",
    "#-second it is highly correlated with displacement(negative correlation):(-0.80)\n",
    "#-third it is equally correlated with cylinders and hp(negative correlation):(-0.77)\n",
    "#-moderately correlated with model_year and origin(positive correlation):(0.58,0.56)\n",
    "#-lightly correlated with acceleration(positive correlation):(0.42)\n",
    "#===================================================================================\n",
    "##Question(4).Which independent variables are useful in predicting mpg?\n",
    "##(Answer): Independent variables that are highly correlated with mpg i.e; weight, displacement, cylinders and hp are useful in predicting mpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(5).Provide a listing of summary descriptive statistics such as average and standard deviation for each relevant attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Descriptive Statistics of the data set :\n",
      "+-------+-----------------+------------------+------------------+------------------+------------------+------------------+-----------------+------------------+\n",
      "|summary|mpg              |cylinders         |displacement      |hp                |weight            |acceleration      |model_year       |origin            |\n",
      "+-------+-----------------+------------------+------------------+------------------+------------------+------------------+-----------------+------------------+\n",
      "|count  |392              |392               |392               |392               |392               |392               |392              |392               |\n",
      "|mean   |23.44591836734694|5.471938775510204 |194.41198979591837|104.46938775510205|2977.5841836734694|15.541326530612228|75.9795918367347 |1.5765306122448979|\n",
      "|stddev |7.805007486571802|1.7057832474527845|104.64400390890465|38.49115993282846 |849.4025600429486 |2.75886411918808  |3.683736543577868|0.8055181834183057|\n",
      "|min    |9.0              |3                 |68.0              |46.0              |1613.0            |8.0               |70               |1                 |\n",
      "|max    |46.6             |8                 |455.0             |230.0             |5140.0            |24.8              |82               |3                 |\n",
      "+-------+-----------------+------------------+------------------+------------------+------------------+------------------+-----------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#===================================================================================================================================\n",
    "#(Answer):To see the descriptive statistics of the data set\n",
    "print(\"Descriptive Statistics of the data set :\")\n",
    "AutoMPG_LinearRegression_DF.describe().show(truncate = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(6).Collapse all independent variables/features into a single vector in preparation for Linear Regression analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#===========================================================================================================================\n",
    "#(Answer):#Feature Engineering\n",
    "#Loading vector libraries to combine all variables into one column to perform linear regression between mpg and independent variables\n",
    "from pyspark.ml.linalg import Vector\n",
    "from pyspark.ml.feature import VectorAssembler\n",
    "#Combining input columns(independent variables) as a vector and naming it as features\n",
    "vec_assemble = VectorAssembler(inputCols = ['cylinders','displacement','hp','weight','acceleration','model_year','origin'], outputCol  = 'features')\n",
    "#Taking data from dataset and combining \n",
    "Features_DF = vec_assemble.transform(AutoMPG_LinearRegression_DF)\n",
    "#Extracting the features column and output column for model generation\n",
    "Model_DF = Features_DF.select('features','mpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------------------------+----+\n",
      "|features                              |mpg |\n",
      "+--------------------------------------+----+\n",
      "|[8.0,307.0,130.0,3504.0,12.0,70.0,1.0]|18.0|\n",
      "|[8.0,350.0,165.0,3693.0,11.5,70.0,1.0]|15.0|\n",
      "|[8.0,318.0,150.0,3436.0,11.0,70.0,1.0]|18.0|\n",
      "|[8.0,304.0,150.0,3433.0,12.0,70.0,1.0]|16.0|\n",
      "|[8.0,302.0,140.0,3449.0,10.5,70.0,1.0]|17.0|\n",
      "|[8.0,429.0,198.0,4341.0,10.0,70.0,1.0]|15.0|\n",
      "|[8.0,454.0,220.0,4354.0,9.0,70.0,1.0] |14.0|\n",
      "|[8.0,440.0,215.0,4312.0,8.5,70.0,1.0] |14.0|\n",
      "|[8.0,455.0,225.0,4425.0,10.0,70.0,1.0]|14.0|\n",
      "|[8.0,390.0,190.0,3850.0,8.5,70.0,1.0] |15.0|\n",
      "|[8.0,383.0,170.0,3563.0,10.0,70.0,1.0]|15.0|\n",
      "|[8.0,340.0,160.0,3609.0,8.0,70.0,1.0] |14.0|\n",
      "|[8.0,400.0,150.0,3761.0,9.5,70.0,1.0] |15.0|\n",
      "|[8.0,455.0,225.0,3086.0,10.0,70.0,1.0]|14.0|\n",
      "|[4.0,113.0,95.0,2372.0,15.0,70.0,3.0] |24.0|\n",
      "|[6.0,198.0,95.0,2833.0,15.5,70.0,1.0] |22.0|\n",
      "|[6.0,199.0,97.0,2774.0,15.5,70.0,1.0] |18.0|\n",
      "|[6.0,200.0,85.0,2587.0,16.0,70.0,1.0] |21.0|\n",
      "|[4.0,97.0,88.0,2130.0,14.5,70.0,3.0]  |27.0|\n",
      "|[4.0,97.0,46.0,1835.0,20.5,70.0,2.0]  |26.0|\n",
      "+--------------------------------------+----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Model_DF.show(20,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape : (392, 2)\n"
     ]
    }
   ],
   "source": [
    "#Question(6).Verify that the shape is now for 2, instead of 8 features. \n",
    "#=======================================================================\n",
    "#(Answer):The shape of the data is the number of rows and columns (m = rows, n = columns) present in the dataset\n",
    "print(\"Shape :\",(Model_DF.count(),len(Model_DF.columns)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------------------------+----+\n",
      "|features                              |mpg |\n",
      "+--------------------------------------+----+\n",
      "|[8.0,307.0,130.0,3504.0,12.0,70.0,1.0]|18.0|\n",
      "|[8.0,350.0,165.0,3693.0,11.5,70.0,1.0]|15.0|\n",
      "|[8.0,318.0,150.0,3436.0,11.0,70.0,1.0]|18.0|\n",
      "|[8.0,304.0,150.0,3433.0,12.0,70.0,1.0]|16.0|\n",
      "|[8.0,302.0,140.0,3449.0,10.5,70.0,1.0]|17.0|\n",
      "+--------------------------------------+----+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "##Question(6).Print the first 5 rows of the new data.\n",
    "#====================================================\n",
    "#(Answer):\n",
    "Model_DF.show(5,False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(7).For training the regression model and its subsequent evaluation, \n",
    "# generate the training data and test data from your refined AutoMPG dataset, the dataset will be split 80/20 \n",
    "# so that 80% of the data will be used to train the model and the remaining 20% to evaluate the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#=================================================================================================================\n",
    "#(Answer):\n",
    "#Splitting the data for training and testing purpose\n",
    "Train_DF, Test_DF = Model_DF.randomSplit([0.80,0.20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question(7).Provide a shape for each dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Training Dataset : (308, 2)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#==============================================\n",
    "##(Answer):\n",
    "#Generating shape of training dataset\n",
    "print(\"Shape of Training Dataset :\" ,(Train_DF.count(), len(Train_DF.columns)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Testing Dataset : (84, 2)\n"
     ]
    }
   ],
   "source": [
    "##Question(7).Provide a shape for each dataset.\n",
    "#==============================================\n",
    "#(Answer):\n",
    "#Generating shape of testing dataset\n",
    "print(\"Shape of Testing Dataset :\" ,(Test_DF.count(), len(Test_DF.columns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(8).Using the training data, evaluate the correlation between mpg and each of the independent "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#Use the training data to fit a regression model to predict mpg given values for the number of cylinders, displacement, hp,  weight, acceleration, model_year and origin.\n",
    "#========================================================================================================================================================================\n",
    "#(Answer):\n",
    "#Using the ml(machine learning) import the required model using sub library regression\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "#Defining the model to understand which column is being dependent \n",
    "#Independent variables data is taken in the fit\n",
    "Linear_Model = LinearRegression(labelCol = 'mpg').fit(Train_DF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(9).For the trained model, what is the y-intercept value? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercept of the equation: \n",
      "-19.069011096140724\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#==================================================================\n",
    "#(Answer):\n",
    "#To see the intercept of the equation\n",
    "print(\"Intercept of the equation: \")\n",
    "print(Linear_Model.intercept)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficients of each independent variable:\n",
      "[-0.6413490814226427,0.0248207687353879,-0.008320242917813208,-0.006766677247601995,0.14927995925772125,0.7529701321319436,1.7077842394292988]\n"
     ]
    }
   ],
   "source": [
    "##Question(9).And what are the coefficients for each of the independent variables in the fitted model. \n",
    "#=====================================================================================================\n",
    "#(Answer):\n",
    "#To see the coefficients of each independent variable\n",
    "print(\"Coefficients of each independent variable:\")\n",
    "print(Linear_Model.coefficients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(10).For the trained model, print the Mean Sum of Squared Error and R-Square values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Mean Square Error values for Training Data is:\n",
      "10.9982702365616\n",
      "The R-Square value for Training Data is:\n",
      "0.812479772616278\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#============================================================================================\n",
    "#(Answer):\n",
    "#To see the predictions using the model for the input variables \n",
    "Training_Predictions = Linear_Model.evaluate(Train_DF)\n",
    "#Checking the Mean Square Error(Closer to zero the better)\n",
    "print(\"The Mean Square Error values for Training Data is:\")\n",
    "print(Training_Predictions.meanSquaredError)\n",
    "#Train R square value\n",
    "print(\"The R-Square value for Training Data is:\")\n",
    "print(Training_Predictions.r2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "##Question(10)What does this tell you about the usefulness of the fitted model to predict mpg? \n",
    "#Is the model any good? Remember, R-Square values range from 0 to 1.00; closer to 1.00 the better the predictive power of the model.\n",
    "#===================================================================================================================================\n",
    "#(Answer): In case of R2(coefficient of determination) it indicates a measure of how close the data are to the fitted regression line. \n",
    "#R-Square = Explained variation / Total variation\n",
    "#0 indicates that the model explains none of the variability of the response data around its mean.\n",
    "#1 indicates that the model explains all the variability of the response data around its mean.\n",
    "#So, an R-square value of 0.82 means that the fit explains 82% of the total variation in the data about the average.\n",
    "#Whereas (MSE) is the average of square of the errors. The larger the number the larger the error, closer to zero the better\n",
    "#We can say the model is good because,\n",
    "            #The lower the value of MSE the better and 0 means the model is perfect, our MSE is 10.918\n",
    "            #In case of r2 the closer the value is to 1 the high is the predictive power, our r2 is 0.816"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question(11).Now that you have trained your model, evaluate it using the test data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The R-Square value for Test Data is:\n",
      "0.8433537269890258\n",
      "The Mean Square Error values for Test Data is:\n",
      "10.614859405188232\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#Using the values of R-Square and Mean sum of Squared Error, what can you say about the reliability of the trained model to predict mpg with test data.\n",
    "#======================================================================================================================================================\n",
    "#(Answer): \n",
    "Test_Predictions = Linear_Model.evaluate(Test_DF)\n",
    "#Test R square values\n",
    "print(\"The R-Square value for Test Data is:\")\n",
    "print(Test_Predictions.r2)\n",
    "#Checking the Mean Square Error(Closer to zero the better)\n",
    "print(\"The Mean Square Error values for Test Data is:\")\n",
    "print(Test_Predictions.meanSquaredError)\n",
    "##This r2 percentage indicates that our Linear Regression model can predict\n",
    "#with more than 81% of accuracy in terms of predicting the mpg given the independent features of the Automobile. \n",
    "#The other 19% can be attributed toward errors that cannot be explained by the model. \n",
    "#Our Linear Regression line fits the model really well which indicates the model is good and reliable\n",
    "# We can say the model is quite reliable as the R2 value and Mean sum of Squared Error values are between 80-85% and 10-20 respectively \n",
    "#even after training the model with 70% train data and 30% test data for more than 10 times\n",
    "#and as we know lower the value of MSE and R2 value closer to 1, the better the model is and also reliable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container { width:100% !important; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}