001_Python_Pandas_Methods/003_Python_Pandas_DataFrame_head()_and_tail().ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<small><small><i>\n",
    "All the IPython Notebooks in this lecture series by Dr. Milan Parmar are available @ **[GitHub](https://github.com/milaan9/10_Python_Pandas_Module/tree/main/001_Python_Pandas_Methods)**\n",
    "</i></small></small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas DataFrame head, tail, at, iat\n",
    "\n",
    "In this class, we learn how to use **`DataFrame.head()`** and **`DataFrame.tail()`** functions to select top and bottom rows of the **[Pandas DataFrame](https://github.com/milaan9/10_Python_Pandas_Module/blob/main/001_Python_Pandas_DataFrame.ipynb)** respectively. Also, learn **`DataFrame.at()`** and **`DataFrame.iat()`** functions to access a specific value in the DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to use `DataFrame.head()` function\n",
    "\n",
    "This function is used to see the first n rows in the DataFrame. It is beneficial when we have massive datasets, and it is not possible to see the entire dataset at once.\n",
    "\n",
    "It takes input as the number of rows to be displayed from the top. The default value is 5.\n",
    "\n",
    "**Syntax:**\n",
    "```python\n",
    "DataFrame.head(n=5)\n",
    "```\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below Student DataFrame with columns like Name, Age, and Marks. If we apply **`DataFrame.head()`** function, we can see that only the first five rows are displayed.\n",
    "\n",
    "<div>\n",
    "<img src=\"img/dfhead.png\" width=\"200\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:50:53.184426Z",
     "start_time": "2021-06-17T11:50:52.603379Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Name  Age  Marks\n",
      "0    Joe   20  85.10\n",
      "1    Nat   21  77.80\n",
      "2  Harry   19  91.54\n",
      "3   Jack   17  72.00\n",
      "4   Jose   18  87.90\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry','Jack','Jose','Jill','Rose'],\n",
    "                'Age': [20, 21, 19,17,18,19,17],\n",
    "                'Marks': [85.10, 77.80, 91.54,72,87.9,90,72]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "\n",
    "# display first 5 rows\n",
    "topRows = student_df.head()\n",
    "print(topRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select top n rows in pandas DataFrame\n",
    "\n",
    "When we want to see a smaller section of data, we can use the function **`DataFrame.head()`** and pass a parameter as the number of rows to display from the top.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, after we apply the **`DataFrame.head(3)`** function, only the first three rows of the DataFrame are displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:50:55.942203Z",
     "start_time": "2021-06-17T11:50:55.922678Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Name  Age  Marks\n",
      "0    Joe   20  85.10\n",
      "1    Nat   21  77.80\n",
      "2  Harry   19  91.54\n"
     ]
    }
   ],
   "source": [
    "# display first 3 rows\n",
    "topRows = student_df.head(3)\n",
    "print(topRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select top rows except for last n rows\n",
    "\n",
    "When we have a vast DataFrame, and we want to see all the rows except for the last n rows, we can pass the negative value as a parameter to **`DataFrame.head()`**.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, if we want to display all the rows except the bottom two rows, we can use **`DataFrame.head(-2)`** function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:50:59.510526Z",
     "start_time": "2021-06-17T11:50:59.482722Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Name  Age  Marks\n",
      "0    Joe   20  85.10\n",
      "1    Nat   21  77.80\n",
      "2  Harry   19  91.54\n",
      "3   Jack   17  72.00\n",
      "4   Jose   18  87.90\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry','Jack','Jose',\"Jill\",\"Rose\"],\n",
    "                'Age': [20, 21, 19,17,18,19,17],\n",
    "                'Marks': [85.10, 77.80, 91.54,72,87.9,90,72]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "\n",
    "# display rows except bottom 2 rows\n",
    "topRows = student_df.head(-2)\n",
    "print(topRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select top rows from multi-index DataFrames\n",
    "\n",
    "When Python pandas DataFrame has multiple row index or column headers, then are called multi-level or hierarchical DataFrame. As we have discussed in the above section, we can use the **`DataFrame.head()`** function on multi-index DataFrames to display the top rows.\n",
    "\n",
    "The below diagram shows hierarchical DataFrame of Student data with two-column headers where column labels **`Name`** and **`Marks`** are at level 0 and **`Surname`** and **`Percentage`** at level 1. Similarly, two-row indexes are index ‘Standard‘ at level 0 and **`Class`** at level 1 of the DataFrame.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "The below example shows how to create such DataFrame and display top rows rather than the whole DataFrame.\n",
    "\n",
    "<div>\n",
    "<img src=\"img/midfhead.png\" width=\"350\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:01.441166Z",
     "start_time": "2021-06-17T11:51:01.410896Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                      Name      Marks\n",
      "                   Surname Percentage\n",
      "Standard   Class                     \n",
      "Standard 1 Class A     Joe      91.56\n",
      "           Class B     Nat      87.90\n",
      "Standard 2 Class A   Harry      70.10\n",
      "           Class B     Sam      65.48\n",
      "Standard 3 Class A    Jill      72.00\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "index = pd.MultiIndex.from_tuples([('Standard 1', 'Class A'), ('Standard 1', 'Class B'),\n",
    "                                   ('Standard 2', 'Class A'), ('Standard 2', 'Class B'),\n",
    "                                   ('Standard 3', 'Class A'), ('Standard 3', 'Class B')],\n",
    "                                  names=['Standard', 'Class'])\n",
    "\n",
    "columns = pd.MultiIndex.from_tuples([('Name', 'Surname'),\n",
    "('Marks', 'Percentage')])\n",
    "\n",
    "# create multi-index dataframe\n",
    "student_df = pd.DataFrame([('Joe', 91.56), ('Nat', 87.90),('Harry', 70.10), \n",
    "                           ('Sam', 65.48), (\"Jill\", 72), (\"Jane\", 80)],\n",
    "                          index=index, columns=columns)\n",
    "\n",
    "topRows = student_df.head()\n",
    "print(topRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to use `DataFrame.tail()` function\n",
    "\n",
    "We can use the **`DataFrame.tail()`** function to display the last n rows of the DataFrame. Like the head function, this function is used when we want to view a smaller section of the entire DataFrame.\n",
    "\n",
    "It takes input as the number of rows to be displayed from the bottom. The default value is 5.\n",
    "\n",
    "**Syntax:**\n",
    "```python\n",
    "DataFrame.tail(n=5)\n",
    "```\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below Student DataFrame with columns like **`Name`**, **`Age`**, and **`Marks`**. If we apply **`DataFrame.tail()`** function, we can see that only the bottom five rows are displayed in the output.\n",
    "\n",
    "<div>\n",
    "<img src=\"img/dftail.png\" width=\"200\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:04.075411Z",
     "start_time": "2021-06-17T11:51:04.050999Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Name  Age  Marks\n",
      "2  Harry   19  91.54\n",
      "3   Jack   17  72.00\n",
      "4   Jose   18  87.90\n",
      "5   Jill   19  90.00\n",
      "6   Rose   17  72.00\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry','Jack','Jose',\"Jill\",\"Rose\"],\n",
    "                'Age': [20, 21, 19,17,18,19,17],\n",
    "                'Marks': [85.10, 77.80, 91.54,72,87.9,90,72]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "\n",
    "# display the bottom 5 rows\n",
    "bottomRows = student_df.tail()\n",
    "print(bottomRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select bottom n rows in pandas DataFrame\n",
    "\n",
    "When we want to see a smaller section of data from the bottom of the DataFrame, we can use the function **`DataFrame.tail()`** and pass a parameter as the number of rows to display from the bottom.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, after we apply the **`DataFrame.tail(3)`** function, we see that only the last 3 rows of the DataFrame are displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:06.989926Z",
     "start_time": "2021-06-17T11:51:06.971376Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Name  Age  Marks\n",
      "4  Jose   18   87.9\n",
      "5  Jill   19   90.0\n",
      "6  Rose   17   72.0\n"
     ]
    }
   ],
   "source": [
    "bottomRows = student_df.tail(3)\n",
    "print(bottomRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select bottom rows except for first n rows\n",
    "\n",
    "When we want to see our entire dataset except for the first few rows, we can use **`DataFrame.tail()`** function and pass the negative value as a parameter to it.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, if we display all the rows except the top 2 rows using **`DataFrame.tail(2)`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:10.120263Z",
     "start_time": "2021-06-17T11:51:10.098782Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Name  Age  Marks\n",
      "2  Harry   19  91.54\n",
      "3   Jack   17  72.00\n",
      "4   Jose   18  87.90\n",
      "5   Jill   19  90.00\n",
      "6   Rose   17  72.00\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry','Jack','Jose',\"Jill\",\"Rose\"],\n",
    "                'Age': [20, 21, 19,17,18,19,17],\n",
    "                'Marks': [85.10, 77.80, 91.54,72,87.9,90,72]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "\n",
    "# display top rows except the last 2 rows\n",
    "bottomRows = student_df.tail(-2)\n",
    "print(bottomRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select bottom rows from the multi index DataFrame\n",
    "\n",
    "We can apply the **`DataFrame.tail()`** function on multi-index DataFrames as well. It works in the same way as normal DataFrames.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In our example, after we have applied the **`DataFrame.tail()`** function, only the bottom 5 row are displayed.\n",
    "\n",
    "<div>\n",
    "<img src=\"img/midftail.png\" width=\"350\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:14.265723Z",
     "start_time": "2021-06-17T11:51:14.244238Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                      Name      Marks\n",
      "                   Surname Percentage\n",
      "Standard   Class                     \n",
      "Standard 1 Class B     Nat      87.90\n",
      "Standard 2 Class A   Harry      70.10\n",
      "           Class B     Sam      65.48\n",
      "Standard 3 Class A    Jill      72.00\n",
      "           Class B    Jane      80.00\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "index = pd.MultiIndex.from_tuples([('Standard 1', 'Class A'), ('Standard 1', 'Class B'),\n",
    "                                   ('Standard 2', 'Class A'), ('Standard 2', 'Class B'),\n",
    "                                   ('Standard 3', 'Class A'), ('Standard 3', 'Class B')],\n",
    "                                  names=['Standard', 'Class'])\n",
    "\n",
    "columns = pd.MultiIndex.from_tuples([('Name', 'Surname'),\n",
    "('Marks', 'Percentage')])\n",
    "\n",
    "# create multi-index dataframe\n",
    "student_df = pd.DataFrame([('Joe', 91.56), ('Nat', 87.90), ('Harry', 70.10), \n",
    "                           ('Sam', 65.48), (\"Jill\", 72), (\"Jane\", 80)],\n",
    "                          index=index, columns=columns)\n",
    "\n",
    "bottomRows = student_df.tail()\n",
    "print(bottomRows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select value using row and column labels using `DataFrame.at`\n",
    "\n",
    "There are cases in the field of Data Science that we need to access a specific element of the DataFrame using its column label and row index. In such cases, we can use the **`DataFrame.at`** property and pass the row index and column labels of the value to access as parameters. This property can be used with Multi-index DataFrame as well.\n",
    "\n",
    ">**Note:** It throws **`KeyError`** if the label does not exist in DataFrame.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, after we use the **`DataFrame.at[2,\"Age\"]`** function, we get 19 as the output because 19 is the value present at row 2 and column Age.\n",
    "\n",
    "<div>\n",
    "<img src=\"img/dfat.png\" width=\"250\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:18.069877Z",
     "start_time": "2021-06-17T11:51:18.059136Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry',], 'Age': [20, 21, 19], \n",
    "                'Marks': [85.10, 77.80, 91.54]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "\n",
    "value = student_df.at[2,\"Age\"]\n",
    "print(value)  # --> Output: 19"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set specific value in pandas DataFrame\n",
    "\n",
    "When we want to update the value of the particular element from DataFrame based on its column label and row index, we can use DataFrame.at property.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, after we have applied **`DataFrame.at[2,\"Age\"]=50`** on our DataFrame, the value at that position changed from 19 to 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:21.789558Z",
     "start_time": "2021-06-17T11:51:21.779795Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19\n",
      "50\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry',], 'Age': [20, 21, 19],\n",
    "                'Marks': [85.10, 77.80, 91.54]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "print(student_df.at[2,\"Age\"])\n",
    "\n",
    "# change the value\n",
    "student_df.at[2,\"Age\"] = 50\n",
    "print(student_df.at[2,\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select value using row and column position using DataFrame.iat\n",
    "\n",
    "We want to access a specific element from a very large DataFrame, but we do not know its column label or row index. We can still access such an element using its column and row positions. For that, we can use **`DataFrame.iat`** property of python pandas. Unlike **`DataFrame.at`** it can work on the row and column index position of the DataFrame.\n",
    "\n",
    ">**Note:** Index positions starts at 0.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, we access the second row and the third column using **`DataFrame.iat[1,2]`**.\n",
    "\n",
    "<div>\n",
    "<img src=\"img/dfiat.png\" width=\"250\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:27.351991Z",
     "start_time": "2021-06-17T11:51:27.331487Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "77.8\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry',], 'Age': [20, 21, 19],\n",
    "                'Marks': [85.10, 77.80, 91.54]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "\n",
    "value = student_df.iat[1,2]\n",
    "print(value) # --> Output: 77.80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set specific value in pandas DataFrame\n",
    "\n",
    "When we want to update the value of the particular element from DataFrame based on its column and row position, we can use **`DataFrame.iat`** property.\n",
    "\n",
    "**Example:**\n",
    "\n",
    "In the below example, we change the value at the second row and the third column using **`DataFrame.iat[1,2]=90`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-17T11:51:31.000877Z",
     "start_time": "2021-06-17T11:51:30.988184Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "77.8\n",
      "90.0\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "student_dict = {'Name': ['Joe', 'Nat', 'Harry'], 'Age': [20, 21, 19], \n",
    "                'Marks': [85.10, 77.80, 91.54]}\n",
    "\n",
    "# create DataFrame from dict\n",
    "student_df = pd.DataFrame(student_dict)\n",
    "print(student_df.iat[1,2])\n",
    "\n",
    "# change value\n",
    "student_df.iat[1,2]=90\n",
    "print(student_df.iat[1,2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}