Lesson4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Computational Methods**\n",
    "## **Introduction to Data Sets**\n",
    "\n",
    "Written by Niv Keren, nivkeren1@mail.tau.ac.il \n",
    "\n",
    "Based on [10 minutes to pandas](https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html#min),\n",
    "from pandas documentation.\n",
    "\n",
    "*Computational Methods* class: 0341-2300\n",
    "\n",
    "2020/Semester I; Tuesdays 14:00-16:00\n",
    "\n",
    "FACULTY OF EXACT SCIENCES | GEOPHYSICS & PLANETARY SCIENCES  \n",
    "Tel Aviv University\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Pandas**\n",
    "Pandas is a library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series.[(from wikipedia)](https://en.wikipedia.org/wiki/Pandas_(software)\n",
    "\n",
    "* DataFrame object for data manipulation with integrated indexing.\n",
    "* Tools for reading and writing data between in-memory data structures and different file formats.\n",
    "* Data alignment and integrated handling of missing data.\n",
    "* Reshaping and pivoting of data sets.\n",
    "* Label-based slicing, fancy indexing, and subsetting of large data sets.\n",
    "* Data structure column insertion and deletion.\n",
    "* Group by engine allowing split-apply-combine operations on data sets.\n",
    "* Data set merging and joining.\n",
    "* Hierarchical axis indexing to work with high-dimensional data in a lower-dimensional data structure.\n",
    "* Time series-functionality: Date range generation[4] and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging.\n",
    "* Provides data filtration.  \n",
    "[**Pandas Documantation**](https://pandas.pydata.org/pandas-docs/stable/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Seires**\n",
    "Creating a **Series** by passing a `list` of values, letting pandas create a default integer index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = pd.Series([1, 3, 5, np.nan, 6, 8])\n",
    "\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **DataFrame**\n",
    "Creating a **DataFrame** by passing a `NumPy array`, with a datetime index and labeled columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dates = pd.date_range('20130101', periods=6)\n",
    "\n",
    "dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame({'A': 1.,\n",
    "                    'B': pd.Timestamp('20130102'),\n",
    "                    'C': pd.Series(1, index=list(range(4)), dtype='float32'),\n",
    "                    'D': np.array([3] * 4, dtype='int32'),\n",
    "                    'E': pd.Categorical([\"test\", \"train\", \"test\", \"train\"]),\n",
    "                    'F': 'foo'})\n",
    "\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Viewing Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`DataFrame.to_numpy()`** gives a NumPy representation of the underlying data.  \n",
    "Note that this can be an expensive operation when your DataFrame has columns with different data types, which comes down to a fundamental difference between pandas and NumPy: **NumPy arrays have one dtype for the entire array, while pandas DataFrames have one dtype per column.** When you call `DataFrame.to_numpy()`, pandas will find the NumPy dtype that can hold *all* of the dtypes in the DataFrame. This may end up being object, which requires casting every value to a Python object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** `DataFrame.to_numpy()`does *not* include the index or column labels in the output.\n",
    "\n",
    "**`describe()`** shows a quick statistic summary of your data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transposing your data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Selection**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting a single column, which yields a Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['A']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting via `[]`, which slices the rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[0:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['20130102':'20130104']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selection by label**  \n",
    "For getting a cross section using a label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[dates[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting on a multi-axis by label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc['20130102':'20130104', ['A', 'B']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting a scalar value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[dates[0], 'A']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Selection by position**  \n",
    "Select via the position of the passed integers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By integer slices, acting similar to numpy/python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[3:5, 0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For slicing columns explicitly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[:, 1:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[1, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Boolean indexing**  \n",
    "Using a single column’s values to select data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.A > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selecting values from a DataFrame where a boolean condition is met."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the **`isin()`** method for filtering:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy = df.copy()\n",
    "df_copy['E'] = ['one', 'one', 'two', 'three', 'four', 'three']\n",
    "df_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy[df_copy['E'].isin(['two', 'four'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Setting**\n",
    "Setting a new column automatically aligns the data by the indexes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range('20130102', periods=6))\n",
    "s1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['F'] = s1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting values by label:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[dates[0], 'A'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting values by position:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0, 1] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting by assigning with a NumPy array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[:, 'D'] = np.array([5] * len(df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy = df.copy()\n",
    "df_copy[df_copy < 0] = -df_copy # abselute value\n",
    "df_copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Operations**\n",
    "### **Stats**\n",
    "Operations in general exclude missing data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.mean(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More statiscal methods are:  \n",
    "**`max()`**,  **`median()`**,  **`var()`**,  **`cov()`** ...\n",
    "### **Histogramming**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = pd.Series(np.random.randint(0, 5, size=10))\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Merge**\n",
    "### **Concat**\n",
    "Merges pandas objects along a particular axis with optional set logic along the other axes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(np.random.randn(3, 4))\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.DataFrame(np.random.randn(3, 4))\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "concat_df = pd.concat([df, df1])\n",
    "concat_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "concat_df = concat_df.reset_index()\n",
    "concat_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Join**\n",
    "Merges by index, possibly different columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left = pd.DataFrame({'lkey': ['foo', 'bar'], 'lval': [1, 2]})\n",
    "right = pd.DataFrame({'rkey': ['bar', 'foo'], 'rval': [4, 5]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left.join(right)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left = pd.DataFrame({'key': ['foo', 'bar'], 'lval': [1, 2]})\n",
    "right = pd.DataFrame({'key': ['bar', 'foo'], 'rval': [4, 5]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "left.merge(right, on='key')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## **Data Sets or Datasets**\n",
    "A collection of data.  \n",
    "Few examples:\n",
    "* **Tabular data -**  data set corresponds to one or more database tables, where every column of a table represents a particular variable, and each row corresponds to a given record of the data set in question.\n",
    "* **Images data -** collection of images. Can be aloso represented in a table\n",
    "* **Time-stamped data -**  Dataset which has a concept of time ordering defining the sequence that each data point was either captured (event time) or collected (processed time).\n",
    "* **Spatial Data -** Some objects have spatial attributes, such as positions or areas, as well as other types of attributes.\n",
    "\n",
    "### **Tabular Data**\n",
    "Two very common formats to save tabular data are:\n",
    "1. **CSV files -** *comma separated values*. Delimited text file that uses a comma to separate values. A CSV file stores tabular data (numbers and text) in plain text. Each line of the file is a data record. Each record consists of one or more fields, separated by commas.\n",
    "2. **JSON files -** *JavaScript Object Notation*. Is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### **CSV Example:**  \n",
    "Forest Fires in Brazil  \n",
    "Number of forest fires reported in Brazil by State [from kaggle](https://www.kaggle.com/gustavomodelli/forest-fires-in-brazil)\n",
    "\n",
    "\n",
    "\"year\",\"state\",\"month\",\"number\",\"date\"  \n",
    "1998,\"Acre\",\"Janeiro\",0,1998-01-01  \n",
    "1999,\"Acre\",\"Janeiro\",0,1999-01-01  \n",
    "2000,\"Acre\",\"Janeiro\",0,2000-01-01  \n",
    "2001,\"Acre\",\"Janeiro\",0,2001-01-01  \n",
    "2002,\"Acre\",\"Janeiro\",0,2002-01-01  \n",
    "2003,\"Acre\",\"Janeiro\",10,2003-01-01  \n",
    "2004,\"Acre\",\"Janeiro\",0,2004-01-01  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An easy way to read CSV files with pandas **`pd.read_csv()`** command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_amazon = pd.read_csv('files/amazon1.csv', encoding=\"windows-1252\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_amazon.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df_amazon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**JSON Example:**  \n",
    "Greenhouse gas emissions in the Netherlands  \n",
    "Emissions from economic activity according to IPCC guidelines [(from kaggle)](https://www.kaggle.com/janheindejong/greenhouse-gas-emissions-in-the-netherlands/data)\n",
    "\n",
    "[{ \"ID\": 0,  \n",
    "   \"Bronnen\": \"T001176  \",  \n",
    "   \"Perioden\": \"1990JJ00\",  \n",
    "   \"CO2_1\": \"163120\",  \n",
    "   \"CH4_2\": \" 1278.17\",  \n",
    "   \"N2O_3\": \"   59.49\"  \n",
    " },  \n",
    " { \"ID\": 1,  \n",
    "   \"Bronnen\": \"T001176  \",  \n",
    "   \"Perioden\": \"1995JJ00\",  \n",
    "   \"CO2_1\": \"  173520\",  \n",
    "   \"CH4_2\": \" 1192.41\",  \n",
    "   \"N2O_3\": \"   59.84\"  \n",
    " },  \n",
    " { \"ID\": 2,  \n",
    "   \"Bronnen\": \"T001176  \",  \n",
    "   \"Perioden\": \"2000JJ00\",  \n",
    "   \"CO2_1\": \"  172290\",  \n",
    "   \"CH4_2\": \"  975.64\",  \n",
    "   \"N2O_3\": \"   53.01\"  \n",
    " }  \n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "json_file_path = \"files/IPCC_emissions.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(json_file_path, 'r') as file:\n",
    "    IPCC_emmisions = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IPCC_emmisions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(IPCC_emmisions)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}