Exercise-1-problem-3-4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "# Problems 3 and 4\n",
    "\n",
    "A very common task handled by GIS is to convert a list of coordinates into geographic objects. For instance, you might have a table with latitude and longitude values, and want to use the locations in a map.\n",
    "Python is an excellent tool for this kind of a task: it can read data from (almost) any input format (CSV, text, Excel, GPX, various databases).\n",
    "\n",
    "In this exercise, we concentrate on reading data using [pandas](https://pandas.pydata.org/), and on creating geometry objects using [shapely](https://shapely.readthedocs.io/). \n",
    "Later on in the course, we will get to know other packages that are better tailored to geographic data, and will also learn how to write data to files, including to GIS file formats.\n",
    "\n",
    "## Sample data set\n",
    "\n",
    "For this exercise, we read the data from a file that lists travel times between different locations in Helsinki.\n",
    "The data is stored in a semicolon-separated text file, which you can find in the `data` folder of this repository, its file name is [`travel_times_2015_helsinki.txt`](data/travel_times_2015_helsinki.txt).\n",
    "\n",
    "The first four rows of our data look like this:\n",
    "\n",
    "```\n",
    "from_id;to_id;fromid_toid;route_number;at;from_x;from_y;to_x;to_y;total_route_time;route_time;route_distance\n",
    "5861326;5785640;5861326_5785640;1;08:10;24.9704379;60.3119173;24.8560344;60.399940599999994;125.0;99.0;22917.6\n",
    "5861326;5785641;5861326_5785641;1;08:10;24.9704379;60.3119173;24.8605682;60.4000135;123.0;102.0;23123.5\n",
    "5861326;5785642;5861326_5785642;1;08:10;24.9704379;60.3119173;24.865102;60.4000863;125.0;103.0;23241.3\n",
    "```\n",
    "\n",
    "In this exercise, we are interested in the following columns:\n",
    "\n",
    "| Column name        | Description                                              |\n",
    "|:------------------ |:-------------------------------------------------------- |\n",
    "| `from_x`           | x-coordinate of the **origin** location (longitude)      |\n",
    "| `from_y`           | y-coordinate of the **origin** location (latitude)       |\n",
    "| `to_x`             | x-coordinate of the **destination** location (longitude) |\n",
    "| `to_y`             | y-coordinate of the **destination** location (latitude)  |\n",
    "| `total_route_time` | Travel time with public transportation at the route      |\n",
    "\n",
    "Read more about this data set on the blog of the Digital Geography Lab: https://blogs.helsinki.fi/accessibility/helsinki-region-travel-time-matrix/."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "## Problem 3: Reading coordinates from a text file, and creating geometries (*5 points*)\n",
    "\n",
    "In this problem, your task is to read data from the file described above, and create two lists of points representing \n",
    "the origins and destinations of the routes described in the data set.\n",
    "\n",
    "This task entails multiple steps:\n",
    "\n",
    "1. Read the data into a `pandas.DataFrame`\n",
    "2. Discard all unnecessary columns (this is good practice, as it helps reduce the memory footprint of a program)\n",
    "3. Create two lists of `shapely.geometry.Point`s\n",
    "\n",
    "Let’s go step-by-step. \n",
    "\n",
    "Remember that there are code cells that you can and should modify (they initially contain only a comment `# ADD YOUR OWN CODE HERE`),\n",
    "and other code cells that you can and should run (but cannot modify) to test whether your code fulfils the requirements.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": [],
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "#### (1)\n",
    "\n",
    "First, use `pandas` to read the file into a variable `data`. You can revisit [lesson 5 of the Geo-Python course](https://geo-python-site.readthedocs.io/en/latest/notebooks/L5/exploring-data-using-pandas.html#reading-a-data-file-with-pandas) and consult the [pandas documentation](https://pandas.pydata.org/docs/user_guide/) to find the best way to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADD YOUR OWN CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "As a little sanity check, print the number of rows and columns of the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# ADD YOU OWN CODE HERE\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "If you loaded the data set successfully, the following code cell will print the first few rows of the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": [],
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "#### (2)\n",
    "\n",
    "Now, select the 4 columns that contain coordinate information (**`from_x`**, **`from_y`**, **`to_x`**, **`to_y`**), and store them in a DataFrame **`data`**. \n",
    "(i.e. update the variable `data`  to contain only these four columns).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADD YOUR OWN CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "Run the following code cell to test whether you have successfully replaced `data` with only the required data columns: it prints an error if you haven’t."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "assert list(data.columns) == [\"from_x\", \"from_y\", \"to_x\", \"to_y\"], \"Error: `data` does not (or not only) contain the four columns it should\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": [],
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "#### (3)\n",
    "\n",
    "Finally, create two lists called **`origin_points`** and **`destination_points`** that contain `shapely.geometry.Point` objects created using the coordinates from `data`. \n",
    "\n",
    "In particular, the origin points in `origin_points` should be based on columns `from_x` and `from_y`, and the destination points in `destination_points` on columns `to_x` and `to_y`.\n",
    "\n",
    "There are many ways to achieve this, find two possible approaches below (you can implement either one of them):\n",
    "\n",
    "##### **Approach A**\n",
    "\n",
    "- Create two empty lists for the origin and destination points, respectively\n",
    "- Use a for-loop to iterate over the rows of your dataframe:\n",
    "    - For each row, create a `shapely.geometry.Point` object based on the coordinate columns\n",
    "    - Append the point object to the `origin_points` and `destination_point` lists\n",
    "\n",
    "You can consult [lesson 6 of Geo-Python](https://geo-python-site.readthedocs.io/en/latest/notebooks/L6/advanced-data-processing-with-pandas.html#iterating-over-rows) to revisit how to loop over the rows of a `pandas.DataFrame`.\n",
    "\n",
    "##### **Approach B (more advanced)**\n",
    "\n",
    "- Make use of the `.apply()` function of the `pandas.DataFrame` to operate on all rows at once (see its [documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html), *HINT:* you might want to use the `axis` parameter)\n",
    "- Use the `shapely.geometry.Point` constructor directly, or wrap it into a [lambda function](https://towardsdatascience.com/apply-and-lambda-usage-in-pandas-b13a1ea037f7)\n",
    "- Finally, convert the output `pandas.Series` into `list`s\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADD YOUR OWN CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "**NOTE: After you have solved this problem, there might be some left-over variables around.<br />We recommend you *restart the kernel and run all cells* from the toolbar or JupyterLab’s menu.***\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "\n",
    "Use the following code cell to test whether your solution works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "\n",
    "# This test print should print out the first origin and destination coordinates in the two lists:\n",
    "print(\"ORIGIN X Y:\", origin_points[0].x, origin_points[0].y)\n",
    "print(\"DESTINATION X Y:\", destination_points[0].x, destination_points[0].y)\n",
    "\n",
    "# Check that you created a correct amount of points:\n",
    "assert len(origin_points) == len(data), \"Number of origin points must be the same as number of rows in the original file\"\n",
    "assert len(destination_points) == len(data), \"Number of destination points must be the same as number of rows in the original file\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "Remember to commit your code using git after each major change (for example, after solving each problem).\n",
    "\n",
    "### Done!\n",
    "\n",
    "That’s it. Now you are ready to continue to problem 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "## Problem 4: Creating LineStrings that represent the movements (*5 points*):\n",
    "\n",
    "This problem continues where we left off after completing *Problem 3*. \n",
    "\n",
    "The task is to:\n",
    "\n",
    "1. create a list lines (`shapely.geometry.LineString`) between each pair of origin and destination points, and \n",
    "2. calculate the over-all total_length of all those lines.\n",
    "\n",
    "Store the list of lines in a variable called `lines`, and the sum of lengths in a variable called `total_length`.\n",
    "\n",
    "Once you have working solutions for both tasks, \n",
    "\n",
    "3. create functions for them so you can apply them to other similar data sets in the future (see instructions below)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": [],
    "deletable": false,
    "editable": false
   },
   "source": [
    "#### (1)\n",
    "\n",
    "To create the `shapely.geometry.LineString`s for each pair of origins and destinations, you need to loop over both lists at the same time.\n",
    "\n",
    "Again, there are many ways to achieve this, here are two suggestions (implement either one):\n",
    "\n",
    "- (alternative 1) Use the `zip()` function that allows you to iterate over multiple lists at the same time. See this week’s [exercise hints](https://autogis-site.readthedocs.io/en/latest/lessons/L1/exercise-1.html#hints).\n",
    "- (alternative 2) Use the [*for-range* pattern from lesson 3 of Geo-Python](https://geo-python-site.readthedocs.io/en/latest/notebooks/L3/for-loops.html#looping-over-the-length-of-lists-using-index-values) and an index variable to access the same value in both lists\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADD YOUR OWN CODE HERE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "**NOTE: After you have solved this problem, there might be some left-over variables around.<br />We recommend you *restart the kernel and run all cells* from the toolbar or JupyterLab’s menu.***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "\n",
    "# Test that the list has correct number of LineStrings\n",
    "assert len(lines) == len(data), \"There should be as many lines as there are rows in the original data\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": [],
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "#### (2)\n",
    "\n",
    "Create a variable called **`total_length`**, and store the total (Euclidian) distance of all the origin-destination LineStrings that we just created into that variable.\n",
    "\n",
    "*Hint*: A simple solution is to start with a `total_length` of `0`, and add each line’s length while iterating over the list of lines.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADD YOUR OWN CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "\n",
    "# This test print should print the total length of all lines\n",
    "print(\"Total length of all lines is\", round(total_length, 2))\n",
    "assert round(total_length, 2) == 3148.57"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": [],
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "#### (3)\n",
    "\n",
    "Now, create functions that automate the functionality you implemented for part (1) and part (2) of this problem:\n",
    "\n",
    "- `create_od_lines()`: accepts two `list`s of `shapely.geometry.Point`s and returns a `list` of `shapely.geometry.LineString`s \n",
    "- `calculate_total_distance()`: takes a `list` of `shapely.geometry.LineString` geometries and returns their total length\n",
    "\n",
    "You can copy and paste the codes you have written earlier into the functions. Be sure to add a **docstring** to each function.\n",
    "Below, you can find a code cell for testing your functions (you should get the same result as earler)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ADD YOUR OWN CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# NON-EDITABLE CODE CELL FOR TESTING YOUR SOLUTION\n",
    "\n",
    "# Create origin-destination lines\n",
    "od_lines = create_od_lines(origin_points, destination_points)\n",
    "\n",
    "# Calculate the total distance\n",
    "tot_dist = calculate_total_distance(od_lines)\n",
    "\n",
    "print(\"Total distance\", round(tot_dist,2))\n",
    "assert tot_dist == total_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "\n",
    "----\n",
    "\n",
    "\n",
    "## Well done!\n",
    "\n",
    "Awesome, now you have successfully practiced how geometries can be created in Python. Next week we will start using them actively."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}