index.Rmd

---
author: "Jan Verbesselt, Jorge Mendes de Jesus, Aldo Bergsma, Johannes Eberenz, Dainius Masiliunas"
date: "`r format(Sys.time(), '%d %B, %Y')`"
output:
  knitrBootstrap::bootstrap_document:
    title: "Week 3: Python for geo-scripting"
    theme: "simplex"
    highlight: Tomorrow Night Bright
    menu: FALSE
    theme.chooser: TRUE
    highlight.chooser: TRUE
---

# [WUR Geoscripting](https://geoscripting-wur.github.io/) <img src="http://www.wur.nl/upload/b43b7095-e452-482a-8969-fed9a50393a8_WUR_RGB_standard.png" alt="WUR logo" style="height: 35px;"/>

# "Week 3: Python for geo-scripting"

Good morning! Today we will start working with Python for geo-scripting and do a refresher of functions in Python. First complete the Intro to Python course in Datacamp and then go through today's tutorial.

### Today's schedule

- Follow DataCamp Intro to Python course
- Learn how to work with virtual environments: Conda
- Learn how to create Jupyter Notebook
- Refresher for python

Using Python within Linux:

- Wide user community and support
- Free
- Flexiblility
- Open-source

How?! via:

- GDAL/OGR
- GEOS
- Rasterio
- Shapely
- Mapplotlib
- Folium
- Fiona
- GeoPandas
- ArcPy (requires ArcMap license)

Have a look at this question on GIS StackExchange:

- [Alternatives to using arcpy](https://gis.stackexchange.com/questions/34509/alternatives-to-using-arcpy)
- [Clipping raster with vector layer using gdal](https://gis.stackexchange.com/questions/16657/clipping-raster-with-vector-layer-using-gdal)

## Python editors and IDEs
* Most modern text editors do nice python highlighting, e.g. Sublime Text can be set up nicely for Python.
* Jupyter notebook or Ipython Notebook are good choices for short scripts. The notebooks allows you to have your source code, equations, visualizations, results and comments in one document! Jupyter Notebook is a continuation of Ipython Notebook. The name “Jupyter” was inspired by the leading open languages for science (that is, Julia, Python, and R) ([blog about history of Jupyter Notebook](https://www.datacamp.com/community/blog/ipython-jupyter)). See here for a simple Jupyter [notebook example](http://nbviewer.ipython.org/github/GeoScripting-WUR/PythonWeek/blob/gh-pages/A%20simple%20notebook.ipynb). [More info](http://jupyter.org/).
* There are a number of proper Integrated Development Environments [IDE] for Python. An IDE is a software application that provides facilities for software development. Personally I have good experience with [PyCharm](https://www.jetbrains.com/pycharm/). For running on a server, [rodeo](https://github.com/yhat/rodeo/) gives a similar interface as RStudio server.
* Spyder is nice a lightweight IDE and can be installed from the terminal (`sudo apt-get install spyder`).

## Python package management
For Python, a set of tools co-exist for installing and managing packages. From most to least desirable, you should try to install packages by:

* Using the distribution's package manager (on Ubuntu, that's `sudo apt-get install python-*`). This is the easiest and guaranteed to work right, although the package version might be older than you expect.
* Using [conda](https://conda.io/docs/user-guide/tasks/manage-pkgs.html) to install packages and to keep separate sets of packages. With Python, often the dependencies and versions can differ from project to project. If you want to specifally use a combination of packages with certain versions and keep that package set saved in an environment, then conda has conda environments to save this set of packages. A conda environment is similar to a virtual environment, but different from a virtual machine. Conda is available for Windows, macOS and Linux.
* Using `pip` to manage packages and `virtualenv` to manage your environment. *pip* is a python package installer, that is pretty much standard nowadays. It is recommended to run it in a virtualenv to prevent conflicts, however, virtualenvs can be problematic when it comes to package dependencies. `virtualenv` can be used to keep separate sets of packages in its own directory tree. A demonstration of `pip` and `virtualenv` can be found here [here](http://www.dabapps.com/blog/introduction-to-pip-and-virtualenv-python/)


### Anaconda or Miniconda
Anaconda is a Python virtual environment and package manager that is very useful to automatically keep track of package versions and dependencies. Anaconda itself comes already preinstalled with a large number of packages such as `numpy` and `gdal`. Instead of using the whole Anaconda, you can instead use the base version called Miniconda. Miniconda has the same features, but only installs the base package and requires manual installation for extra packages.

It is useful to learn how to use Conda, because it is an easy and cross-platform way of installing latest Python packages without affecting any Python installations that you already have on your system. In addition, Anaconda is required to set up `sen2cor`, the Sentinel-2 imagery atmospheric correction tool.

#### Miniconda installation
To install Miniconda in your Linux environment, we have prepared a short *Bash* script for you. Just run the following lines of code, line by line, in a new terminal window.

```{r, eval=FALSE, engine='bash'}
MINICONDA_VERSION="Miniconda3-latest-Linux-x86_64"
pushd /tmp
curl -O https://repo.continuum.io/miniconda/${MINICONDA_VERSION}.sh
## This installation script will require user input
bash ${MINICONDA_VERSION}.sh
rm ${MINICONDA_VERSION}.sh
popd
```

When prompted, you can just use the default options (i.e. press Enter). However, if you don't want Conda to replace the default Python interpreter in your system, you should say `No` in the last prompt. In that case, every time you wish to use Conda you need to run the following line of code in each terminal window.

```{r, eval=FALSE, engine='bash'}
# $HOME expands to your home directory
export PATH=$HOME/miniconda3/bin:$PATH
```

For installation instructions in other operating systems, please go to the [Miniconda installation page](https://conda.io/docs/user-guide/install/index.html).

#### Usage
The basic usage of Conda, after installed, is as follows.

To search for a package:

```{r, eval=FALSE, engine='bash'}
conda search spyder
```

This would give you a list of all packages that have "spyder" in the name and list all available versions.

```{r, eval=FALSE, engine='bash'}
conda install spyder
```

This would install the latest version of the `spyder` package (Python IDE). Note that this would install it into your user's root virtual environment (by default it is `$HOME/miniconda3`). Conda is able to create any number of isolated virtual environments, for example:

```{r, eval=FALSE, engine='bash'}
conda create --name geotest python=2.7 numpy
```

This would create a new environment called `geotest` with Python 2.7 and `numpy` installed into it. To list the available environments:

```{r, eval=FALSE, engine='bash'}
conda info --envs
```

Conda puts an asterisk (*) in front of the active environment. To activate an environment:

```{r, eval=FALSE, engine='bash'}
## Linux, macOS
source activate geotest
## Windows
activate geotest
```

After this, the current environment is shown in (parentheses) or [brackets] in front of your prompt (`(astrolab)$`). To deactivate the environment and go back to the default one:

```{r, eval=FALSE, engine='bash'}
## Linux, macOS
source deactivate
## Windows
deactivate
```

To remove the environment `geotest`:

```{r, eval=FALSE, engine='bash'}
conda remove --name geotest --all
```

Note that the activated environment is only valid for the shell in which you activated it. For instance, if you close the shell window and open a new one you will have to activate it again. Additionally, if you use `sudo` commands to call Python, it will use the system's Python interpreter and not the active environment for security reasons. There should be no reason to call Python code in Conda with `sudo` rights in any case, since all packages are installed with your user permissions rather than root's.

In addition, as you saw before, Conda is able to install some non-Python packages that have Python bindings, such as Spyder and GDAL. This is useful for making sure your Python and binary versions match and do not interfere with the system-wide ones. However, since those packages are installed into a virtual environment, they will not be accessible from your system menu. Instead, to run e.g. Spyder from within a Conda environment called `ide`, you would need to do something like this:

```{r, eval=FALSE, engine='bash'}
source activate ide
spyder
```

It is useful to check whether the executable comes from the system or a virtual environment by using the `which` or `type` commands:

```{r, eval=TRUE, engine='bash'}
type gdalinfo
```

This shows that you would be running GDAL from the system rather than the Conda virtual environment, otherwise the path would include `miniconda` and the virtual environment name.

Some helpful utilities are:

* `conda list` to check which packages are installed in `root` or in the active environment;
* `which python` or `python --version` to check which python verison is used in the environment;
* `conda install --name astrolab matplotlib` to install extra modules in your (running) conda environment.

## Getting started with Python within a Linux OS

- Launch a Linux virtual machine and login.

- Open the Terminal and type the following to check the installed GDAL version:

```{r, eval = TRUE, engine='bash'}
## from R: system("gdal-config --version")
## From the terminal:
python2 --version
python3 --version
gdal-config --version
```

- Type the following to start Python 2.7:

```{r, eval=FALSE, engine='bash'}
python3 # type this in the terminal to start the python interpreter
```

An example script to find out what the installed Python version is ([more info in question asked on stackoverflow](https://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-python-is-running-my-script))

```{r, engine='python', engine.path='python3', engine.path='python3'}
import sys
print(sys.version)
```

To exit Python in the terminal:
```{r, eval=FALSE, engine='python', engine.path='python3', engine.path='python3'}
exit()
# or 
quit()
```

## Running Python in Jupyter Notebooks

You can program Python in your terminal, but more facilities are available to make coding and documenting in Python easier through notebooks or Python IDEs. Today we will have a go with Jupyter Notebooks. In other lessons you can use the Python IDE Spyder. For now try the following commands in your terminal:

```{r, eval=FALSE, engine='bash'}
# Set directory at home
cd

# Create conda environment
conda create -n geoscripting numpy jupyter # geoscripting is name of your new conda environment

# Activate conda environment
source activate geoscripting
```

By creating your conda environment, you created a set of packages and a Python version to be used only in that specific conda environment. In your terminal you can see the name of your conda environment at the start of the command line before your user information. Now that we are working in our conda environment with all the necessary packages with correct versions, we can start with setting up our notebook.

```{r, eval=FALSE, engine='bash'}
# Start a Jupyter Notebook from the terminal
jupyter notebook
```

If everything goes according to plan, Jupyter will pop up in your browser. You will see a menu with all the files in your working directory. **Note**: the Jupyter notebook will only be able to see files that are accessible from the working directory in which from which you launch it! So keep track of the working directory in your terminal. A good practice is to start it in your project's directory.

Once you are in the desired working directory, in the right top click on *New* → *Folder* (if you don't have folder/project structure yet) and/or click on *Python 3* to create a Jupyter Notebook. Click on help and have a go at the `User Interface Tour`. Give your notebook a name.

These are the basic functions you will need today:

- `Save and checkpoint`
- `Insert cell below`
- `Run`
- `Code/Markdown/Heading`

Similar to RMarkdown, Jupyter Notebooks has code cells (called *Code*) and text cells (called *Markdown*). Insert some extra cells by clicking the + button and change the first cell from code to markdown. Enter some documentation for your code (e.g. your team name, exercise and date). Leave the other cells on code. To run code in a code cell, select it and press the *Run* button.

## A short Python refresher

### Finding help
Now we can try some coding. First we learn how to look for help while coding in Python. In the second cell type the code below and run it (ctrl + enter is shortcut for run cell).


```{r, engine='python', engine.path='python3', eval=FALSE}
import sys
help(sys)
print("-------------------------------------")
help(1)
```

See how the functions in the `sys` module got listed and how we got information how to work with integers. Sometimes you also need to use the internet to find information.

> **Question** 1: What does this mean `__ __` around words: e.g: `__doc__`?

Try out the following!!!

```{r, engine='python', engine.path='python3'}
help('hamster')
```


See also: 

* [http://www.rafekettler.com/magicmethods.html](http://www.rafekettler.com/magicmethods.html)
* [https://stackoverflow.com/questions/1090620/special-magic-methods-in-python](https://stackoverflow.com/questions/1090620/special-magic-methods-in-python)

### Finding information via Pydoc

Type the script below in your terminal to start a HTTP server with information from pydoc or go to [https://docs.python.org/2/library/pydoc.html](https://docs.python.org/2/library/pydoc.html).

```{r, engine='bash', eval=FALSE}
pydoc -p 1234
echo "pydoc server ready at http://localhost:1234/"
```

Then go to `http://localhost:1234/` via your preferred browser. You can see a list of built-in modules and available modules.

### Numbers and variables

We continue working in Python. 

> **Question 2**: What is the difference between 10 and 10.0 when dealing with data types in Python?

```{r, engine='python', engine.path='python3'}
print(int(10.6))
```

Variable is a storage location or symbolic name to a value e.g.:

```{r, engine='python', engine.path='python3'}
building = 'Gaia'
buildingNumber = 101
'Gaia'
"doesn't"
'Gaia' + ' is in Wageningen'
```

There is no need to say or define the datatype, `python` has a loose type variable declaration. 

*If it walks like a duck, swims like a duck and quacks like a duck I call it a duck*

Python is basically a list of objects.

```{r setup, include=FALSE}
library(knitr)
opts_chunk$set(engine = 'python', engine.path='python3')
```

### Lists
Now we will have a go with lists.

**Tip**: Variables, functions and methods that you define in one of your Jupyter Notebook cells can be used in other cells too.

Run this code in one cell:
```{r campus, engine='python', engine.path='python3'}
campus = ['Gaia', 'Lumen', 'Radix', 'Forum']
# how to can we print Forum?
print(campus[3])
# how to access the end of the list (while having no idea how big it is)
print(campus[-1])
# how to access the first 3 items
print(campus[0:3])
```

Run this code in another cell. We will do some appending, inserting, extending and steps:
```{r, engine='python', engine.path='python3', eval = FALSE}
campus.append("Atlas")
campus.insert(1,"SoilMuseum")
campus.extend(["Action", "Vitae", "Zodiac"])
print(campus)
print(campus[::2])
## list[start:end:step]
```

See how the notebook remembered how you set the variable `campus`. 

> **Question 3:** What are the major differences between Append/Extend?

> **Question 4:** What building is `campus[-2]`?

### Dictionaries, loops, if/else

Let there be dictionaries... A dictionary is an unordered set of key:value pairs. Like in the dictionary, 'food':'voedsel'.

```{r, engine='python', engine.path='python3'}
# dictionary
campusDic = {101:'Gaia',
             100:'Lumen',
             107:'Radix',
             102:'Forum',
             104:'Altas'}
print(campusDic[102])
```

Loops: watch out here with code indentation. Python uses indentation to define code blocks and not `{}` like other languages.
`print(building)` in the following code has to be *indented* by 1 tab or 4 spaces ( [recommended](https://stackoverflow.com/questions/119562/tabs-versus-spaces-in-python-programming) ).

```{r, engine='python', engine.path='python3'}
campus = ['Gaia','Lumen', 'Radix', 'Forum']
for building in campus:
    print(building)
```


Here, `building` is a variable that will contain any item in the campus list.

Generic loops in Python have to interact over a sequence of objects e.g.

```{r, engine='python', engine.path='python3'}
range(5)
for number in range(5):
    print(number)
```

Object interaction and functional programming is an important part of Python programming and its tools are extensive.
`if`/`else`:

```{r, engine='python', engine.path='python3'}
x = 3
if x < 3:
    print("below 3")
else:
    print("above 3")
```

```{r, engine='python', engine.path='python3'}
x = 3
if x == 1:
    print("it is one")
elif x==2:
    print("it is two")
elif x==3:
    print("it is three")
else: 
    print("above 3")
```

### Functions

A function is a section of code that does something specific that you want to use multiple times without having to type the full function again but just call the function by its name.

```{r, engine='python', engine.path='python3'}
def printPotato():
    print("potato")

printPotato()
```

Functions accept arguments and return variables e.g.:

```{r, engine='python', engine.path='python3'}
def printHelloName(name):
    print("Good morning " + name)

printHelloName("Jan")
```

`return` is used to indicate what you want to obtain from the function, you can `return` multiple items and return can be used to assign output to variables outside of the function.

```{r, engine='python', engine.path='python3'}
def times3(number):
    tmp = number*3
    return tmp, number
    
print(times3(4))

output, input = times3(4)
print(output)
print(input)
```

### Importing modules

Try this!

```{r, engine='python', engine.path='python3', eval=FALSE}
import this
```

This poem is called the Zen of Python and describes how Python should be used. It is an inside joke, but has some good practices to it. There are more best practice guides for Python [best of best practices guide in Python](https://gist.github.com/sloria/7001839).

```{r, engine='python', engine.path='python3', eval=FALSE}
from __future__ import braces
```

Another inside joke ... where your Python says that it will never delimit coding blocks by braces instead of indentation. Let's continue with more serious programming.

```{r, engine='python', engine.path='python3'}
import math
print(dir(math)) #show names in math module
```

- The best way to learn a module is by checking [documentation](https://docs.python.org/2/).
- Modules are Python's butter and bread. 
- A module contains code that can be used or excuted.

Basically there are three ways to load a module:
```{r, engine='python', engine.path='python3'}
import math
print(math.pi)

from math import pi
print(pi)

import numpy as np
print(np.pi)
```

> **Question 5**: Which is the best way to import modules?

### Some important internal modules:

- `os`: Access to operating system features
- `os.path`: Manipulating of file names
- `sys`: System specific configuration
- `glob`: Filename pattern matching
- `math`: Mathametical functions
- `datetime`: Date/Time manipulation

> **Question 6**: What is the difference between `os` and `os.path`?

Some examples:

```{r, eval=FALSE, engine='python', engine.path='python3'}
import glob
glob.glob("*")
```

```{r,engine='python', engine.path='python3'}
from datetime import timedelta, date
delta = timedelta(days=7)
print(date.today())
print(date.today()+delta)
```

### File access

File access is very simple for 99% of the cases.

Write something to file:

```{r, engine='python', engine.path='python3'}
fileObj = open('test.txt','w')
fileObj.write('some simple text')
fileObj.close()
```

And read something from a file:

```{r, engine='python', engine.path='python3'}
fileObj = open('test.txt','r')
a = fileObj.read()
print(a)
fileObj.close()
```

> **Question 7**: What does `w` and `r` mean?

### Error handling

Sometime problems occur... Errors detected during execution are called *exceptions*.

Good code deals with exceptions:

```{r, error=TRUE, engine='python', engine.path='python3'}
open("/foo0")
```

The file doesn't exist, so the script stops and outputs an ugly message.

How to deal with this I/O error? Good programming!

```{r, engine='python', engine.path='python3'}

try:
    open("foo")
except IOError:
    print("no file")

## we can be more precise:
try:
    open("/foo")
except IOError:
    print("no file")
```

### Visualization

Jupyter Notebooks can display output of your code, such as graphs, images and maps in the notebook. A lot of cool visualizations including code in Python are available from the [Python Graph Gallery](https://python-graph-gallery.com/). Before we can do the visualizations, we want to add some Python modules via the terminal.

```{r, engine='bash', eval=FALSE}
## Add Matplotlib and seaborn modules
conda install --name geoscripting matplotlib seaborn
# as conda install --name env_name pythonmodule pythonmodule
```

We will make a graph with the `Seaborn` module and plot it with `Matplotlib`. Give it a try! (If you have just installed it, you might need to close the Jupyter notebook and reopen it.)

```{r, engine='python', engine.path='python3', eval=FALSE}
# Load library
import matplotlib.pyplot as plt
import seaborn as sns

# Load dataset
df = sns.load_dataset('iris')
 
# Create plot
sns.pairplot(df, kind = "reg", hue = "species")
plt.show()
```

Nice visualization huh! Good job.

Now make a map with `Folium`. Note: Folium is very new and so is not in the main conda channel, so install it with `conda install folium -c conda-forge`.

```{r, engine='python', engine.path='python3', eval=FALSE}
import folium

SF_COORDINATES = (51.9871868, 5.6593948)

map = folium.Map(location=SF_COORDINATES, tiles='Mapbox Control Room', zoom_start=5)
display(map)
```

### Exiting the Jupyter Notebook
Your Jupyter Notebook is automatically saved as an .ipynb file (extension comes from the historic name "IPython Notebook") on your computer, but you can also download it as a python script, pdf or html. You can also save it manually. To exit a notebook properly, use *File* → *Close and Halt*.

By pressing Ctrl + c in the terminal where Jupyter notebook server is running, you cancel the running process. The terminal goes back to command line and you can exit the virtual environment by typing `source deactivate`.

```{r, engine='bash', eval=FALSE}
source deactivate
```

### Python statements

- `if`
- `for`
- `while`
- `try`
- `type` shows type (e.g. int, float, str) of object
- `class` which executes a block of code and attaches its local names to a `class`, for use in object oriented programming
- `def` which defines a function or statement
- `with` which encloses a code block within a context manager
- `pass` statement, which serves as a `NOP` (no operation)
- `assert`, used during debugging to check for conditions that ought to apply
- `yield`
- `import`

## Python from R

By the way, as you may have noticed, this document is created from an RMarkdown source, which can also include Python code blocks (have a look at the [source](https://github.com/GeoScripting-WUR/PythonWeek/blob/gh-pages/index.Rmd) on github). Similarly Jupyter Notebooks can also use R and other languages!


# Assignment
The assignment for today is to finish the datacamp course: Intro to Python for Data Science. If you finished early and still want to write more scripts, then you can follow one of the fun tutorials below.

# More info
- A great book: [Python Geospatial Development](http://www.amazon.com/Python-Geospatial-Development-Second-Edition/dp/178216152X)
- [The official python tutorial](https://docs.python.org/2/tutorial/)
- [Python Tutorial](http://www.tutorialspoint.com/python/)
- [Best of the best practices in Python - Style guide ](https://gist.github.com/sloria/7001839)
- [Stack OverFlow Python](https://stackoverflow.com/questions/tagged/python)
- [GIS Stack Exchange](https://gis.stackexchange.com/)
- [QGIS can be scripted in python](http://www.qgistutorials.com/en/docs/getting_started_with_pyqgis.html)
- [A simple Iphython Notebook](http://nbviewer.ipython.org/github/GeoScripting-WUR/PythonWeek/blob/gh-pages/A%20simple%20notebook.ipynb)
- [Buffer Example](http://nbviewer.ipython.org/github/GeoScripting-WUR/PythonWeek/blob/gh-pages/Buffer%20example.ipynb)
- [Spatial visualization with Matplotlib](https://github.com/GeoScripting-WUR/PythonWeek/blob/gh-pages/Matplotlib_basemap.ipynb)
- [Spatial visualization with Cartopy](https://github.com/GeoScripting-WUR/PythonWeek/blob/gh-pages/Plot%20Map.ipynb)
- [Reading and formatting geometries with OGR](https://github.com/GeoScripting-WUR/PythonWeek/blob/gh-pages/OGR%20examples.ipynb)