-
Notifications
You must be signed in to change notification settings - Fork 0
/
intro_to_pandas.qmd
106 lines (64 loc) · 3.58 KB
/
intro_to_pandas.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
---
filters:
- pyodide
---
# An Introduction to Pandas
{{< video https://youtu.be/YiecfDcq40o >}}
Let’s talk about Pandas.
And by that, we of course mean the beautiful, glorious, intelligent…
… open source Python library that provides powerful data structures and analysis tools.
(You can judge the level of geekiness of someone fairly easily by simply making the statement “I love Pandas” and seeing if they reply “Me too, and NumPy!”)
Pandas is very powerful for manipulating data in large arrays (unlike some spreadsheet software…), and allows for indexing of data. NumPy and Pandas are often used together, with NumPy used for mathematical functions applied to the data, and Pandas used to manipulate the data.
As with NumPy, there is a conventional alias under which Pandas should be imported :
```{python}
#| eval: true
import pandas as pd
```
## The Pandas Dataframe
One of the most useful structures in Pandas is the Pandas DataFrame. A DataFrame is like a table with different columns (which can have names) for different data fields, and different rows for each entry in the data.
Imagine a table in Excel. Only much more powerful.
## Creating a new dataframe
{{< video https://youtu.be/UWQtgQUPBY8 >}}
We have two choices when we want to create a new Pandas DataFrame :
- We can create a new DataFrame and then build the DataFrame manually
- We can read in existing data from a .csv file
Given that 99% of the time you’ll be doing the latter for real world applications, let’s just focus on that. And it’s also really easy!
Pandas has a fantastic function that allows us to read in data from a .csv file, and it will automatically stuff it into a new DataFrame for us!
In one line of code!!
Don’t believe me? Observe…
:::{.callout-tip}
One fantastic feature of pandas is that we can point to either a **local** file (one stored on our machine) or a **remote** file (one stored somewhere else - like the web).
:::
In the example below, we are accessing a file called `input_data.csv`.
If this file was stored in the same folder as our python file, we would just do
```{python}
#| eval: true
df = pd.read_csv("input_data.csv")
```
We can then show the first 5 rows of the data with the `.head()` method.
```{python}
#| eval: true
df.head()
```
However, we could also use a web location. We're going to use this method in the rest of the examples to allow us to interact with pandas dataframes in this book.
```{pyodide-python}
import pandas as pd
df = pd.read_csv("https://raw.githubusercontent.com/hsma-programme/h6_1f_python_part_2/main/1f_python_programming_part_2/lecture_examples/input_data.csv")
df.head()
```
You can see that the outputs are identical!
## The index
## Specifying an existing column as the index
In our data, we already have a unique identifier in our data - “Patient ID”. When we read in the .csv, we can tell Pandas that we want to set this column to be the Index (rather than creating a new one).
```{pyodide-python}
import pandas as pd
df = pd.read_csv(
"https://raw.githubusercontent.com/hsma-programme/h6_1f_python_part_2/main/1f_python_programming_part_2/lecture_examples/input_data.csv",
index_col="Patient ID"
)
df.head()
```
IMPORTANT - Pandas will check that the column you specify does truly have unique values. If it doesn’t, it will raise an exception (error) and if you don’t catch it, the code will terminate.
Be careful when importing data where multiple records refer to the same patient - patient ID would not be a unique identifier in that case.