Episode 08 (Putting it all together): complete rewrite by stijnvanhoey · Pull Request #299 · datacarpentry/python-ecology-lesson

stijnvanhoey · 2018-06-18T22:38:56Z

This PR aims to change/improve the matplotlib section of the Data Ingest and Visualization - Matplotlib and Pandas lesson, providing the following changes:

focus on the link between matplotlib, plotnine(ggplot) and Pandas instrad of learning this new library
add a section and challenge about saving plots, as requested in Add instructions for saving figures to Matplotlib lesson #293
it overcomes the issue of opening and closing plots, see from @crocodoyle: Pyplot tutorial should close plots #277, by focusing on the object oriented approach of matplotlib

It indeed basically excludes the whole introduction of the stateful approach (cfr. Matlab style), but I do think this is an improvement, as:

the link with the other episodes is made in the new version. The main concept is: start working in Pandas/plotnine, but be able to further customize with matplotlib when required.
the object oriented way is more powerful for adjustments and using the TAB, people can really explore the structure of the object
it is a bit strange to start new concepts (creating plots from scratch in matplotlib) in a 'putting it together( episode.

Still, this PR is a suggestion and I'm certainly open to comments, other ideas,... It was just easier to provide the PR than trying to explain the concept. I also realize that does not complete reacts to the discussion in #254, but I do think the matplotlib-section can still be integrated in a real capstone project ad maybe this can (re)open the discussion on what to do with this episode.

maxim-belkin

I left stylistic comments - please fix the issues I noted (and similar ones throughout the PR).
We can then discuss the change in cognitive load and the PR in general.

Thanks for the work! 🥇

maxim-belkin · 2018-06-20T06:25:09Z

-    - "FIXME"
+	- "Matplotlib is the engine behind plotnine and Pandas plots."
+	- "The object-oriented way of matplotlib makes detailed customization possible."
+	- Use `savefig` method to save a figure as a file;


use double quotes. remote semicolon at the end

maxim-belkin · 2018-06-20T06:27:26Z

-toolbox `matplotlib.pyplot` is a collection of functions that make matplotlib
-work like MATLAB. In most cases, this is all that you will need to use, but
-there are many other useful tools in matplotlib that you should explore.
+[Matplotlib](http://matplotlib.org/) is a Python package used widely throughout the scientific Python community to produce high quality and publication-ready graphics. It supports a wide range of output formats including PNG (and other raster formats), PostScript/EPS, PDF and SVG. 


Title says Matplotlib plot *library*, the first sentence says ...Python *package*. It might be useful to use one or the other but not both.

high-quality because it refers to graphics

maxim-belkin · 2018-06-20T06:29:58Z


-We will cover a few basic commands for formatting plots in this lesson. A great
-resource for help styling your figures is the matplotlib gallery
+At the same time, matplotlib is the actual engine behind the plotting capabilities of both the Pandas as well as the plotnine package. So, if you you call the `.plot` functionality of Pandas, as we did in the previous episodes, you actually used the matplotlib package:


as well as ~> and (this is my personal preference, so you can ignore it if you disagree)

Fix: So, if you call.... and you actually used (either: When you called or you actually use)

maxim-belkin · 2018-06-20T06:31:44Z

+
+~~~
+    import pandas as pd
+    surveys = pd.read_csv("./data/surveys.csv")


remove whitespaces in the beginning

of the first two lines

maxim-belkin · 2018-06-20T06:32:20Z

+~~~
+{: .language-python}
+
+![png](../fig/08_scatter_surveys.png)


provide a better description than png

maxim-belkin · 2018-06-20T06:37:25Z

 ~~~
-    plt.plot([6.8, 4.3, 3.2, 8.1], list_numbers)
-    plt.show()
+    plt.plot(x, y, '-')


same as above.. remove whitespaces in the beginning of the line

maxim-belkin · 2018-06-20T06:37:47Z

-line. For example, we can make the line red (`'r'`), with circles at every data
-point (`'o'`), and a dot-dash pattern (`'-.'`). Look through the matplotlib
-gallery for more examples.
+![png](../fig/08_line_plot.png)


provide better description than png

maxim-belkin · 2018-06-20T06:38:03Z

-gallery for more examples.
+![png](../fig/08_line_plot.png)
+
+or create a figure and ax object first and add the plot to the created ax object:


maxim-belkin · 2018-06-20T06:38:39Z

-    plt.plot([6.8, 4.3, 3.2, 8.1], list_numbers, 'ro-.')
-    plt.axis([0,10,0,6])
-    plt.show()
+    fig, ax = plt.subplots()  # initiate an empty figure and ax matplotlib object


remove whitespace before fig

maxim-belkin · 2018-06-20T06:40:26Z

-A single figure can include multiple lines, and they can be plotted using the
-same `plt.plot()` command by adding more pairs of x values and y values (and
-optionally line styles):
+Although the latter requires a little bit more code to create the same plot, the advantage is that we now have **full control** of where the plot axes are placed, and we can easily add new items or, for example more than one axis to the figure and adapting the labels::


:: ~> :
or, for example, more than...
adapting ~> adapt ?

stijnvanhoey · 2018-06-20T07:28:13Z

Thanks for the detailed review, the comments on the 'template styling' helped me a lot! I hope I addressed all the comments, feel free to provide more feedback.

wrightaprilm · 2018-06-26T21:03:41Z

 ~~~
-    import matplotlib.pyplot as plt
+import pandas as pd
+surveys = pd.read_csv("./data/surveys.csv")


previous lessons just do data/surveys.csv, would you mind switching to that?

IMHO, there is nothing wrong with using ./ but I agree that for the sake of consistency it makes sense to use one approach/notation. It does not mean it won't change in the future 😉

consistency matter, I'll fix it

wrightaprilm · 2018-06-26T21:05:48Z

Sorry I'm so late in getting to this. I think once Maxim's notes are addressed, we're good to go. Thanks for all your effort on this, it looks really nice.

maxim-belkin · 2018-06-26T21:19:52Z

-    - "FIXME"
+	- "Matplotlib is the engine behind plotnine and Pandas plots."
+	- "The object-oriented way of matplotlib makes detailed customization possible."
+	- "Use `savefig` method to save a figure as a file."


I've just noticed that you used tabs here. Let's use 4 spaces instead.

maxim-belkin · 2018-06-26T21:44:34Z

+## Matplotlib package

-### Using pyplot:
+[Matplotlib](http://matplotlib.org/) is a Python package used widely throughout the scientific Python community to produce high-quality and publication-ready graphics. It supports a wide range of output formats including PNG (and other raster formats), PostScript/EPS, PDF and SVG. 


How about something like this

[Matplotlib](https://matplotlib.org/) is a Python package that is widely used throughout the scientific Python community to create high-quality and publication-ready graphics. It supports a wide range of raster and vector graphics formats including PNG, PostScript, EPS, PDF, SVG, and .

maxim-belkin · 2018-06-26T21:47:36Z

+[Matplotlib](http://matplotlib.org/) is a Python package used widely throughout the scientific Python community to produce high-quality and publication-ready graphics. It supports a wide range of output formats including PNG (and other raster formats), PostScript/EPS, PDF and SVG. 

-First, import the pyplot toolbox:
+At the same time, matplotlib is the actual engine behind the plotting capabilities of both the Pandas and the plotnine package. When you called the `.plot` functionality of Pandas, as we did in the previous episodes, you actually used the matplotlib package:


At the same time ~> Moreover

For example, when we call the `.plot` method on Pandas data objects, we actually use the matplotlib package:

maxim-belkin · 2018-06-26T21:48:59Z

 ~~~
-    import matplotlib.pyplot as plt
+import pandas as pd
+surveys = pd.read_csv("./data/surveys.csv")


IMHO, there is nothing wrong with using ./ but I agree that for the sake of consistency it makes sense to use one approach/notation. It does not mean it won't change in the future 😉

maxim-belkin · 2018-06-26T21:52:42Z

-writing:
+![Scatter plot of survey data set](../fig/08_scatter_surveys.png)
+
+The returned object is a `matplotlib.axes._subplots.AxesSubplot` matplotlib object (check it yourself with `type(my_plot)`) and the power of matplotlib is available to further adjust these plots as it is created with matplotlib itself.


I'm not sure that the fact that the object is of matplotlib.axes._subplots.AxesSubplot type tells us anything and/or is important for this episode.

Perhaps, we should say that the returned object is of special type and matplotlib can further tune its appearance.

maxim-belkin · 2018-06-26T22:01:14Z


-    plt.legend(loc='upper left', shadow=True, fontsize='x-large')
+~~~
+fig, ax1 = plt.subplots() #prepare a matplotlib figure


add a space after #

maxim-belkin · 2018-06-26T22:01:30Z

-    # This is a second figure
-    plt.figure(2)
-    plt.plot(t, t**2, 'bs-', label='square')
+fig, ax1 = plt.subplots() #prepare a matplotlib figure


add a space after #

maxim-belkin · 2018-06-26T22:03:11Z

-    plt.legend(loc='upper left', shadow=True, fontsize='x-large')
-    plt.title('This is figure 2')
+surveys.plot("hindfoot_length", "weight", 
+kind="scatter", ax=ax1) # use Pandas for plotting


surveys.plot("hindfoot_length", "weight", kind="scatter", ax=ax1)

maxim-belkin · 2018-06-26T22:11:11Z

+p9_ax.set_xlabel("Hindfoot length")
+p9_ax.tick_params(labelsize=16, pad=8)
+p9_ax.set_title('Scatter plot of weight versus hindfoot length', fontsize=15)
+my_plt_version


my_plt_version.show()?

Is not required when using the matplotlib inline magic function as far as I know...

This works fine on my install of jupyter notebook.

I think it's better to add my_plt_version.show() # not necessary in Jupyter Notebooks

maxim-belkin · 2018-06-26T22:12:37Z

-2 dimensional line plots. Look through the examples in
-http://matplotlib.org/users/screenshots.html and try a few of them (click on the
+Matplotlib can make many other types of plots in much the same way that it makes 2 dimensional line plots. Look through the examples in
+<http://matplotlib.org/users/screenshots.html and try a few of them (click on the


<http://matplotlib.org/users/screenshots.html>

stijnvanhoey · 2018-09-16T15:26:48Z

I tried to cover the remaining comments as far as I could.

maxim-belkin · 2018-09-16T18:52:49Z

Thanks, @stijnvanhoey. I made minor changes.
The last thing I suggest we do in this PR before merging it is update time estimates for the episode

maxim-belkin · 2018-09-16T21:38:46Z

-point (`'o'`), and a dot-dash pattern (`'-.'`). Look through the matplotlib
-gallery for more examples.
+
+### `plt` pyplot versus object-based matplotlib


I'm not sure I understand the "comparison" (implied by "versus") here...

Agree - I actually sort of struggle with articulating for the sorts of learners we have (who are generally not familiar with matlab) what the two ways of interaction really mean. 'Building layered plots with pyplot', perhaps?

we import pyplot module of matplotlib package as plt (plt === matplotlib.pyplot).
@stijnvanhoey, what were you trying to convey with this title?

...layered plots... have not been defined yet :)

I actually got versus from the matplotlib documentation itself: https://matplotlib.org/tutorials/introductory/lifecycle.html#a-note-on-the-object-oriented-api-vs-pyplot

Using plt pyplot to refer to what learners will typically recognise already or encounter online (e.g. stackoverflow) instead of adding more technical terms such as state-based interface...

Oh, that! That's pretty "deep"! I'd suggest moving this discussion (about stateful plt vs OO matplotlib) to a callout box or something (or drop entirely). Or add to instructors notes so that at least instructors are aware of such a thing.

Maybe we should create a separate issue on how to tackle this with respect to squashing/merging?

wrightaprilm · 2018-09-17T01:16:57Z

-point (`'o'`), and a dot-dash pattern (`'-.'`). Look through the matplotlib
-gallery for more examples.
+
+### `plt` pyplot versus object-based matplotlib


Agree - I actually sort of struggle with articulating for the sorts of learners we have (who are generally not familiar with matlab) what the two ways of interaction really mean. 'Building layered plots with pyplot', perhaps?

wrightaprilm · 2018-09-19T17:53:00Z

I'm going to run this tomorrow in my computational biology class, and will have a time estimate then.

maxim-belkin · 2018-09-19T18:37:46Z

Will you be using a locally-rendered version of the website?

wrightaprilm · 2018-09-19T18:41:28Z

I'll be converting it to a jupyter notebook and serving it on my jupyter hub. So I'll Stijn's fork and build from there.

wrightaprilm · 2018-09-20T20:37:57Z

Adds about 1:45. I'm counting from this episode functionally not existing right now - not sure how long what is there right now takes. I'm in favor of merge; I'll make an updated schedule once that's done.

maxim-belkin · 2018-09-21T12:53:36Z

Sounds good. Let's squash and merge this PR and update timings in a separate PR

wrightaprilm · 2018-09-21T19:36:24Z

Squashed and merged. Thanks so much, both of you, this is a really nice addition. I might make some edits in the coming weeks, according to sticky points I saw in the run-through, but I'll tag you both in when I get it done.

Very exciting stuff!

maxim-belkin · 2018-09-21T20:12:45Z

Excellent work, @stijnvanhoey!

stijnvanhoey · 2018-09-21T21:29:10Z

Thanks for the in-depth reviewing @maxim-belkin, certainly improved the material.

* Rewrite intor about matplotlib * Explain the bst of both worlds strategy * Replace matplotlib introduction section by link other packages * Update objectives and keypoints * Convert challenge * Add figures to episode * Add figures to fig folder * Subtitle adjustments * Add section about saving figures, fix #293 * Fix filenames figures * Fix relative links to figures * Fix keypoint style * Use package consistent * Fix typo on high-quality * Fix grammar style * Remove whitespaces at start codeblocks * Improve naming of figures * Fix syntax of callout section * Fix URL formatting * Remove redundant plot description * Adapt numpy import to python-novice-inflammation style * Use full object name for axis * Fix paragraph text * Use spaces instead of tabs * Update matplotlib description * Update the link Pandas and mpl * Use relative paths without explicit link to current folder * Remove explicit reference to mpl object * Improve language * Update ipython to jupyter * Add short intro on interaction mpl and np * Add space after hashtags * Fix URL tags * Fix indendation * adding language on .show() * a couple language edits * image tag * clarify objectives * remove 'further' * Update 08-putting-it-all-together.md * Update 08-putting-it-all-together.md * Fix y versus x * Add plot method to support none-notebook users * Rephrase the control on axis * Adapt keypoint * Use characters instead number * Use link to data file * Use method consistently for savefig * Update 08-putting-it-all-together.md * Update 08-putting-it-all-together.md * Update 08-putting-it-all-together.md * Update 08-putting-it-all-together.md

stijnvanhoey added 11 commits June 18, 2018 22:45

Rewrite intor about matplotlib

efb39eb

Explain the bst of both worlds strategy

a161865

Replace matplotlib introduction section by link other packages

add4f88

Update objectives and keypoints

65868d8

Convert challenge

ff96558

Add figures to episode

4ef8f4d

Add figures to fig folder

a41f1e6

Subtitle adjustments

2a9c134

Add section about saving figures, fix datacarpentry#293

60f60f8

Fix filenames figures

f9593b2

Fix relative links to figures

a22c8e1

maxim-belkin suggested changes Jun 20, 2018

View reviewed changes

stijnvanhoey added 12 commits June 20, 2018 09:05

Fix keypoint style

ccc192a

Use package consistent

b130f01

Fix typo on high-quality

da75e2b

Fix grammar style

2375fb0

Remove whitespaces at start codeblocks

139d15b

Improve naming of figures

88585aa

Fix syntax of callout section

09811b0

Fix URL formatting

ab684db

Remove redundant plot description

8d98c79

Adapt numpy import to python-novice-inflammation style

a6f0100

Use full object name for axis

d872ecd

Fix paragraph text

a16bf2e

wrightaprilm reviewed Jun 26, 2018

View reviewed changes

maxim-belkin suggested changes Jun 26, 2018

View reviewed changes

stijnvanhoey added 2 commits June 27, 2018 12:02

Use spaces instead of tabs

1ef7dc8

Update matplotlib description

53a6d2f

stijnvanhoey added 6 commits September 16, 2018 16:40

Add plot method to support none-notebook users

160b9ef

Rephrase the control on axis

15402df

Adapt keypoint

0d4ef9e

Use characters instead number

d89eb66

Use link to data file

bd138d0

Use method consistently for savefig

41db242

maxim-belkin added 2 commits September 16, 2018 18:39

Update 08-putting-it-all-together.md

722066c

Update 08-putting-it-all-together.md

5563a78

maxim-belkin approved these changes Sep 16, 2018

View reviewed changes

maxim-belkin changed the title ~~rewrite matplotlib section lesson 08~~ Episode 08 (Putting it all together): complete rewrite Sep 16, 2018

maxim-belkin assigned maxim-belkin and wrightaprilm Sep 16, 2018

maxim-belkin added type:enhancement Propose enhancement to the lesson high priority Need to be addressed ASAP labels Sep 16, 2018

maxim-belkin added 2 commits September 16, 2018 21:07

Update 08-putting-it-all-together.md

485495a

Update 08-putting-it-all-together.md

b30113e

maxim-belkin reviewed Sep 16, 2018

View reviewed changes

wrightaprilm reviewed Sep 17, 2018

View reviewed changes

wrightaprilm merged commit 34ec842 into datacarpentry:gh-pages Sep 21, 2018

Uh oh!

Conversation

stijnvanhoey commented Jun 18, 2018

Uh oh!

maxim-belkin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stijnvanhoey commented Jun 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wrightaprilm commented Jun 26, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stijnvanhoey commented Sep 16, 2018

Uh oh!

maxim-belkin commented Sep 16, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment