forked from sta523-fa20/lecture_slides
-
Notifications
You must be signed in to change notification settings - Fork 1
/
lec_00.html
478 lines (344 loc) · 13.4 KB
/
lec_00.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
<!DOCTYPE html>
<html lang="" xml:lang="">
<head>
<title>Welcome and Introduction</title>
<meta charset="utf-8" />
<meta name="author" content="Shawn Santo" />
<link rel="stylesheet" href="slides.css" type="text/css" />
</head>
<body>
<textarea id="source">
class: center, middle, inverse, title-slide
# Welcome and Introduction
## Programming for Statistical Science
### Shawn Santo
---
## Supplementary materials
Companion videos
- [Overview](https://warpwire.duke.edu/w/eyAEAA/)
---
class: center, inverse, middle
# Introduction
---
## Who am I?
- [Shawn Santo](https://www.shawnsanto.com)
- [shawn.santo@duke.edu](mailto:shawn.santo@duke.edu)
- Office hours (Zoom link in Sakai)
- Mondays 8:15pm - 9:15pm
- Fridays 11:30am - 12:30pm
<br/><br/>
.small-text[
*All times listed are in Eastern Time.*
]
---
## Who else is involved?
- [Federico Ferrari](https://fedfer.github.io/)
- [federico.ferrari@duke.edu](mailto:federico.ferrari@duke.edu)
- Office hours (Zoom link in Sakai)
- Tuesdays 4:00 - 5:00pm
- [Quinn Frank](https://github.com/quinnfrank)
- [quinn.frank@duke.edu](mailto:quinn.frank@duke.edu)
- Office hours (Zoom link in Sakai)
- Tuesdays 1:30pm - 2:30pm
- Thursdays 1:30pm - 2:30pm
<br/><br/>
.small-text[
*All times listed are in Eastern Time.*
]
---
## What is statistical programming?
.middle.center[
<img src="images/statistical_computing_venn.png" height="400px">
]
.small-text[
*Source:* Deborah Nolan & Duncan Temple Lang (2010) Computing in the Statistics
Curricula, The American Statistician, 64:2, 97-107, DOI:
[10.1198/tast.2010.09132](https://doi.org/10.1198/tast.2010.09132)
]
---
## Some topic covered
.pull-left[
- Fundamentals of R
- S3 OO system
- Data visualization with package `ggplot2`
- Package `tidyverse`
- Web scraping
- Web based applications with RShiny
- Wrangling and managing big data
- Databases, SQL, and NoSQL
]
.pull-right[
- Data types and functions
- Parallelization
- Version control with git and GitHub
- Shell
- Reproducible reports with R Markdown
- Debugging and testing
- Spark
- Make
]
<br><br>
[Full course schedule](http://www2.stat.duke.edu/courses/Fall20/sta523/schedule.html)
---
## Why this class matters
>Programming Languages: Python and R continue to dominate. The new entry this
year was Javascript, which got a respectable 6.8% share. Julia share has
increased, while most other languages have declined.
.tiny-text[
| Platform | 2019 % share | 2018 % share | % change |
|--------------------------------------|--------------|--------------|----------|
| Python | 65.8% | 65.6% | 0.2% |
| R Language | 46.6% | 48.5% | -4.0% |
| SQL Language | 32.8% | 39.6% | -17.2% |
| Java | 12.4% | 15.1% | -17.7% |
| Unix shell/awk | 7.9% | 9.2% | -13.4% |
| C/C++ | 7.1% | 6.8% | 3.7% |
| Javascript | 6.8% | na | na |
| Other programming and data languages | 5.7% | 6.9% | -17.1% |
| Scala | 3.5% | 5.9% | -41.0% |
| Julia | 1.7% | 0.7% | 150.4% |
| Perl | 1.3% | 1.0% | 25.2% |
| Lisp | 0.4% | 0.3% | 46.1% |
]
.small-text[
*Source:* https://www.kdnuggets.com/2019/05/poll-top-data-science-machine-learning-platforms.html
]
---
## Why this class matters
Some 2020 internships:
- <b>Mayo Clinic :</b> <i> Interns will work with statisticians,
bioinformaticists, and clinical investigators on research projects in areas
such as clinical trials, statistical genetics, and bioinformatics.
Experience with SAS and/or R preferred.</i>
- <b>Netflix (Science and Analytics):</b> <i> Comfortable coding
in at least one language (e.g., R, Python, Java, Scala, C++), experience
preferred with version control (e.g., git), great communication skills,
both oral and written.</i>
- <b>Two Sigma:</b>
<i> Use the scientific method to develop sophisticated investment models and
shape our insights into how the markets will behave. Create and test complex
investment ideas and partner with our engineers to test your theories.
You should possess the following qualifications: Demonstrate intermediate
skills in at least one programming language, performed an in-depth research
project, examining real-world data, are an independent thinker who can
creatively approach data analysis and communicate complex ideas clearly.</i>
<br/>
.small-text[
*Source:* https://stattrak.amstat.org/2019/12/01/2020-internship-listings/
]
---
class: center, inverse, middle
# Course essentials
---
## Text toolkit
.middle.center[
![](images/texts.png)
]
These are recommended textbooks - **all are available for free online**.
There is no required textbook for this course.
---
## Software toolkit
.middle.center[
![](images/software.png)
]
---
## Course structure
This class is about you doing as opposed to you just watching or listening.
In-person and Zoom, lectures and labs will be interactive. My role as instructor
is to introduce you to new tools and techniques, but it is up to you to take
them and make use of them. If you only read the code and never run it or
experiment with it, then you will not get much out of this course. Most slides
will include supplemental resources for you to delve deeper into the topic of
discussion. Occasionally, there will be pre-class readings in order to enrich
our lecture and lab experiences.
---
## Grading
| Grade Item | Percentage |
|-----------:|:----------:|
| Homework | 45% |
| Exam 1 | 20% |
| Exam 2 | 20% |
| Project | 15% |
<br/><br/>
The exact ranges for letter grades may be curved and cutoffs will be determined
at the end of the semester. However, if you have a cumulative numerical average
of 90 - 100, you are guaranteed at least an A-, 80 - 89 at least a B-, 70 - 79
at least a C-, and so on.
---
## Teams
I will initially construct teams based on the first-day class survey (link
available later in the slides).
Team expectations:
- Each member must commit to giving equal effort.
- Each member must read, run, and understand all code in a final submission.
- Each member must honestly complete the intragroup peer evaluation.
---
## Policies - sharing / reusing code
- Similar reproducible examples (reprex) exist online that will help you
answer many of the questions posed on labs and homework assignments. Use of
these resources is allowed unless it is written explicitly on the assignment.
- You must always cite any code you copy or use as inspiration. Copied code
without citation is plagiarism and will result in a 0 for the assignment.
There may also be additional punitive measures taken depending on the
severity of plagiarism.
- Copying and citing a large amount of code to satisfy a main objective of an
assignment will result in a 0 for the assignment.
- Discussion (not code sharing / copying) with other students and groups is
allowed unless stated otherwise on an assignment.
- Carefully read each assignment so you know what is permitted and
what is not. If you are ever unsure about what is allowed, please ask myself
or one of the TAs.
---
## Getting help
- Post your content and course related questions on Slack.
- Attend office hours.
- Set up a Zoom meeting with me or one of the TAs.
- Email me or one of the TAs.
---
## Links to bookmark
- Course page: http://www2.stat.duke.edu/courses/Fall20/sta523/
- GitHub organization: https://github.com/sta523-fa20
- I'll send out an invite to join once I have everyone's user name
- DSS RStudio servers: to access, first connect to Duke's network via VPN.
Navigate to one of:
- rook - http://rook.stat.duke.edu:8787/
- knight - http://knight.stat.duke.edu:8787/
---
## To do list
Before Wednesday's lecture please complete the following (in order).
1. Create a GitHub account.
- https://github.com/join
2. Join the course's Slack workspace.
- link was sent in the welcome email on 08-14-20
3. Verify you can log-in to the Department's RStudio Pro servers.
4. Complete the first-day survey.
- posted in Slack under `#general`
5. Download the most up-to-date versions of R and RStudio locally.
---
## References
1. Deborah Nolan & Duncan Temple Lang (2010) Computing in the Statistics
Curricula, The American Statistician, 64:2, 97-107,
DOI: 10.1198/tast.2010.09132
2. Piatetsky, G. (2019). Python leads the 11 top Data Science, Machine Learning
platforms: Trends and Analysis. Kdnuggets.com. Retrieved from
https://www.kdnuggets.com/2019/05/poll-top-data-science-machine-learning-platforms.html
</textarea>
<style data-target="print-only">@media screen {.remark-slide-container{display:block;}.remark-slide-scaler{box-shadow:none;}}</style>
<script src="https://remarkjs.com/downloads/remark-latest.min.js"></script>
<script>var slideshow = remark.create({
"highlightStyle": "github",
"highlightLines": true,
"countIncrementalSlides": false
});
if (window.HTMLWidgets) slideshow.on('afterShowSlide', function (slide) {
window.dispatchEvent(new Event('resize'));
});
(function(d) {
var s = d.createElement("style"), r = d.querySelector(".remark-slide-scaler");
if (!r) return;
s.type = "text/css"; s.innerHTML = "@page {size: " + r.style.width + " " + r.style.height +"; }";
d.head.appendChild(s);
})(document);
(function(d) {
var el = d.getElementsByClassName("remark-slides-area");
if (!el) return;
var slide, slides = slideshow.getSlides(), els = el[0].children;
for (var i = 1; i < slides.length; i++) {
slide = slides[i];
if (slide.properties.continued === "true" || slide.properties.count === "false") {
els[i - 1].className += ' has-continuation';
}
}
var s = d.createElement("style");
s.type = "text/css"; s.innerHTML = "@media print { .has-continuation { display: none; } }";
d.head.appendChild(s);
})(document);
// delete the temporary CSS (for displaying all slides initially) when the user
// starts to view slides
(function() {
var deleted = false;
slideshow.on('beforeShowSlide', function(slide) {
if (deleted) return;
var sheets = document.styleSheets, node;
for (var i = 0; i < sheets.length; i++) {
node = sheets[i].ownerNode;
if (node.dataset["target"] !== "print-only") continue;
node.parentNode.removeChild(node);
}
deleted = true;
});
})();
(function() {
"use strict"
// Replace <script> tags in slides area to make them executable
var scripts = document.querySelectorAll(
'.remark-slides-area .remark-slide-container script'
);
if (!scripts.length) return;
for (var i = 0; i < scripts.length; i++) {
var s = document.createElement('script');
var code = document.createTextNode(scripts[i].textContent);
s.appendChild(code);
var scriptAttrs = scripts[i].attributes;
for (var j = 0; j < scriptAttrs.length; j++) {
s.setAttribute(scriptAttrs[j].name, scriptAttrs[j].value);
}
scripts[i].parentElement.replaceChild(s, scripts[i]);
}
})();
(function() {
var links = document.getElementsByTagName('a');
for (var i = 0; i < links.length; i++) {
if (/^(https?:)?\/\//.test(links[i].getAttribute('href'))) {
links[i].target = '_blank';
}
}
})();
// adds .remark-code-has-line-highlighted class to <pre> parent elements
// of code chunks containing highlighted lines with class .remark-code-line-highlighted
(function(d) {
const hlines = d.querySelectorAll('.remark-code-line-highlighted');
const preParents = [];
const findPreParent = function(line, p = 0) {
if (p > 1) return null; // traverse up no further than grandparent
const el = line.parentElement;
return el.tagName === "PRE" ? el : findPreParent(el, ++p);
};
for (let line of hlines) {
let pre = findPreParent(line);
if (pre && !preParents.includes(pre)) preParents.push(pre);
}
preParents.forEach(p => p.classList.add("remark-code-has-line-highlighted"));
})(document);</script>
<script>
slideshow._releaseMath = function(el) {
var i, text, code, codes = el.getElementsByTagName('code');
for (i = 0; i < codes.length;) {
code = codes[i];
if (code.parentNode.tagName !== 'PRE' && code.childElementCount === 0) {
text = code.textContent;
if (/^\\\((.|\s)+\\\)$/.test(text) || /^\\\[(.|\s)+\\\]$/.test(text) ||
/^\$\$(.|\s)+\$\$$/.test(text) ||
/^\\begin\{([^}]+)\}(.|\s)+\\end\{[^}]+\}$/.test(text)) {
code.outerHTML = code.innerHTML; // remove <code></code>
continue;
}
}
i++;
}
};
slideshow._releaseMath(document);
</script>
<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
var script = document.createElement('script');
script.type = 'text/javascript';
script.src = 'https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML';
if (location.protocol !== 'file:' && /^https?:/.test(script.src))
script.src = script.src.replace(/^https?:/, '');
document.getElementsByTagName('head')[0].appendChild(script);
})();
</script>
</body>
</html>