Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
245 lines (225 sloc) 13 KB

Scientific Methodology and Performance Evaluation for Computer Scientists

Reporting errors: Although I do my best there may definitely be typos and broken links. This is github so please report me everything you find so that I can improve for others. :)

This webpage gathers information about the lectures given at the University of Rennes at the 2nd year Master students in Computer Science.

Useful links

  • Here is the Pad you will use to collaborate.
  • Here is a poll. Please fill it as soon as you have a few minutes but do not waste your time doing it during the lecture.

Links to the slides are provided below.

Course Objective and Organization

The aim of this course is to provide the fundamental basis for sound scientific methodology of performance evaluation of computer systems. This lecture emphasize on methodological aspects of measurement and on the statistics needed to analyze computer systems. I first sensibilize the audience to the experiment and analysis reproducibility issue in particular in computer science. Then I present tools that help answering the analysis problem and may also reveal useful for managing the experimental process through notebooks. The audience is given the basis of probabilities and statistics required to develop sound experiment designs. Unlike some other lectures, my goal is not to provide analysis recipes that people can readily apply but to make people really understand some simple tools so that they can then dig deeper later on.

The lecture will be based on the following set of slides, which I will probably adapt to the audience on the fly.

  1. Epistomology
  2. Reproducible research
  3. Literate programming
  4. R crash course
  5. Descriptive statistics of univariate data
  6. Data presentation
  7. Correlation and causation
  8. Introduction to probabilities/statistics (Proof of the Central limit theorem)
  9. Linear regression
  10. Design of Experiments

All the examples given in this series of lecture use the R language and the source is provided so that people can reuse them. The slides are composed with org-mode, beamer, and verbments.

Requirements

All the examples given in this series of lecture use the R language and the source is provided so that people can reuse them. The slides are composed with org-mode, beamer, and verbments.

It is not expected that students already knows the R language as I will briefly present it. However, they should have already installed Rstudio and R (check the next section if you need information) on their laptop so as to try out the examples I provide for themselves.

Using R

Installing R and Rstudio

Here is how to proceed on debian-based distributions:

sudo apt-get install r-base r-cran-ggplot2 r-cran-reshape 

Make sure you have a recent (>= 3.2.0) version or R. For example, here is what I have on my machine:

R --version
R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"
Copyright (C) 2016 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.

Rstudio and knitr are unfortunately not packaged within debian so the easiest is to download the corresponding debian package on the Rstudio webpage and then to install it manually (depending on when you do this and on the version of your OS, you can obviously change the version number).

wget https://download1.rstudio.org/rstudio-xenial-1.0.153-amd64.deb
sudo dpkg -i rstudio-xenial-1.0.153-amd64.deb
sudo apt-get -f install # to fix possibly missing dependencies

You will also need to install knitr. To this end, you should simply run R (or Rstudio) and use the following command.

install.packages("knitr")

If r-cran-ggplot2 or r-cran-reshape could not be installed for some reason, you can also install it through R by doing:

install.packages("ggplot2")
install.packages("reshape")

Producing documents

The easiest way to go is probably to use R+Markdown (Rmd files) in Rstudio and to export them via Rpubs to make available whatever you want.

We can roughly distinguish between three kinds of documents:

  1. Lab notebook (with everything you try and that is meant mainly for yourself)
  2. Experimental report (selected results and explanations with enough details to discuss with your advisor)
  3. Result description (rather short with only the main point and, which could be embedded in an article)

We expect you to provide us the last two ones and to make them publicly available so as to allow others to comment on them.

Learning R

For a quick start, you may want to look at R for Beginners. A probably more entertaining way to go is to follow a good online lecture providing an introduction to R and to data analysis such as this one: https://www.coursera.org/course/compdata.

A quite effective way is to use SWIRL, an interactive learning environment that will guide through self-paced lesson.

install.packages("swirl")
library(swirl)
install_from_swirl("R Programming")
swirl()

I suggest in particular to follow the following lessons from R programming (max 10 minutes each):

1: Basic Building Blocks      2: Workspace and Files     
3: Sequences of Numbers       4: Vectors                 
5: Missing Values             6: Subsetting Vectors      
7: Matrices and Data Frames   8: Logic                   
9: Functions                 12: Looking at Data         

Finally, you may want to read this excellent tutorial on data frames (attach, with, rownames, dimnames, notions of scope…).

Learning ggplot2, plyr/dplyr, reshape/tidyR

All these packages have been developed by hadley wickam.

  • Although the package is called ggplot2, it provides you the ggplot command. This package allows you to produce nice looking and highly configurable graphics.
  • Old generation: plyr allows you expressively compute aggregate statistics on your data-frames and reshape allows you to reshape your data-frames if they’re not in the right shape for ggplot2 or plyr.
  • New generation: dplyr is the new generation of plyr and allows you to expressively compute aggregate statistics on your data-frames. tidyr is the new generation of reshape and allows you to reshape your data-frames if they’re not in the right shape for ggplot2 or dplyr. If you have a recent R installation, go for these new packages. Their syntax is better and their implementation is much faster.

I recently stumbled on this nice ggplot2 tutorial.

Hadley Wickam provides a nice tour of dplyr and gentle introduction to tidyR. Here is a nice link on merging data frames.

The Rstudio team has designed a nice series of cheatsheets on R and in particular one on ggplot2 and on R/markdown/knitr.

References

  • R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, Wiley-Interscience, New York, NY, April 1991. A new edition will be available in September 2015.

    This is an easy-to-read self-content book for practical performance evaluation. The numerous checklists make it a great book for engineers and every CS experimental scientist should have read it.

  • David J. Lilja, Measuring Computer Performance: A Practitioner’s Guide, Cambridge University Press 2005

    A short book suited for brief presentations. I follow a similar organization but I really don’t like the content of this book. I feel it provides very little insight on why the theory applies or not. I also think it is too general and lacks practical examples. It may be interesting for those willing a quick and broad presentation of the main concepts and “recipes” to apply.

  • Jean-Yves Le Boudec. Methods, practice and theory for the performance evaluation of computer and communication systems, 2006. EPFL electronic book.

    A very good book, with a much more theoretical treatment than the Jain. It goes way farther on many aspects and I can only recommand it.

  • Douglas C. Montgomery, Design and Analysis of Experiments, 8th Edition. Wiley 2013.

    This is a good and thorough textbook on design of experiments. It’s so unfortunate it relies on “exotic” softwares like JMP and minitab instead of R…

  • Julian J. Faraway, Practical Regression and Anova using R, University of Bath, 2002.

    This book is derived from material that Pr. Faraway used in a Master level class on Statistics at the University of Michigan. It is mathematically involved but presents in details how linear regression, ANOVA work and can be done with R. It works out many examples in details and is very pleasant to read. A must-read if you want to understand this topic more thoroughly.

  • Peter Kosso, A Summary of Scientific Method, Springer, 2011. [hidden PDF that google found on the webpage of a university in Macedonia

    A short nice book summarizing the main steps of the scientific method and why having a clear definition is not that simple. It illustrates these points with several nice historical examples that allow the reader to take some perspective on this epistemological question.

  • R. Nelson, Probability stochastic processes and queuing theory: the mathematics of computer performance modeling. Springer Verlag 1995.

    For those willing to know more about queuing theory.