Scientific Methodology and Performance Evaluation for Computer Scientists
Reporting errors: Although I do my best there may definitely be typos and broken links. This is github so please report me everything you find so that I can improve for others. :)
This webpage gathers information about the lectures given at the University of Rennes at the 2nd year Master students in Computer Science.
- Here is the Pad you will use to collaborate.
- Here is a poll. Please fill it as soon as you have a few minutes but do not waste your time doing it during the lecture.
Links to the slides are provided below.
Course Objective and Organization
The aim of this course is to provide the fundamental basis for sound scientific methodology of performance evaluation of computer systems. This lecture emphasize on methodological aspects of measurement and on the statistics needed to analyze computer systems. I first sensibilize the audience to the experiment and analysis reproducibility issue in particular in computer science. Then I present tools that help answering the analysis problem and may also reveal useful for managing the experimental process through notebooks. The audience is given the basis of probabilities and statistics required to develop sound experiment designs. Unlike some other lectures, my goal is not to provide analysis recipes that people can readily apply but to make people really understand some simple tools so that they can then dig deeper later on.
The lecture will be based on the following set of slides, which I will probably adapt to the audience on the fly.
- Reproducible research
- Literate programming
- R crash course
- You may want to look at the Learning R section or (probably better) follow this short series of tutorials from software carpentry that are quite pedagogical:
- Descriptive statistics of univariate data
- Data presentation
- Correlation and causation
- Introduction to probabilities/statistics (Proof of the Central limit theorem)
- Linear regression
- Design of Experiments
It is not expected that students already knows the R language as I will briefly present it. However, they should have already installed Rstudio and R (check the next section if you need information) on their laptop so as to try out the examples I provide for themselves.
Installing R and Rstudio
Here is how to proceed on debian-based distributions:
sudo apt-get install r-base r-cran-ggplot2 r-cran-reshape
Make sure you have a recent (>= 3.2.0) version or R. For example, here is what I have on my machine:
R version 3.3.1 (2016-06-21) -- "Bug in Your Hair" Copyright (C) 2016 The R Foundation for Statistical Computing Platform: x86_64-pc-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under the terms of the GNU General Public License versions 2 or 3. For more information about these matters see http://www.gnu.org/licenses/.
Rstudio and knitr are unfortunately not packaged within debian so the easiest is to download the corresponding debian package on the Rstudio webpage and then to install it manually (depending on when you do this and on the version of your OS, you can obviously change the version number).
wget https://download1.rstudio.org/rstudio-xenial-1.0.153-amd64.deb sudo dpkg -i rstudio-xenial-1.0.153-amd64.deb sudo apt-get -f install # to fix possibly missing dependencies
You will also need to install knitr. To this end, you should simply run R (or Rstudio) and use the following command.
r-cran-reshape could not be installed for some
reason, you can also install it through R by doing:
We can roughly distinguish between three kinds of documents:
- Lab notebook (with everything you try and that is meant mainly for yourself)
- Experimental report (selected results and explanations with enough details to discuss with your advisor)
- Result description (rather short with only the main point and, which could be embedded in an article)
We expect you to provide us the last two ones and to make them publicly available so as to allow others to comment on them.
For a quick start, you may want to look at R for Beginners. A probably more entertaining way to go is to follow a good online lecture providing an introduction to R and to data analysis such as this one: https://www.coursera.org/course/compdata.
A quite effective way is to use SWIRL, an interactive learning environment that will guide through self-paced lesson.
install.packages("swirl") library(swirl) install_from_swirl("R Programming") swirl()
I suggest in particular to follow the following lessons from R programming (max 10 minutes each):
1: Basic Building Blocks 2: Workspace and Files 3: Sequences of Numbers 4: Vectors 5: Missing Values 6: Subsetting Vectors 7: Matrices and Data Frames 8: Logic 9: Functions 12: Looking at Data
Finally, you may want to read this excellent tutorial on data frames
dimnames, notions of scope…).
Learning ggplot2, plyr/dplyr, reshape/tidyR
All these packages have been developed by hadley wickam.
- Although the package is called
ggplot2, it provides you the
ggplotcommand. This package allows you to produce nice looking and highly configurable graphics.
- Old generation:
plyrallows you expressively compute aggregate statistics on your data-frames and
reshapeallows you to reshape your data-frames if they’re not in the right shape for
- New generation:
dplyris the new generation of
plyrand allows you to expressively compute aggregate statistics on your data-frames.
tidyris the new generation of
reshapeand allows you to reshape your data-frames if they’re not in the right shape for
dplyr. If you have a recent R installation, go for these new packages. Their syntax is better and their implementation is much faster.
I recently stumbled on this nice ggplot2 tutorial.
- R. Jain, The Art of Computer Systems Performance Analysis:
Techniques for Experimental Design, Measurement, Simulation, and
Modeling, Wiley-Interscience, New York, NY, April 1991.
A new edition will be available in September 2015.
This is an easy-to-read self-content book for practical performance evaluation. The numerous checklists make it a great book for engineers and every CS experimental scientist should have read it.
- David J. Lilja, Measuring Computer Performance: A Practitioner’s
Guide, Cambridge University Press 2005
A short book suited for brief presentations. I follow a similar organization but I really don’t like the content of this book. I feel it provides very little insight on why the theory applies or not. I also think it is too general and lacks practical examples. It may be interesting for those willing a quick and broad presentation of the main concepts and “recipes” to apply.
- Jean-Yves Le Boudec. Methods, practice and theory for the
performance evaluation of computer and communication
systems, 2006. EPFL electronic book.
A very good book, with a much more theoretical treatment than the Jain. It goes way farther on many aspects and I can only recommand it.
- Douglas C. Montgomery, Design and Analysis of Experiments, 8th
Edition. Wiley 2013.
This is a good and thorough textbook on design of experiments. It’s so unfortunate it relies on “exotic” softwares like JMP and minitab instead of R…
- Julian J. Faraway, Practical Regression and Anova using R,
University of Bath, 2002.
This book is derived from material that Pr. Faraway used in a Master level class on Statistics at the University of Michigan. It is mathematically involved but presents in details how linear regression, ANOVA work and can be done with R. It works out many examples in details and is very pleasant to read. A must-read if you want to understand this topic more thoroughly.
- Peter Kosso, A Summary of Scientific Method, Springer, 2011. [hidden
PDF that google found on the webpage of a university in Macedonia
A short nice book summarizing the main steps of the scientific method and why having a clear definition is not that simple. It illustrates these points with several nice historical examples that allow the reader to take some perspective on this epistemological question.
- R. Nelson, Probability stochastic processes and queuing theory: the
mathematics of computer performance modeling. Springer Verlag 1995.
For those willing to know more about queuing theory.