New post for WoBloMo

files/css/screen.css

@@ -1,15 +1,12 @@
-/* @override http://mark.reid.dev/files/css/screen.css */
-
-/* @override http://mark.reid.dev/css/screen.css */
-
 /* screen.css
  *
  * A clean, simple stylesheet that aims for
  * a consistent vertical rhythm.
  *
  * Base font height: 16px
- * Base line lenght: 24px
+ * Base line length: 24px
  * 
+ * AUTHOR: Mark Reid <mark@reid.name>
  */
 
 /* @group Reset */
@@ -20,7 +17,7 @@ body {
 	font-family: Palatino, georgia, "times new roman", serif;
 	background-color: whitesmoke;
 	background-position: center -18em;
-    background-repeat: repeat-x; 
+	background-repeat: repeat-x; 
 }
 
 /* Hack via Joel Spolsky to make image rescaling nicer in IE */
@@ -40,9 +37,7 @@ h1,h2,h3,h4 {
 	clear: left; 	
 	font-family: 'lucida grande', sans-serif; 
 }
-h1 {
-	margin-bottom: 1em;
-}
+h1 { margin-bottom: 1em; }
 h2 {
 	font-size: 100%;
 	line-height: 1.5em;
@@ -65,7 +60,6 @@ hr {
 
 #page, #disqus_thread {
 	background: white;
-/*	background-image: url(grid); */
 	padding: 3em 3em 1.5em 3em;
 	border: 1px solid #ccc;
 }
@@ -139,8 +133,6 @@ caption {
 
 /* @end */
 
-
-
 /* @group Lists */
 ul {
 	margin-top: 1.5em;
@@ -242,7 +234,6 @@ body#Kith #header a.kith
 	line-height: 1.5em;
 }
 .footnotes ol li > a { display: none; }
-
 /* @end */
 
 /* @group Sections */
@@ -265,9 +256,7 @@ body#Kith #header a.kith
 	font-weight: bold;
 	font-variant: small-caps;
 }
-.section p {
-	margin-bottom: 1.43em;
-}
+.section p { margin-bottom: 1.43em; }
 
 /* @end */
 
@@ -277,7 +266,6 @@ p.excerpt { margin: 0; }
 .comments { font-size: smaller; position: absolute; color: silver; right: 0; top: 0; }
 .excerpt { color: black; }
 
-
 /* @group Signature */
 .signature { 
 	margin-top: 3em; 
@@ -417,8 +405,6 @@ p.delicious-extended { font-size: 100%; display: inline; }
 p.delicious-extended:before { content: ' — '; }
 /* @end */
 
-
-
 /* @group Footer */
 #footer {
 	margin-top: 0;

iem/_drafts/2009-02-18-demystifying-kernels.markdown

@@ -37,7 +37,7 @@ We will use $\mathcal{X}$ to denote the
 complete set of instances we are interested in. These could be documents, 
 images, graphs, whatever. 
 To measure the similarity between two instances $x_1, x_2 \in \mathcal{X}$
-we will use a function $k(x_1, x_2)$.
+we will use a function $k(x_1, x_2)$. Large values of 
 
 Implicit Representation
 -----------------------

iem/_posts/2009-03-01-proper-losses.markdown

@@ -0,0 +1,124 @@
+---
+title: "Probability Estimation: An Introduction"
+
+excerpt: Probability estimation is an important class of problem in machine learning. In this, the first of a series of posts, I discuss a natural class of losses for these problems.
+
+location: Canberra, Australia
+
+layout: post
+
+---
+
+In my recent research I have been looking at relationships between various types
+of learning problems. One surprisingly rich class of problems is that of 
+probability estimation. In this series of posts I will 
+highlight some of the interesting theory I've uncovered about them in the
+machine learning, statistics and economics literature.
+
+Binary Classification
+---------------------
+The quintessential type of learning problem in machine learning is binary 
+classification. Given a training sample of instances, each labelled "positive" 
+or "negative", the aim is to learn to predict the correct label from previously 
+unseen instances. A well known example of a binary classification problem is 
+predicting whether an email is spam or not spam.
+
+A binary classification problem can be stated as an optimisation: find 
+function that minimises the average number of misclassifications on new 
+instances drawn from the distribution that generated the training sample.
+Put another way, if we have to pay a penalty of $1 each time we predict a 
+positive instance as negative or _vice versa_ then we want to find a predictor 
+that minimises our expected loss.[^1]
+
+[^1]: Of course, in the case of spam the penalty is not as symmetric as 
+described here. Incorrectly predicting spam as "not spam" is mildly annoying
+whereas predicting that an important email that your career depends upon as
+"spam" and sending it to the Trash folder is potentially disastrous!
+
+Formally, we will say an instance $x$ is positive if it has associated label
+$y = 1$ and negative if its label $y = 0$.
+We then define the _0-1 misclassification loss_ for a binary prediction $p$
+when the label is $y$ to be
+\[	\displaystyle
+	\ell_{01}(y,p) = 
+	\begin{cases}
+		1, &	y = p \\
+		0, &	\text{otherwise}.
+	\end{cases}
+\]
+Now suppose that an instance $x$ has a positive label with probability 
+$\eta$ and we have made a prediction $p$. For that $x$ the _point-wise risk_
+is
+\[
+	\displaystyle
+	L_{01}(\eta,p) 
+	= \eta\,\ell_{01}(1,p) + (1-\eta)\,\ell_{01}(0,p).
+\]
+The first term is the average loss of a prediction $p$ in the case of a positive 
+example, occurring with probability $\eta$, and the second term is the 
+average loss for a negative example, occurring with probability $1-\eta$. 
+
+Returning to the spam example, suppose that with probability 0.95 a randomly 
+chosen recipient says a particular email is spam. A prediction of "spam" for
+that email will incur an average loss of $0.95\times 0 + 0.05\times 1 = 0.05$
+whereas a prediction of "not spam" incurs a loss of 0.95.
+
+Probability Estimation
+----------------------
+Now suppose that instead of merely predicting the correct label we wanted to 
+know the _probability_ that an email was considered spam. In this case we would 
+have a different but related type of learning problem: binary class probability 
+estimation.
+
+As predictions here are probabilities instead of concrete predictions, there 
+is no sensible notion of a misclassification.
+How can a prediction that an email is spam with probability 0.9 be wrong? If it
+really isn't spam it may just be one of the 10% of cases that are consistent 
+with the probability estimate. 
+
+What we really want is a penalty with an _expected value_ that is minimised 
+if our probability estimates are consistent with the _true probability_ of 
+a positive label for a given instance. This fairly natural requirement on
+the loss for a probability estimation problem is known as 
+_Fisher consistency_.
+
+If $\ell(y,p)$ is a loss for probability estimation then the above requirement
+can be framed in terms of its associated point-wise risk:
+$L(\eta,p) = \eta\,\ell(1,p) + (1-\eta)\,\ell(0,p)$. 
+Stated formally, Fisher consistency says that no matter what true probability
+$\eta$ we have
+\[  \displaystyle
+	\min_{p\in[0,1]} L(\eta,p) = L(\eta,\eta).
+\]
+That is, predicting $p = \eta$ always achieves the smallest possible point-wise 
+risk.
+
+We will call losses that have this Fisher consistency property _proper losses_
+in line with the terminology of _proper scoring rules_ used when 
+probability elicitation is studied in economics.[^2] We will see several 
+interesting connections between these two concepts in future posts.
+
+[^2]: Scoring rules are usually framed in terms of rewards rather than penalties
+so loss and scoring rules are additive inverse and minimisation here becomes 
+maximisation there.
+
+Examples
+--------
+One common loss functions used for probability estimation is
+_square loss_
+\[
+	\displaystyle
+	\ell_{\text{sq}}(y,p) = y(1-p)^2 + (1-y) p^2.
+\]
+The easiest way to convince yourself this is Fisher consistent is to
+examine when the derivatives of its point-wise risk with respect to $p$ 
+vanishes. In the case of square loss we see that
+\[
+	\frac{\partial}{\partial p} L_{\text{sq}}(\eta,p)
+	= -2\eta(1-p) + 2(1-\eta)p = 2(p-\eta)
+\]
+which is 0 only when $p=\eta$ and so $\ell_{\text{sq}}$ is proper.
+
+While Fisher consistency seems like a fairly innocuous and natural constraint
+for probability estimation it has some impressive implications that I will
+explore in some future posts.

iem/info.markdown

@@ -57,10 +57,10 @@ Colophon
 --------
 The posts in the blog are written in a hybrid of [Markdown][] (for text) and
 [LaTeX][] (for equations) and converted into HTML by [MaRuKu][] as part of 
-the [Jekyll][] engine. In fact, [my entire site](/) is managed this way.
+the [Jekyll][] engine. 
 
-There are more details about the design and technology at the 
-main site's [colophon][].
+[My entire site](/) is managed this way and there are more details about the 
+design and technology I use at the main site's [colophon][].
 
 [jekyll]: http://github.com/mreid/jekyll/
 [markdown]: http://daringfireball.net/projects/markdown/
@@ -70,9 +70,6 @@ main site's [colophon][].
 
 Licence
 -------
-<a rel="license" class="right inset" href="http://creativecommons.org/licenses/by-nc-sa/3.0/">
-	<img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-nc-sa/3.0/88x31.png" />
-</a>     
 
 Unless specified otherwise, all design and content within this site is 
-licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License</a>.
+licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/3.0/">Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License</a>.

images/latex/026c7653f26b03f640160ebafe5aba8b.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>14</height>
+<depth>6</depth>
+<md5>c98af96fe8ba2731158202ed577f8deb</md5>
+</png>
+</blahtex>

images/latex/06044014dfc852a83ace6c35d6290fe2.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>16</height>
+<depth>5</depth>
+<md5>b69ac48334edb38b980f2822e4065a0a</md5>
+</png>
+</blahtex>

images/latex/1a164001115c792d7a41524b6693feba.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>14</height>
+<depth>5</depth>
+<md5>291e39803fba76cfc9a59fcee9cfac27</md5>
+</png>
+</blahtex>

images/latex/42df1328780943886554e7a8489ef35f.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>14</height>
+<depth>4</depth>
+<md5>57f17482e91b2599eef6e6e5431cf646</md5>
+</png>
+</blahtex>

images/latex/648ff7c2e86ef3e13d00c331b3585551.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>35</height>
+<depth>25</depth>
+<md5>544a93e308e2f22304e864bfed3a2961</md5>
+</png>
+</blahtex>

images/latex/6ccfb3ff6045d1316be511a971ad7ece.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>9</height>
+<depth>5</depth>
+<md5>85c94d02a95009b5fd9410bf973ab842</md5>
+</png>
+</blahtex>

images/latex/7a35593b4e4a22ad316b4b8bd2c669ed.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>17</height>
+<depth>6</depth>
+<md5>83f74bea05c65573dda9adf4305b564a</md5>
+</png>
+</blahtex>

images/latex/7e8869c2cbf3f2f55a739fb5a8dbfb50.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>14</height>
+<depth>4</depth>
+<md5>b6b3c7c8a7d341752f70ca83759eefdd</md5>
+</png>
+</blahtex>

images/latex/8ab53bcf4142d96a650addc4b6076c01.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>9</height>
+<depth>4</depth>
+<md5>4d879bde964e549c6085ad630f56ad41</md5>
+</png>
+</blahtex>

images/latex/969737ce6c37817fb8003779589804d6.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>16</height>
+<depth>17</depth>
+<md5>3f19c18ceb40b7e00183da3b2cf15015</md5>
+</png>
+</blahtex>

images/latex/a152a93aa3d2cc2643952f1855979b91.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>9</height>
+<depth>4</depth>
+<md5>1673ac8aa9b48203da9b40ba52ef26de</md5>
+</png>
+</blahtex>

images/latex/a3ffa6f179e92000c7aa8b9519297572.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>16</height>
+<depth>5</depth>
+<md5>592597c51b7a4aa2b4ead62743024112</md5>
+</png>
+</blahtex>

images/latex/abca339ec226640f38049216f1be8021.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>14</height>
+<depth>2</depth>
+<md5>e966d0bd2400ad87ef936ea3dd8c7741</md5>
+</png>
+</blahtex>

images/latex/b19e514b37a4c733aa5682bd1345f0b6.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>9</height>
+<depth>5</depth>
+<md5>85c94d02a95009b5fd9410bf973ab842</md5>
+</png>
+</blahtex>

images/latex/d4842af15797c5ffc441da3e797dab58.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>16</height>
+<depth>5</depth>
+<md5>36588215fd5810c17501bb1214777fc8</md5>
+</png>
+</blahtex>

images/latex/debe5830087379768cc854cb2bbe3d0c.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>9</height>
+<depth>5</depth>
+<md5>12e861a5f85f27422a1a536ee1247b43</md5>
+</png>
+</blahtex>

images/latex/fd4f6a80d2590534d47f7dc2b000b951.txt

@@ -0,0 +1,7 @@
+<blahtex>
+<png>
+<height>18</height>
+<depth>10</depth>
+<md5>393c4ba7d359c057e09ed82a89884954</md5>
+</png>
+</blahtex>