# rmoorman/gwern.net forked from briansniffen/gwern.net

Fetching contributors…
Cannot retrieve contributors at this time
631 lines (419 sloc) 104 KB
 --- description: My betting track record ... Everything old is new again. Wikipedia is the collaboration of amateur gentlemen, writ in countless epistolary IRC or email or talk page messages. And the American public's [untrammeled betting on elections](http://www.unc.edu/~cigar/papers/PoP_submit4.pdf) and victories has been reborn as [prediction markets](!Wikipedia). # Prediction markets Wikipedia admirably summarizes the basic idea: > "Prediction markets...are speculative markets created for the purpose of making predictions. Assets are created whose final cash value is tied to a particular event (e.g., will the next US president be a Republican) or parameter (e.g., total sales next quarter). The *current market prices can be interpreted as predictions of the probability of the event or the [expected value](!Wikipedia)* of the parameter[^interpretation]. Prediction markets are thus structured as betting exchanges, without any risk for the bookmaker." [^interpretation]: As is true of every short description, this is a little over-simplified. People are risk-averse and fundamentally uncertain, so their beliefs about the true probability won't directly translate into the percentage/price they will buy at, and one can't even average out and say 'this is what the market believes the probability is'. See economist Rajiv Sethi's ["On the Interpretation of Prediction Market Data"](http://rajivsethi.blogspot.com/2011/03/on-interpretation-of-prediction-market.html) & ["From Order Books to Belief Distributions"](http://rajivsethi.blogspot.com/2011/03/from-order-books-to-belief.html); for more rigor, see Wolfers & Zitzewitz's paper, ["Interpreting Prediction Market Prices as Probabilities"](http://bpp.wharton.upenn.edu/jwolfers/Papers/InterpretingPredictionMarketPrices.pdf) Emphasis is added on the most important characteristic of a prediction market, the way in which it differs from regular stock markets. The idea is that by tracking accuracy - punishing ignorance & rewarding knowledge in equal measure - a prediction market can elicit one's *true* beliefs, and avoid the failure mode of predictions as pundit's bloating or wishful thinking or signaling alignment: > "The usual touchstone of whether what someone asserts is mere persuasion or at least a subjective conviction, i.e., firm belief, is betting. Often someone pronounces his propositions with such confident and inflexible defiance that he seems to have entirely laid aside all concern for error. A bet disconcerts him. Sometimes he reveals that he is persuaded enough for one ducat but not for ten. For he would happily bet one, but at 10 he suddenly becomes aware of what he had not previously noticed, namely that it is quite possible that he has erred."^[[Immanuel Kant](!Wikipedia), _Critique of Pure Reason_ (A824/B852)] ### Events, not dividends or sales Imagine a prediction market in which every day the administrator sells off pairs of shares (he doesn't want to risk paying out more than he received) for $1 a share, and all the shares say either heads or tails. Then he flips a coin and gives everyone with a 'right' share$2. Obviously if people bid up heads to $5, this is crazy and irrational - even if heads wins today, one would still lose. Similarly for any amount greater than$2. But $2 is also crazy: the only way this share price doesn't lose money is if heads is 100% guarantee. Of course, it isn't. It is quite precisely guaranteed to not be the case - 50% not the case. Anything above 50% is going to lose in the long run. A smart investor could come into this market, and blindly buy any share whatsoever that was less than$1; they would make money. If their shares were even 99¢, then about half would turn into $2 and half into 0... This is all elementary and obvious, and its how we can convince ourselves that market prices can indeed be interpreted as predictions of expected value. But that's only because the odds are known in advance! We specified it was a fair coin. If the odds of the event were not known, then things would be much more interesting. No one bets on a coin flip: we bet on whether John is bluffing. Real prediction markets famously prefer to make the subject of a share a topic like the party of the victor of the 2008 American presidential elections; a topic with a relatively clear outcome (barring the occasional George W. Bush or coin landing on its edge) and of considerable interest to many. Interest, I mean, not merely for speculating on, but possibly of real world importance. Advocates for prediction markets as tools, such as [Robin Hanson](!Wikipedia), tirelessly remind us of the possible benefits in 'aggregating information'. A prediction market rewards clear thinking and insider information, but they focus on topics it'd be difficult to clearly bet for or against on regular financial markets. Yes, if I thought the financial markets were undervaluing green power stocks because they were too weighing Senator John McCain's presidential candidacy too heavily, then I could do something like short those stocks. But suppose that's all I know about the green power stocks and the financial markets? It'd be madness to go and trade on that belief alone. I'd be exposing myself to countless risks, countless ways for the price of green stocks to be unconnected to McCain's odds, countless intermediaries, countless other relations of green stocks which may cancel out my correct appraisal of one factor. Certainly in the long run, weakly related factors will have exactly the effect they deserve to have. But this is a long run in which the investor is quite dead. Prediction markets offer a way to cut through all the confounding effects of proxies, and bet directly and precisely on that bit of information. If I believe Senator Barack Obama has been unduly discounted, then I can directly buy shares in him instead of casting about for some class of stocks that might be correlated with him - which is a formidable task in and of itself; perhaps oil stocks will rise because Obama's platform includes withdrawal from Iraq which render the Middle East less stable, or perhaps green stocks will rise for similar reasons, or perhaps they'll all fall because people think he'll be incompetent, or perhaps optimism over a historic election of a half-black man and enthusiasm over his plans will lift all boats... One will never get a faithful summation of all the information about Obama scattered among hundreds or thousands of traders if one places multiple difficult barriers in front of a trader who wishes to turn his superior knowledge or analysis into money. Or here's another example: many of the early uses of prediction markets have been inside corporations, betting on metrics like quarterly sales. Now, all of those metrics are important and will in the long run affect stock prices or dividends. But what employee working down in the R&D department is going to say 'People are too optimistic about next year's sales, the prototypes just aren't working as well as they would need to' and go short the company's stock? No one, of course. A small difference in their assessment from everyone else's is unlikely to make a noticeable price difference, even if the transaction costs of shorting didn't bar it. And yet, the company wants to know what this employee knows. ## How much to bet There's something of an [efficient market](!Wikipedia) issue with prediction markets, specifically a [no-trade theorem](!Wikipedia). Given that unlike the regular stock market, prediction markets are necessarily [zero-sum](!Wikipedia)^[Or negative-sum, when you consider the costs of running the prediction market and the various fees that might be assessed on participants.], and so lots of traders are going to be net losers. If you don't have any particular reason to think you are one of the wolves canny enough to make money off the sheep, then you're one of the sheep, and why trade at all? (I understand poker players have a saying - if you can't spot the fish at the table, you're the fish.) So, the bad-and-self-aware won't participate. If you are trading in a prediction market, you are either good-and-aware or good-and-ignorant or bad-but-ignorant. Ironically, the latter two can't tell whether they are the first group or not. It reminds me of the [smoking lesion](http://wiki.lesswrong.com/wiki/Smoking_lesion) puzzle or ["King Solomon's problem"](http://lesswrong.com/lw/3pf/does_evidential_decision_theory_really_fail/) in [decision theory](!Wikipedia): you may have many [cognitive bias](!Wikipedia)es such as the [overconfidence effect](!Wikipedia) (the lesion), and they may cause you to fail or succeed on the prediction market (get cancer or not) and also want to participate therein. What do you do? Best of course is to test for the lesion directly - to test whether our predictions are [calibrated](!Wikipedia "Calibration (statistics)")^[See also the [LessWrong articles](http://lesswrong.com/r/lesswrong/tag/calibration/) on the topic.], whether events we confidently predict at 0% do in fact never happen and so on. If we manage to overcome our biases, we can give [calibrated probability assessment](!Wikipedia)s. We can do this sort of testing with the relevant biases - just knowing about them and introspecting about one's predictions can improve them. Coming up with the precise reasons one is making a prediction can also help with the [hindsight bias](!Wikipedia)^[See ["Eliminating the hindsight bias"](/docs/1988-arkes.pdf), Arkes 1988.] or the temptation to [falsify your memories](http://www.wired.com/wiredscience/2011/10/how-friends-ruin-memory-the-social-conformity-effect/) [based on social feedback](http://www.weizmann.ac.il/neurobiology/labs/dudai/uploads/files/Science-2011-Edelson-108-11.pdf), all of which is important to figuring out how well you will do in the future. We can quickly test calibration using our partial ignorance about many factual questions, eg. the links in ["Test Your Calibration!"](http://lesswrong.com/lw/1f8/test_your_calibration/). [My recent practice](http://predictionbook.com/users/gwern) with thousands of real-world predictions on [PredictionBook.com](http://predictionbook.com/) has surely helped my calibration. So, how much better are you than your competing traders? What is your edge? This, believe it or not, is pretty much all you need to know to know how much to bet on any contract. The exact fraction of your portfolio to bet given a particular edge is defined by the [Kelly criterion](!Wikipedia) which gives the greatest possible expected utility of [your growth rate](http://r6.ca/blog/20070816T193609Z.html). (But you need to be psychologically tough[^wp] to use it lest you begin to [play irrationally](!Wikipedia "Tilt (poker)"): it's not a [risk averse](!Wikipedia) strategy.) [^wp]: Wikipedia's [criticism section](!Wikipedia "Kelly criterion#Reasons to bet less than Kelly") remarks that "Kelly betting leads to highly volatile short-term outcomes which many people find unpleasant, even if they believe they will do well in the end." The formula is:$x = \frac{o \times e - (1 - e)}{o}$- _o_ = odds - _e_ = your edge - _x_ = the fraction to invest To quote the Wikipedia explanation: > "As an example, if a gamble has a 60% chance of winning ($e = 0.60$), but the gambler receives 1-to-1 odds on a winning bet ($o = 1$), then the gambler should bet 20% of the bankroll at each opportunity ($x = 0.20$), in order to maximize the long-run growth rate of the bankroll." So, suppose the President's re-election contract was floating at 50%, but based on his performance and past incumbent re-election rates, you decide the true odds are 60%; you can buy the contract at 50% and if you hold until the election and are right, you get back double your money, so the odds are 1:1. The filled-in equation looks like 1.$x = \frac{1 \times 0.6 - (1 - 0.6)}{1}$2.$x = 1 \times 0.6 - (1 - 0.6)$3.$x = 0.6 - (1 - 0.6)$4.$x = 0.6 - 0.4$5.$x = 0.2$Hence, you ought to put 20% of your portfolio into buying the President's contract. (If we assume that all bets are double-or-nothing, Wikipedia tells us it simplifies to$x = (2 \times p) - 1$, which in this example would be$(2 \times 0.6) - 1$=$1.2 - 1$=$0.2$. But usually our contracts in prediction markets won't be that simple, so the simplification isn't very useful here.) It's not too hard to apply this to more complex situations. Suppose the president were at, say, 10% but you are convinced the unfortunate equine sex scandal will soon be forgotten and the electorate will properly appreciate _el Presidente_ for winning World War III by making his true re-election odds 80%. You can buy in at 10% and you resolve to sell out at 80%, for a reward of 70% or 7 times your initial stake (7:1). And we'll again say you're right 60% of the time. So your Kelly criterion looks like: 1.$x = \frac{7 \times 0.6 - (1 - 0.6)}{7}$2.$x = \frac{(7 \times 0.6) - 0.4}{7}$3.$x = \frac{4.2 - 0.4}{7}$4.$x = \frac{3.8}{7}$5.$x = 0.54$Wow! We're supposed to bet more than half our portfolio despite knowing we'll lose the bet 40% of the time? Well, yes. With an upside like 7x, we can lose several bets in a row and eventually make up our loss. And if we win the first time, we just won huge. It goes both ways, of course. If we have a market/true-odds of 80%/90% and we do the same thing, we have a return of 12.5% (9/8) rather than 100%, and for that little return ought to risk only: 1.$x = \frac{0.875 \times 0.6 - (1 - 0.6)}{0.875}$2.$x = \frac{0.875 - 0.4}{0.875}$3.$x = \frac{0.125}{0.875}$4.$x = 0.142856$As one would expect, with a smaller reward but equal risk compared to our first example, the KC recommends a smaller than 0.2 fraction invested. If one doesn't enjoy calculating the KC, one could always write a program to do so; Russell O'Connor has a nice Haskell blog post on ["Implementing the Kelly Criterion"](http://r6.ca/blog/20070820T175938Z.html) (who also has an interesting post on the [KC and the lottery](http://r6.ca/blog/20090522T015739Z.html).) ## Specific markets So once we are interested in prediction markets and would like to try them out, we need to pick one. There are several. I generally ignore the 'play money' markets like the [Hollywood Stock Exchange](!Wikipedia), despite their similar levels of accuracy to the real money markets; I just have a prejudice that if I make a killing, then I ought to have a real reward like a nice steak dinner and not just increment some bits on a computer. The primary markets to consider are: * [Betfair](!Wikipedia) and [BETDAQ](!Wikipedia) are probably the 2 largest prediction markets, but unfortunately, it is difficult for Americans to make use of them - Betfair bans them outright. * [Intrade](!Wikipedia) is another European prediction market, similar to Betfair and BETDAQ, but it does not go out of its way to bar Americans, and thus is likely the most popular market in the United States. (Its sister site [TradeSports](!Wikipedia) was sport-only, and is now defunct.) * [HedgeStreet](!Wikipedia) is some sort of hybrid of derivatives and predictions. I know little about it. * [The Iowa Electronic Markets](!Wikipedia) (IEM) is an old prediction markets, and one of the better covered in American press. It's a research prediction market, so it handles only small quantities of money and trades and has only a few traders[^size]. Accounts max out at$500, a major factor in limiting the depth & liquidity of its markets. [^size]: On 27 January 2008, the IEM sent out an email which accidentally listed all recipients in the CC; the listed emails totaled 292 emails. Given that many of these traders (like myself) are surely inactive or infrequent, and only a fraction will be active at a given time, this means the 10 or so markets are thinly inhabited. I didn't want to wager too much money on what was only a lark, and the IEM has the favorable distinction of being clearly legal in the USA. So I chose them. ### IEM In 2003, I sent in a check for $20. A given market's contracts in the IEM are supposed to sum to$1, so $20 would let me buy around 40 shares - enough to play around with. #### My IEM trading ##### 2004 > "Like all weak men he laid an exaggerated stress on not changing one's mind."^[[William Somerset Maugham](!Wikipedia), writer (1874-1965)] Prediction markets are known to have a number of biases. Some of these biases are shared with other betting exchanges; horse-racing is plagued with a 'long-shot favoritism' just like prediction markets are. (An example of long-shot favoritism would be Intrade and IEM shares for libertarian Ron Paul winning the 2008 Republican nomination trading at ludicrous valuations like 10¢, or Al Gore - who wasn't even running - for the Democratic nomination at 5¢.) The financial structure of markets also seems to make shorting of such low-value (but still over-valued) shares more difficult. They can be manipulated, consciously or unconsciously, due to not being very good markets (["They are thin, trading volumes are anemic, and the dollar amounts at risk are pitifully small"](http://www.wired.com/techbiz/it/magazine/16-06/st_essay)), and that's where they aren't reflecting the prejudices of their users (one can't help but suspect Ron Paul shares were overpriced because he has so many fans among techies). I began experimenting with some small trades on IEM's Federal Reserve interest rate market; I had a theory that there was a 'favorites bias' (the inverse of long-shot favoritism, where traders buck the conventional wisdom despite it being more correct). I simply based my trades on what I read in the _New York Times_. It worked fairly well. In 2005, I also dabbled in the markets on Microsoft and Apple share prices, but I didn't find any values I liked. 2004 was, of course, a presidential election year. I couldn't resist, and traded heavily. I avoided Democratic nominations, reasoning that I was too ignorant that year - which was true, I did not expect John Kerry to eventually win the nomination - and focused on the party-victory market. The traders there were far too optimistic about a Democratic victory; I knew 'Bush is a war-time president' (in addition to the incumbency!) as people said, and that this matter a lot to the half of the electorate that voted for him in 2000. Giving him a re-election probability of under 40% was too foolish for words. I did well on these trades, and then in October, I closed out all my trades, sold my Republican/Bush shares, and bought Kerry. I thought the debates had gone well for Kerry and was confident the Swift Boating wouldn't do much in the end, and certainly couldn't compensate for the albatross of Iraq. As you know, I was quite wrong in this strategy. Bush did win, and won more than in 2000. And I lost$5-10. (Between a quarter and a half my initial capital. Ouch! I was glad I hadn't invested some more substantial sum like $200.) I had profited early on from people who had confused what they *wanted* to happen with what *would*, but then I had succumbed to the same thing. Yes, everyone around me (I live in a liberal state) was sure Kerry would win, but that's no excuse for starting off with a correct assessment and then choosing a false one. It was a valuable lesson for me; this experience makes me sometimes wonder whether 'personal' prediction markets, if you will, could be a useful tool. ##### 2005/2006 In 2005 & 2006, I did minimal interesting trading. I largely continued my earlier strategies in the interest rate markets. Slowly, I made up for my failures in 2004. ##### 2007 In 2007, the presidential markets started back up! I surveyed the markets and the political field with great excitement. As anyone remembers, it was the most interesting election in a very long, with such memorable characters (Hillary Clinton, Ron Paul, Barack Obama, John McCain, Sarah Palin) and unexpected twists. ###### The Republicans As in 2004, the odds of an ultimate Republican victory were far too low - hovering in the range of 30-40%. This is obviously wrong on purely historical considerations (Democrats don't win the presidency *that* often), and seems particularly wrong when we consider that George W. Bush won in 2004. Anyone arguing that GWB poisoned the well for a succeeding Republican administration faces the difficult task of explaining (at least) 2 things: 1. How association with GWB would be so damaging when GWB himself was re-elected in 2004 with a larger percentage of votes than 2000 2. How association with GWB policies like Iraq would be so damaging when the daily security situation in Iraq has clearly improved since 2004. 3. And in general: how a fresh Republican face (with the same old policies) could do any worse than GWB did, given that he will possess all the benefits of GWB's policies and none of the personal animus against GWB. The key to Republican betting was figuring out who was hopeless, and work from there by essentially short selling them. As time passed, one could sharpen one's bets and begin betting for a candidate rather than against. My list ultimately looked like this: 1. **Ron Paul** was so obviously not going to win. He appealed to only a small minority of the Republican party, had views idiosyncratic where they weren't offensive, and wanted to destroy important Republican constituencies. If the Internets were America, perhaps he could've won. 2. **Rudy Giuliani** was another easy candidate to bet against. He had multiple strikes: he was far too skeevy, questionable ethically (the investigations of Bernard Kerik were well underway at this point), had made himself a parody, had few qualifications, and a campaign strategy that was as ruinous as it was perplexing. He was unacceptable culturally, what with his divorces, loose living, humorous cross-dressing, and New York ways. He would not play well in Peoria. 3. **Jack Thompson** was undone by being a bad version of Reagan. He didn't campaign nearly as industriously as he needed to. The death knell, as far as I was concerned, was when national publications began mentioning the 'lazy like a fox' joke as an old joke. No special appeal, no special resources, no conventional ability... 4. **Mitt Romney** had 2 problems: he was slick and seemed inauthentic, and people focused too much on his being Mormon and Massachusetts governorship (a position that would've been a great aid - if it hadn't been in that disgustingly liberal state). I was less confident about striking him off, but I decided his odds of 20% or so were too generous. 5. **Mike Huckabee** struck me as not having the resources to make it to the nomination. I was even less sure about this one than Mitt, but I lucked out - the supporters of Huckabee began infighting with Romney supporters. This didn't leave very many candidates for consideration. By this process of elimination, I was in fact left with only John McCain as a serious Republican contender. If you remember the early days, this was in fact a very strange result to reach: John McCain appeared tired, a beaten man from 2004 making one last _pro forma_ try, his campaign inept and riven by infighting, and he was just in general - old, old, old. But hey, his shares were trading in the 5-15% range. They were the best bargain going in the market. I held them for a long time and ultimately would sell them at 94-99¢ for a roughly 900% gain. (I sold them instead of waiting for the Republican convention because I was forgoing minimal gains, and I was concerned by reports on his health.) ###### The Democrats A similar process obtained for the Democrats. A certain dislike of Hillary Clinton led me to think that her status as the heir presumptive (reflected in share proces) would be damaged at some point. All of the other candidates struck me as flakes and hopeless causes, with the exception of John Edwards and Barack Obama. I eventually ruled out John Edwards as having no compelling characteristics and smacking of phoniness (much like Romney). I was never tempted to change my mind on him, and the adultery and hair flaps turned out to be waiting in the wings for him. So I could get rid of Edwards as a choice. Is it any surprise I lighted on Obama? He had impressed me (and just about everyone else) with his 2004 convention speech, his campaign seemed quite competent and well-funded, the media clearly loved him, and so on. Best of all, his shares were relatively low (30-40%) and I had money left after the Republicans. So I bought Obama and sold Clinton. I eventually sold out of Obama at the quite respectable 78¢. ### Summing up By the end of the election, I had made a killing on my Obama and McCain shares. My account balance stood at$38; so over the 3 or 4 years of trading I had nearly doubled my investment. $18 is perhaps enough for a steak dinner. Further, I had learned a valuable lesson in 2004 about my own political biases and irrationality, and had earned the right in 2008 to be smug about foreseeing a McCain and Obama match-up when the majority of pundits were trying to figure out whether Hillary would be running against Huckabee or Romney. And finally, I've concluded that my few observations aside, prediction markets are pretty accurate. I often use them to sanity-check myself by asking 'If I disagree, what special knowledge do I have?' Often I have none. I'm currently out of the IEM, but I had a good experience, and I've come out a believer. One day I'd like to try out a more substantial and varied market, like Intrade. #### IEM logs The following is an edited IEM trading history for me, removing many limit positions and other expired or canceled trades: Order date O.time Market Contract Order # Unit price Expiry Resolution type R.# R.price ---------- ------ ------ -------- ----- ----- ---------- ------ ----------------- --- ------- 12/29/04 20:16:23 FedPolicyB FRsame0205 Purchase 20 0.048 Traded 10 0.048 12/29/04 20:17:26 FedPolicyB FRup0205 Purchase 2 0.956 Traded 2 0.956 12/29/04 20:17:46 FedPolicyB FRsame0205 Purchase 10 0.049 Traded 10 0.049 02/12/05 17:48:51 Comp-Ret AAPL-05b Bid 5 0.96 3/14/2005 11:59PM Cancel-Manager 02/13/05 16:43:33 Comp-Ret AAPL-05b Bid 7 0.982 3/15/2005 11:59PM Traded 7 0.98 02/21/05 10:03:45 FedPolicyB FRsame0505 Bid 12 0.053 4/23/2005 11:59PM Traded 12 0.053 02/21/05 10:04:35 FedPolicyB FRup0305 Bid 7 0.988 3/23/2005 11:59PM Traded 7 0.988 02/21/05 10:04:35 FedPolicyB FRup0305 Traded 3 0.007 3/3/2005 9:23AM 02/21/05 10:06:59 FedPolicyB FRsame0305 Bid 6 0.007 3/23/2005 11:59PM Traded 3 0.007 02/21/05 10:07:51 Comp-Ret AAPL-05b Bid 5 0.998 3/23/2005 11:59PM Cancel-Manager 02/21/05 10:07:51 Comp-Ret AAPL-05b Traded 4 0.889 2/28/2005 8:56:AM 02/26/05 10:14:08 Comp-Ret AAPL-05c Bid 5 0.889 3/28/2005 11:59PM Traded 1 0.889 02/26/05 10:14:30 Comp-Ret MSFT-05c Bid 1 0.889 3/28/2005 11:59PM Traded 1 0.889 02/26/05 10:15:43 MSFT-Price ? Traded 1 0.4 3/5/2005 10:39PM 03/05/05 12:51:45 MSFT-Price MS025-05cL Bid 5 0.4 4/7/2005 11:59PM Traded 4 0.4 03/05/05 12:53:27 Comp-Ret AAPL-05c Ask 4 0.95 7/7/2005 11:59PM Cancel-Manager 03/05/05 12:53:56 Comp-Ret MSFT-05c Ask 1 0.5 7/7/2005 11:59PM Cancel-Manager 03/05/05 12:54:38 FedPolicyB FRsame0505 Ask 12 0.7 9/7/2005 11:59PM Cancel-Manager 03/05/05 12:55:07 FedPolicyB FRsame0305 Ask 6 0.2 9/7/2005 11:59PM Cancel-Manager 03/05/05 12:55:33 FedPolicyB FRup0305 Ask 6 0.998 6/7/2005 11:59PM Traded 6 0.998 03/05/05 12:55:33 FedPolicyB ? Traded 2 0.803 9/16/2005 3:37PM 03/05/05 12:55:33 FedPolicyB ? Traded 5 0.803 9/16/2005 3:34PM 09/16/05 14:38:57 FedPolicyB FRup0905 Bid 12 0.803 9/20/2005 11:59PM Traded 5 0.803 09/16/05 14:39:34 FedPolicyB FRsame0905 Bid 6 0.17 9/22/2005 11:59PM Traded 6 0.17 09/28/05 23:49:01 FedPolicyB FRsame1105 Bid 15 0.066 10/1/2005 11:59PM Traded 15 0.066 10/07/05 12:28:48 FedPolicyB FRsame1105 Ask 15 0.07 10/9/2006 11:59PM Cancel-Manager 10/07/05 12:29:23 FedPolicyB FRup1105 Bid 2 0.95 10/9/2006 11:59PM Cancel-Manager 10/10/05 14:54:45 FedPolicyB FRup1105 Bid 3 0.97 10/12/2005 11:59PM Traded 3 0.97 12/09/05 15:02:02 FedPolicyB FRup1205 Bid 15 0.995 12/12/2005 11:59PM Traded 15 0.995 12/09/05 15:02:20 FedPolicyB FRsame1205 Bid 10 0.002 12/12/2005 11:59PM Traded 10 0.002 12/09/05 15:02:43 FedPolicyB FRdown1205 Bid 2 0.001 12/13/2005 11:59PM Traded 2 0.001 12/09/05 15:02:43 FedPolicyB ? Traded 2 0.719 6/2/2006 8:41:40AM 12/09/05 15:02:43 FedPolicyB ? Traded 10 0.719 6/2/2006 8:39:46AM 05/31/06 21:28:25 FedPolicyB FRup0606 Bid 22 0.719 6/6/2006 11:59PM Traded 10 0.719 08/07/06 21:19:08 FedPolicyB FRup0806 Bid 20 0.27 8/22/2006 11:59PM Traded 20 0.27 08/07/06 21:19:08 FedPolicyB ? Traded 7 0.608 8/8/2006 1:13:17PM 08/07/06 21:19:47 FedPolicyB FRsame0806 Bid 10 0.608 8/9/2006 11:59PM Traded 3 0.608 08/07/06 21:19:47 FedPolicyB ? Traded 7 0.7 8/7/2006 9:52:43PM 08/07/06 21:20:29 FedPolicyB FRsame0906 Bid 10 0.7 8/9/2006 11:59PM Traded 3 0.7 08/07/06 21:20:54 FedPolicyB FRdown0906 Bid 10 0.006 8/9/2006 11:59PM Traded 10 0.006 08/07/06 21:23:04 PRES08-WTA DEM08-WTA Bid 15 0.5 12/23/2006 11:59PM Traded 15 0.5 08/28/06 09:20:10 PRES08-VS UREP08-VS Bid 10 0.48 12/30/2006 11:59PM Traded 10 0.48 08/28/06 09:20:10 PRES08-VS ? Traded 3 0.5 9/19/2006 10:24AM 08/28/06 09:20:26 PRES08-VS UDEM08-VS Bid 10 0.5 12/30/2006 11:59PM Traded 1 0.5 06/01/07 20:00:20 PRES08-WTA DEM08-WTA Ask 10 0.66 9/3/2007 11:59PM Traded 10 0.66 06/01/07 20:01:24 PRES08-WTA DEM08-WTA Ask 5 0.7 6/3/2008 11:59PM Traded 5 0.7 06/01/07 20:02:21 PRES08-WTA REP08-WTA Bid 10 0.33 9/3/2007 11:59PM Traded 10 0.33 06/01/07 20:04:26 RConv08 ROMN-NOM Bid 5 0.2 7/3/2007 11:59PM Traded 5 0.2 06/01/07 20:05:33 DConv08 OBAM-NOM Purchase 5 0.322 6/1/2007 8:05:33PM Traded 1 0.322 06/06/07 23:41:39 DConv08 DConv08 Buy-bundle 3 1 Traded 3 1 06/06/07 23:42:20 DConv08 EDWA-NOM Ask 3 0.1 6/8/2008 11:59PM Traded 3 0.1 06/06/07 23:42:46 DConv08 DROF-NOM Ask 3 0.13 6/8/2008 11:59PM Traded 3 0.13 06/06/07 23:44:29 RConv08 RConv08 Buy-bundle 3 1 Traded 3 1 06/06/07 23:45:12 RConv08 GIUL-NOM Ask 3 0.21 9/20/2007 11:59PM Traded 3 0.21 06/06/07 23:45:34 RConv08 MCCA-NOM Ask 3 0.15 9/20/2007 11:59PM Traded 3 0.15 06/06/07 23:46:55 PRES08-VS UDEM08-VS Ask 4 0.56 6/8/2008 11:59PM Traded 4 0.56 12/11/07 16:08:57 RConv08 HUCK-NOM Ask 3 0.22 12/13/2007 11:59PM Traded 3 0.22 12/11/07 16:10:08 RConv08 ROMN-NOM Ask 4 0.25 12/13/2007 11:59PM Traded 4 0.25 12/11/07 16:14:22 RConv08 RROF-NOM Ask 3 0.03 12/13/2007 11:59PM Traded 3 0.03 12/11/07 16:16:12 RConv08 MCCA-NOM Bid 5 0.1 12/13/2008 11:59PM Traded 5 0.1 12/11/07 16:16:57 RConv08 RConv08 Buy-bundle 5 1 12/11/2007 4:16PM Traded 5 1 12/11/07 16:17:39 RConv08 GIUL-NOM Sell 5 0.375 12/11/2007 4:17PM Traded 5 0.375 12/11/07 16:18:01 RConv08 HUCK-NOM Sell 5 0.207 12/11/2007 4:18PM Traded 5 0.207 12/11/07 16:18:10 RConv08 MCCA-NOM Sell 5 0.108 12/11/2007 4:18PM Traded 5 0.108 12/11/07 16:18:22 RConv08 ROMN-NOM Sell 5 0.241 12/11/2007 4:18PM Traded 5 0.241 12/11/07 16:18:33 RConv08 THOMF-NOM Sell 5 0.04 12/11/2007 4:18PM Traded 5 0.04 12/11/07 16:18:46 RConv08 RROF-NOM Sell 5 0.02 12/11/2007 4:18PM Traded 5 0.02 12/11/07 16:19:03 RConv08 ROMN-NOM Sell 4 0.24 12/11/2007 4:19PM Traded 4 0.24 12/11/07 16:20:28 DConv08 DConv08 Buy-bundle 10 1 12/11/2007 4:20PM Traded 10 1 12/11/07 16:20:51 DConv08 DROF-NOM Ask 10 0.03 12/13/2008 11:59PM Traded 10 0.03 12/11/07 16:20:51 DConv08 ? Traded 5 0.09 12/19/2007 3:34PM 12/11/07 16:21:31 DConv08 EDWA-NOM Ask 10 0.09 12/13/2008 11:59PM Traded 5 0.09 12/11/07 16:21:31 DConv08 ? Traded 1 0.58 12/11/2007 9:40PM 12/11/07 16:21:31 DConv08 ? Traded 9 0.58 12/11/2007 9:40PM 12/11/07 16:25:21 DConv08 CLIN-NOM Ask 13 0.58 12/13/2008 11:59PM Traded 3 0.58 12/11/07 16:26:08 DConv08 OBAM-NOM Ask 14 0.45 12/13/2008 11:59PM Traded 14 0.45 12/11/07 16:27:05 DConv08 OBAM-NOM Bid 5 0.3 12/31/2007 11:59PM Traded 5 0.3 12/11/07 16:28:51 FedPolicyB FRsame0108 Bid 3 0.31 12/31/2007 11:59PM Traded 3 0.31 02/05/08 22:41:41 RConv08 THOMF-NOM Sell 3 0.002 2/5/2008 10:41PM Traded 3 0.002 02/05/08 22:47:46 DConv08 OBAM-NOM Bid 10 0.42 2/7/2008 11:59PM Traded 10 0.42 02/05/08 22:48:09 DConv08 OBAM-NOM Bid 5 0.43 2/7/2008 11:59PM Traded 5 0.425 02/07/08 14:46:34 DConv08 DConv08 Buy-bundle 5 1 2/7/2008 2:46PM Traded 5 1 02/07/08 14:47:21 DConv08 EDWA-NOM Sell 5 0.002 2/7/2008 2:47PM Traded 5 0.002 02/07/08 14:47:34 DConv08 DROF-NOM Sell 5 0.006 2/7/2008 2:47PM Traded 5 0.006 02/07/08 14:47:54 DConv08 OBAM-NOM Ask 15 0.6 2/9/2008 11:59PM Traded 15 0.6 02/07/08 15:11:51 PRES08-WTA REP08-WTA Ask 10 0.51 2/9/2009 11:59PM Traded 10 0.51 02/07/08 15:13:24 RConv08 RConv08 Buy-bundle 4 1 2/7/2008 3:13PM Traded 4 1 02/07/08 15:13:42 RConv08 GIUL-NOM Sell 4 0.001 2/7/2008 3:13PM Traded 4 0.001 02/07/08 15:13:49 RConv08 HUCK-NOM Sell 4 0.017 2/7/2008 3:13PM Traded 4 0.017 02/07/08 15:13:58 RConv08 ROMN-NOM Purchase 4 0.005 2/7/2008 3:13PM Traded 4 0.005 02/07/08 15:14:06 RConv08 THOMF-NOM Sell 4 0.003 2/7/2008 3:14PM Traded 4 0.003 02/07/08 15:14:14 RConv08 RROF-NOM Sell 4 0.009 2/7/2008 3:14PM Traded 4 0.009 02/07/08 15:14:29 RConv08 RConv08 Buy-bundle 1 1 2/7/2008 3:14PM Traded 1 1 02/07/08 15:14:44 RConv08 ROMN-NOM Sell 9 0.002 2/7/2008 3:14PM Traded 9 0.002 02/07/08 15:14:54 RConv08 GIUL-NOM Sell 1 0.001 2/7/2008 3:14PM Traded 1 0.001 02/07/08 15:15:02 RConv08 HUCK-NOM Sell 1 0.017 2/7/2008 3:15PM Traded 1 0.017 02/07/08 15:15:10 RConv08 THOMF-NOM Purchase 1 0.006 2/7/2008 3:15PM Traded 1 0.006 02/07/08 15:15:22 RConv08 RROF-NOM Sell 1 0.009 2/7/2008 3:15PM Traded 1 0.009 02/07/08 15:15:30 RConv08 THOMF-NOM Sell 2 0.003 2/7/2008 3:15PM Traded 2 0.003 04/06/08 13:52:28 DConv08 CLIN-NOM Ask 5 0.15 4/8/2008 11:59PM Traded 4 0.15 04/06/08 13:52:51 DConv08 CLIN-NOM Ask 1 0.14 4/8/2008 11:59PM Traded 1 0.14 04/06/08 13:52:51 DConv08 ? Traded 3 0.79 4/10/2008 6:45PM 04/06/08 13:55:08 DConv08 OBAM-NOM Bid 5 0.79 4/8/2009 11:59PM Traded 2 0.79 04/06/08 13:59:43 RConv08 RConv08 Buy-bundle 10 1 4/6/2008 1:59PM Traded 10 1 04/06/08 14:00:27 RConv08 GIUL-NOM Sell 10 0.004 4/6/2008 2:00PM Traded 10 0.004 04/06/08 14:00:41 RConv08 HUCK-NOM Sell 10 0.007 4/6/2008 2:00PM Traded 10 0.007 04/06/08 14:00:54 RConv08 ROMN-NOM Sell 10 0.01 4/6/2008 2:00PM Traded 10 0.01 04/06/08 14:01:07 RConv08 THOMF-NOM Sell 10 0.004 4/6/2008 2:01PM Traded 10 0.004 04/06/08 14:01:20 RConv08 RROF-NOM Sell 10 0.025 4/6/2008 2:01PM Traded 10 0.025 04/14/08 13:51:41 DConv08 OBAM-NOM Bid 3 0.78 4/16/2008 11:59PM Traded 3 0.78 05/03/08 12:06:18 DConv08 OBAM-NOM Ask 18 0.78 5/5/2008 11:59PM Traded 18 0.78 05/05/08 20:21:52 RConv08 MCCA-NOM Ask 20 0.94 5/7/2008 11:59PM Traded 20 0.94 05/20/08 15:44:10 PRES08-VS UREP08-VS Sell 10 0.483 5/20/2008 3:44PM Traded 1 0.483 05/20/08 15:45:29 PRES08-VS UREP08-VS Sell 10 0.482 5/20/2008 3:45PM Traded 9 0.482 ### Intrade In 2010, I signed up for Intrade since the IEM was too small and had too few contracts to maintain my interest. #### Payment Paying Intrade, as a foreign company in Ireland, was a little tricky. I first looked into paying via debit card, but Intrade demanded considerable documentation, so I abandoned that approach. I then tried a bank transfer since that would be quick; but my credit union failed me and said Intrade had not provided enough information (which seemed unlikely to me, and Intrade's customer service agreed) - and even if they had, they would charge me$10! Finally, I decide to just snail-mail them a check. I was pleasantly surprised to see that postage to Ireland was ~$1, and it made it there without a problem. But very slowly: perhaps 15 days or so before the check finally cleared and my initial$200 was deposited. #### My Intrade trading Intrade has a considerably less usable system than IEM. In IEM, selling short is very easy: you purchase a pair of contracts (yes/no) which sum to $0, and then you sell off the opposite. If I think DEM08 is too high compared to REP08, I get 1 share of each and sell the DEM08. Intrade, on the other hand, requires you to 'sell' a share. I don't entirely understand it, but it *seems* to be equivalent. I wanted to sell short some of the more crazy probabilities such as on Japan going nuclear or the USA attacking North Korea or Iran, but it turned out that to make even small profits on them, I would have to hold them a long time and because their probabilities were so low already, Intrade was demanding large [margins](!Wikipedia "Margin (finance)") - to buy 4 or 5 shorts would lock up half my account!^[The problem is that if a contract is at 10%, and you buy 10 contracts, then if the contract actually pays off, you have to come up with 100% to pay the other people their winnings. Intrade, to guarantee them payment, will make you pay the full 10%, and then freeze the 90% in your account.] My first trade was to sell short the [Intrade contract](https://www.intrade.com/jsp/intrade/common/c_cd.jsp?conDetailID=702407&z=1285870999458) on [California Proposition 19 (2010)](!Wikipedia), which would legalize non-medical marijuana possession.I reasoned that California recently banned gay marriage at the polls, and medical marijuana is well-known as a joke (lessening the incentive to pass Prop 19), and that its true probability of passing was more like 30% - significantly below its current price. The contract would expire in just 2 months, making it even more attractive. It was at 49 when I shorted it. I put around 20% of my portfolio (or ~$40) after consulting with the [Kelly criterion](#how-much-to-bet). 2 days later, the price had increased to 53.3, and on 4 October, it had spiked all the way to 76%. I began to seriously consider how confident I was in my prediction, and whether I was faced with a choice between losing the full $40 I had invested or buying shares at 76% (to fulfill my shorting contracts) and eating the loss of ~$20. I meditated, and reasoned that there wasn't *that* much liquidity and I had found no germane information online (like a poll registering strong public support), and decided to hold onto my shares. As of 27 October, the price had plummeted all the way to 27%, and continued to bounce around the 25-35% price range. I had at the beginning decided that the true probability was in the 30% decile, and if anything, it was now *underpriced*. Given that, I was running a risk holding onto my shorts. So on 30 October, I bought 10 shares at 26%, closing out my shorts, and netting me $75.83, for a return of$25.83, or 50% over the month I held it. My second trade dipped into the highly liquid 2012 US presidential elections. The partisan contracts were trading at ~36% for the Republicans and ~73% for the [Democrats](https://www.intrade.com/jsp/intrade/common/c_cd.jsp?conDetailID=639648&z=1285871684768). I would agree that the true odds are >50% for the Democrats since presidents are usually re-elected and the Republicans have few good-looking candidates compared to Obama, who has accomplished quite a bit in office. However, I think 73% is overstated, and further, that the markets always panic during an election and squish the ratio to around 50:50. So I sold Democrat and bough Republican. (I wound up purchasing more Republican contracts than selling Democrat contracts because of the aforementioned margin issues.) I bought 5 Reps at 39, and shorted 1 Dem at 60.8. 2 days later, they had changed to 37.5 and 62.8 respectively. By 26 November 2010, it was 42 and 56.4. By 1 January 2011, Republicans was at 39.8 and Democrats at 56.8. Finally, I decided that Sarah Palin has next to no chance at the Republican nomination since she blew a major hole in her credentials by her bizarre resignation as governor, and [her shares](https://www.intrade.com/jsp/intrade/common/c_cd.jsp?conDetailID=652756&z=1285872122100) at 18% were just crazy. I shorted 10 at 18% since I thought the true odds are more like [10%](http://predictionbook.com/predictions/460). 2 days later, they had risen to 19%. By 26 November, they were still at 19%, but the odds of her [announcing a candidacy](https://www.intrade.com/jsp/intrade/common/c_cd.jsp?conDetailID=686537&z=1290800258450) had risen to 75%. I'd put the odds of her announcing a run at [~90%](http://predictionbook.com/predictions/378) (a mistake, given that she ultimately decided against running in October 2011), but I don't have any spare cash to buy contracts. I *could* sell out of the anti-nomination contracts and put that money into announcement, but I'm not sure this is a good idea - the announcement is very volatile, and I dislike eating the fees. She hasn't done too well as the Tea Party _eminence grise_, but maybe she prefers it to the hard work of a national campaign? By 1 January 2011, the nominee odds were still stuck at 18% but the announcement had fallen to 62%. The latter is dramatic enough that I'm wondering whether my 90% odds really are correct (it probably wasn't). By June, I've begun to think that Palin knows she has little chance of winning either the nomination or presidency, and is just milking the speculation for all its worth. Checking on 8 June, I see that the odds of an announcement have fallen from 62% to 33% and a nomination from 18% to 5.9% - so I would have made out very nicely on the nomination contract had I held the short, but been mauled if I had made any shorts on the announcement. I am not sure what lesson to draw from this observation; probably that I am better at assessing outcomes based on a great many people (like a nomination) than outcomes based on a single individual person's psychology (like whether to announce a run or not). ##### Cashing out In January 2011[^lw], Intrade announced a new fee structure - instead of paying a few cents per trade, one has free trading but your account is charged [$5 every month](http://www.intrade.com/jsp/intrade/help/index.jsp?page=general.html%23fees) or$60 a year (see also the [forum announcement](http://bb.intrade.com/intradeForum/posts/list/4797.page#44860)). Fees have been a problem with Intrade in the past due to the small amounts usually wagered - see for example financial journalist [Felix Salmon](!Wikipedia)'s [2008 complaints](http://www.portfolio.com/views/blogs/market-movers/2008/12/03/the-problem-with-intrade/). [^lw]: This section first appeared on [LessWrong.com](http://lesswrong.com/) as ["2011 Intrade fee changes, or, Intrade considered no longer useful for LessWrongers"](http://lesswrong.com/r/discussion/lw/3l2/2011_intrade_fee_changes_or_intrade_considered_no/) and includes some discussion. Initially, the new changes didn't seem so bad to me, but then I compared the annual cost of this fee to my trading stake, ~$200. I would have to earn a return of 30% just to cover the fee! (This is also pointed out by many in the forum thread above.) I don't trade very often since I think I'm best at spotting mispricings over the long-term (the CA Proposition 19 contract (WP) being a case in point; despite being ultimately correct, I could have been mauled by some of the spikes if I had tried only short-term trades). If this fee had been in place since I joined, I would be down by$30 or $40. I'm confident that I can earn a good return like 10 or 20%, but I can't do >30% without taking tremendous risks and wiping myself out. And more generally, assuming that this isn't raiding accounts[^raid] as a prelude to shutting down (as a number of forumers claim), Intrade is no longer useful for LessWrongers like me as it is heavily penalizing small long-term bets like the ones we are usually concerned with - bets intended to be educational or informative. It may be time to investigate other prediction markets like Betfair, or just resign ourselves to non-monetary/play-money sites like [PredictionBook.com](http://predictionbook.com/). [^raid]: When I submitted my withdrawal request for my balance, I received an email offering to instead set my account to 'inactive' status such that I could not trade but would not be charged the fee; if I wanted to trade, I would simply be charged that month's$5. I declined the offer, but I couldn't help wonder - why didn't they simply set all accounts to 'inactive' and then let people opt in to the new fee structure? Or at least set 'inactive' all accounts which have not engaged in any transactions within X months? Regardless, here are my probabilities for Intrade ending in the next few years: - [Intrade will close/merge/be sold by 2012](http://predictionbook.com/predictions/2051): 5% - [Intrade will close/merge/be sold by 2013](http://predictionbook.com/predictions/2052): 8% - [Intrade will close/merge/be sold by 2015](http://predictionbook.com/predictions/2053): 18% - [Intrade will not be open for business in 2020](http://predictionbook.com/predictions/4236): 35% Fortunately for my decision to cash out (I didn't see anything I wanted to risk holding for more than a few weeks), prices had moved enough that I didn't have to take any losses on any positions^[I made $0.31 on DEM.2012,$3.65 on REP.2012, and $1.40 on 2012.REP.NOM.PALIN for a total profit of$5.36.], and I wound up with $223.32. The$5 for January had already been assessed, and there is a 5 euro fee for a check withdrawal, so my check will actually be for something more like $217, a net profit of$17. I requested my account be closed on 5 January and the check arrived 16 January; the fee for withdrawal was $5.16 and my sum total$218.16 (a little higher than the $217 I had guessed). ### Bitcoin In May-June 2011, [Bitcoin](!Wikipedia), an online currency, underwent approximately 5-6 doublings of its exchange rate against the US dollar, drawing the interest of much of the tech world and myself. (I had first heard of it when it was at 50 cents to the dollar, but had written it off as not worth my time to investigate in detail.) During the first doubling, when it hit parity with the dollar, I began reading up on it and acquired a Bitcoin of my own - a donation from Kiba on [#lesswrong](irc://irc.freenode.net#lesswrong) to try out [Witcoin](http://en.bitcoin.it/wiki/Witcoin), which was a social news site where votes are worth fractions of bitcoins. I then [gave my thoughts](http://lesswrong.com/lw/4cs/making_money_with_bitcoin/3kde) on LessWrong when the topic came up: > "After thinking about it and looking at the current community and the surprising amount of activity being conducted in bitcoins, I estimate that bitcoin has somewhere between 0 and 0.1% chance of eventually replacing a decent size fiat currency, which would put the value of a bitcoin at anywhere upwards of$10,000 a bitcoin. (Match the existing outstanding number of whatever currency to 21m bitcoins. Many currencies have billions or trillions outstanding.) Cut that in half to $5000, and call the probability an even 0.05% (average of 0 and 0.1%), and my expected utility/value for possessing a coin is$25 a bitcoin ($5000 \times 0.005$)." I was more than a little surprised that by June, my expected value had already been surpassed by the market value of bitcoins. Which leads to a tricky question: should I sell now? If Bitcoin is a bubble as frequently argued, then I would be foolish not to sell my 5 bitcoins for a cool $130 (excluding transaction costs). But... I had not expected Bitcoin to rise so much, and if Bitcoin did better than I expected, doesn't it follow that I should no longer believe the probability of success is merely 0.05%? Shouldn't it have increased a bit? Even if it increased only to 0.07%, that would make the EV more like$35 and so I would continue to hold bitcoins. The stakes are high. It is a curious problem, but it's also a prediction market. One is simply predicting what the ultimate price of bitcoins will be. Will they be worthless, or a global currency? The current price is the probability, against an unknown payoff. To predict the latter, one simply holds bitcoins. To predict the former, one simply sells bitcoins. Bitcoins are not commodities in *any* sense. Buying a cow is not a prediction market on beef because the value of beef can't drop to literally 0: you can always eat it. You can't eat bitcoins or do anything at all with them. They are even more purely money than fiat money (the US government having perpetual problems with the zinc or nickel or copper in its coins being worth more as metal than as coins, and dollars are a tough linen fabric). [Mencius Moldbug](http://unqualified-reservations.blogspot.com/2011/04/on-monetary-restandardization.html) turns out to have a similar analysis of the situation: > "If Bitcoin becomes the new global monetary system, one bitcoin purchased today (for 90 cents, last time I checked) will make you a very wealthy individual. You are essentially buying Manhattan for a quarter. There are only 21 million bitcoins (including those not yet minted). (In my design, this was a far more elegant 2^64^, with quantities in exponential notation. Just sayin'.) Mapped to $100 trillion of global money, to pull a random number out of the air, you become a millionaire. Wow! > > So even if the probability of Bitcoin succeeding is epsilon, a million to one, it's still worthwhile for anyone to buy at least a few bitcoins now. The currency thus derives an initial value from this probability, and boots itself into existence from pure worthlessness - becoming a viable repository of savings. If a very strange, dangerous and unstable one. > > I think the probability of Bitcoin succeeding is very low. I would not put it at a million to one, though, so I recommend that you go out and buy a few bitcoins if you have the technical chops. My financial advice is to not buy more than ten^[An aside: there's not much point in accumulating more than, say, 1000 bitcoins. It's generally believed that Bitcoin's ultimate fate will be victory or failure - it'd be very strange if Bitcoin leveled off as a stable permanent alternative currency for only part of the Internet. In such a situation, the difference between 1000 bitcoins and 1500 bitcoins is like the difference to Bill Gates between$60 billion and $65 billion; it matters in some abstract sense, but not even a tiny fraction as much as the difference between$1 and $100 million. Money is logarithmic in utility, as the saying goes.], which should be F-U money if Bitcoin wins." Bitcoin currently represents my largest ever wager in a prediction market; at stake is$130 in losses (if bitcoins go to zero), or indefinite thousands. It will be very interesting to see what happens. (By 5 August 2011, Bitcoin has worked its way down to around $10/btc, making my net worth$26; I did spend several bitcoins on the [Silk Road](), though. By 23 November 2011, it had trended down to $2.35/btc, but due to a large donation of 20 bitcoins, I spent most of my balance at the Silk Road, leaving me with 4.7 bitcoins. Overall, not a good start.) # Predictions > "I recall, for example, suggesting to a regular loser at a weekly poker game that he keep a record of his winnings and losses. His response was that he used to do so but had given up because it proved to be unlucky."^[Ken Binmore, _Rational Decisions_] Writing down precise predictions is like [spaced repetition](): it's brutal to do because it is almost a paradigmatic long-term activity, being wrong is *physically* unpleasant[^dopamine], and it requires 2 skills, formulating precise predictions and then actually predicting. (For spaced repetition, writing good flashcards and then actually regularly reviewing.) There are lots of exercises to try to (calibrate yourself using trivia questions obscure historical events, geography, etc.), but they only take you so far; it's the real world near term and long term predictions that give you the most food for thought, and those require a year or three at minimum. I've used PB heavily for 11 months now, and I used prediction markets for years before PB, and only now do I begin to feel like I am getting a grasp on predicting. We'll look at these alternatives. [^dopamine]: The famous neurotransmitter [dopamine](!Wikipedia "Dopamine#Motivation and pleasure") is intimately involved with feelings of happiness and pleasure (which is why dopamine is affected by most addictions or addictive drugs). It also is involved in learning - make an error and no dopamine for you; ["Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal"](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564381/) (Bayer & Glimcher 2005, _Neuron_): > "The midbrain dopamine neurons are hypothesized to provide a physiological correlate of the reward prediction error signal required by current models of [reinforcement learning](!Wikipedia). We examined the activity of single dopamine neurons during a task in which subjects learned by trial and error when to make an eye movement for a juice reward. We found that these neurons encoded the difference between the current reward and a weighted average of previous rewards, a reward prediction error, but only for outcomes that were better than expected. Thus, the firing rate of midbrain dopamine neurons is quantitatively predicted by theoretical descriptions of the reward prediction error signal used in reinforcement learning models for circumstances in which this signal has a positive value. We also found that the dopamine system continued to compute the reward prediction error even when the behavioral policy of the animal was only weakly influenced by this computation." ## Prediction sites > "The best salve for failure - to have quite a lot else going on."^[[Alain de Botton](https://twitter.com/#!/alaindebotton/status/112772752041181184)] Besides the specific mechanism of prediction markets, one can just make and keep track of predictions oneself. They are much cheaper than prediction markets or informal betting and correspondingly tend to elicit many more responses^[For example, no one has actually taken up [Kevin's offer](http://lesswrong.com/r/discussion/lw/6g0/making_an_amanda_knox_prediction_market/) to wager on the outcome to [Amanda Knox's](!Wikipedia "Amanda Knox") appeal, while there are dozens of specific probabilities given in an [earlier survey](http://lesswrong.com/r/lesswrong/lw/1ir/you_be_the_jury_survey_on_a_current_event/).] There are a number of relevant websites I have a little experience with; some aspire to be like David Brin's proposed [prediction registries](http://www.davidbrin.com/predictionsregistry.htm), some do not: 1. [PredictionBook](http://www.predictionbook.com) (PB) is a general-purpose free-form prediction site. PB is a site intended for personal use and small groups registering predictions; the hope was that LessWrongers would use it whenever they made predictions about things (as they ought to in order to keep their theories grounded in reality). It hasn't seen much uptake, though not for the lack of my trying. I [personally use it](http://predictionbook.com/users/gwern) heavily and have input somewhere around 1000 predictions, of which around 300 have been judged. (I apparently am significantly *under*confident.) A good way to get started is to go to the list of [upcoming predictions](http://predictionbook.com/predictions/future) and start entering in your own assessment; this will give you feedback quickly. 2. [Long Bets](http://longbets.org/) I find the Long Bets concept interesting, but it has serious flaws for anyone who wants to do more than make a public statement like [Warren Buffet has](http://longbets.org/362/). forcing people to put up money has kept real-money prediction markets pretty small in both participants and volume; and how much more so when all proceeds go to charity? No wonder that half a decade or more later, there's only a few hundred money-bets going, even with prominent participants like Warren Buffet. Non-money markets or prediction registries can work in the higher volumes necessary for learning to predict better. Single-handedly on PB I have made 10 times the number of predictions on all of Long Bets. Where will I learn & improve more, Long Bets or PB? (It was easy for me to borrow all the decent predictions and register them on PB.) 3. [FutureTimeline](http://www.futuretimeline.net/) is a maintained list of projected technological milestones, events like the Olympics, and mega-construction deadlines. FutureTimeline does not assign any probabilities and doesn't attempt to track which came true; hence, it's more of a list of suggestions than predictions. I have copied over many of the more falsifiable ones to PB. 4. WrongTomorrow: a site that was devoted solely to registering and judging predictions made by pundits (such as the infamous [Tom Friedman](!Wikipedia "Friedman (unit)")). Unfortunately, WT was moderated and when WT didn't see a sudden massive surge in contributions, moderation fell behind badly until eventually the server was just turned off for the author's other projects. I still managed to copy a number of predictions off it into PB, however. WT is an example of a general failure mode for collections of predictions: no follow-through. Predictions are the paradigmatic [Long Content](About#long-content), and WT will probably not be the first site to learn this the hard way. And the last site demonstrates like Brin's prediction registries have not come into existence. One of the few approximations to a prediction registry is [Philip Tetlock](!Wikipedia)'s justly famous 2005 book _Expert Political Judgment: How Good Is It? How Can We Know?_, which discusses an ongoing study which has tracked >28000 predictions by >284 experts, proves why: experts are not accurate and [can be outperformed](http://www.overcomingbias.com/2006/11/foxes_vs_hedgho.html) by [embarrassingly simple models](http://lesswrong.com/lw/3gv/statistical_prediction_rules_outperform_expert/), and they do not learn from their experience, attempting to retroactively justify their predictions with reference to counterfactuals. (If wishes were fishes... Predictions are about the real world, and in the real world, hacks and bubbles are normal expected phenomena. A verse I saw somewhere runs: "Since the beginning / not one unusual thing has happened". If your predictions can't handle normal exogenous events, then they are still wrong. Tetlock identifies this as a common failure mode of hedgehog-style experts: "I was actually right! but for X Y Z...") And looking around, I think I agree with [Eliezer Yudkowsky](http://lesswrong.com/lw/hi/futuristic_predictions_as_consumable_goods/) that when the vast majority of people make a prediction, it is not an actual prediction to be judged right or wrong but an entertaining [performative utterance](!Wikipedia) intended to [signal partisan loyalties](http://www.overcomingbias.com/2009/08/tetlock-wisdom.html). Another feature worth mentioning is that prediction sites do not generally allow *retrospective* predictions, because that is easily abused even by the honest (who may be suffering [confirmation bias](!Wikipedia)). Prediction markets, needless to say, universally ban retrospective predictions. So, predicting generally doesn't give fast feedback - intrinsically, you can't learn very much from short-term predictions because either there's serious randomness involved such that it takes hundreds of predictions to begin to improve, or the predictions are badly over-determined by available information that one learns little from the successes. ### IARPA: The Good Judgment Project In 2011, the [Intelligence Advanced Research Projects Activity](!Wikipedia) agency (IARPA) began the [Aggregative Contingent Estimation (ACE) Program](http://www.iarpa.gov/solicitations_ace.html), pitting 5 research teams against each other to investigate and improve prediction of geopolitical events. One team, [the Good Judgment Project](http://goodjudgmentproject.blogspot.com/) (see the [_Wired_ interview](http://www.wired.com/wiredscience/2011/08/do-political-experts-know-what-theyre-talking-about/) with [Philip Tetlock](!Wikipedia)), solicited college graduates for the 4 year time period of ACE to register predictions on selected events, for a$150 honorarium. A last-minute notice was posted [on LessWrong](http://lesswrong.com/lw/6ya/link_get_paid_to_train_your_rationality/), and I immediately signed up and [was accepted](http://predictionbook.com/predictions/2973). The initial survey upon my acceptance was long and detailed (calibration on geopolitics, finance, and religion; personality surveys with a lot of fox/hedgehog questions; basic probability; a critical thinking test, the CRT; educational test scores; and then what looked like a full matrix IQ test - we were allowed to see some of [our own results](/docs/2011-gwern-gjp-psychsurveys.html)). The final results will no doubt turn up many interesting correlations or lack of correlation. I look forward [to completing the study](http://predictionbook.com/predictions/2978). At the very least, they will supply a few hundred predictions I can put on PredictionBook.com - formulating a quality prediction (falsifiable, objective, and interesting) can be the hardest part of predicting. #### 2011 results My initial batch of short-term predictions did well; even though I make a major mistake when I fumble-fingered a prediction about Mugabe (I bet that he would fall from office in a month, when I believed the opposite), I was still up by $700 in its play-money. I have, naturally, been copying my predictions [onto PredictionBook.com](http://www.google.com/search?q=%22GJP%3A%20%22%20site%3Apredictionbook%2Ecom) the entire time. Despite a very questionable prediction closure by IARPA which cost me$200[^China], I finished 2011 well in the green. My [results](/docs/2011-gwern-gjp-forecastresults.html); > - Your total earnings for 22 out of 24 closed forecasts is 2,537. > - You are currently ranked 50 among the 206 forecasters in Group 3c. Not *too* shabby; I was actually under the impression I was doing a lot worse than that. Hopefully I can do better in 2012. (By 18 February 2012, I was up to 38 of 204 in my group.) [^China]: Specifically, prediction [#1007](http://predictionbook.com/predictions/3254). In its preface to the results page, GJP told us: > "Question 1007 (the 'lethal confrontation' question) illustrates this point. Many of our best forecasters got 'burned' on this question because a Chinese fishing captain killed a South Korean Coast Guard officer late in the forecasting window – an outcome that the tournament’s sponsors deemed to satisfy the criteria for resolving the question as 'yes', but one that had little geopolitical significance (it did not signify a more assertive Chinese naval policy). These forecasters had followed our advice (or their own common sense) by lowering their estimated likelihood of a lethal confrontation as time elapsed and made their betting decisions based on this assumption." ## Calibration > "The best lack all conviction, while the worst are full of passionate intensity."^[["The Second Coming"](!Wikipedia "The Second Coming (poem)")] Faster even than making one's own predictions is the procedure of [*calibrating*](http://lesswrong.com/r/lesswrong/tag/calibration/) yourself. Simply put, instead of buying shares or not, you give a direct probability: your 10% predictions should come true 10% of the time, your 20% predictions true 20% of the time, etc. This is not so much about figuring out the true probability of the event or fact in the real world but rather about *your* own ignorance. It is as much about learning humility and avoiding hubris as it is about accuracy. You can be well-calibrated even making predictions about topics you are completely ignorant of - simply flip a coin to choose between 2 possibilities. You are still better than someone who is equally ignorant but arrogantly tries to pick the right answers anyway and fails - he will be revealed as miscalibrated. If they are ignorant and don't know it, they will come out overconfident; and if they are knowledgeable and don't realize it, they will come out underconfident. (Note that learning of your overconfidence is less painful than in a prediction market, where you lose your money.) Thus, one can simply compile a trivia list and test people on their calibration; there are [at least 4](http://lesswrong.com/r/lesswrong/lw/1f8/test_your_calibration/) such online quizzes along with the board game [Wits & Wagers](http://boardgamegeek.com/boardgame/20100/wits-wagers). (Consultant Douglas Hubbard has a book _How to Measure Anything: Finding the Value of "Intangibles" in Business_ which is principally on the topic of applying a combination of calibration and [Fermi estimates](Notes#fermi-calculations) to many business problems, which I found imaginative & interesting.) These tests are also useful for occasional independent checks on whether you easily succumb to bias or miscalibration in other domains; I personally seem to do reasonably well[^YM]. [^YM]: For example, in the [YourMorals.org](http://yourmorals.org/) tests dealing with calibration/bias, I usually do well above average, even for LessWrongers; see: - ["an experimental investigation of how people evaluate research evidence that either supports or opposes their pre-existing beliefs"](/docs/2011-gwern-yourmorals.org/evalinfo_process.html) - ["Over-claiming Technique"](/docs/2011-gwern-yourmorals.org/cog_ab_process.html) - ["Balanced Inventory of Desirable Responding"](/docs/2011-gwern-yourmorals.org/bidr_process.html) - ["Marlowe-Crowne Social Desirability Scale"](/docs/2011-gwern-yourmorals.org/crowne_process.html) - ["This scale is designed to measure the better-than-average effect, which is also known as the illusory superiority bias."](/docs/2011-gwern-yourmorals.org/selfeval_process.html) ## 1001 PredictionBook Nights (Initial discussion was [on LessWrong](http://lesswrong.com/lw/7z9/1001_predictionbook_nights/#comments).) > "I am the [core of my mind.](http://www.paulgraham.com/identity.html) \ > [Belief](http://lesswrong.com/lw/s6/probability_is_subjectively_objective/) is my body and [choice](http://wiki.lesswrong.com/wiki/Rationalists_should_win) is my blood. \ > [I have recorded](http://predictionbook.com/users/gwern) over a thousand predictions, \ > [Unaware of fear](!Wikipedia "Loss aversion") \ > Nor [aware of hope](http://lesswrong.com/lw/uk/beyond_the_reach_of_god/) \ > [Have](http://lesswrong.com/lw/21b/ugh_fields/) [withstood](http://lesswrong.com/lw/5xx/overcoming_suffering_emotional_acceptance/) [pain](http://lesswrong.com/lw/jy/avoiding_your_beliefs_real_weak_points/) [to update](http://lesswrong.com/lw/i9/the_importance_of_saying_oops/) many times \ > Waiting for [truth's arrival](http://predictionbook.com/predictions/future). \ > This is the [one uncertain path](http://wiki.lesswrong.com/wiki/How_To_Actually_Change_Your_Mind). \ > My whole life has been... \ > [Unlimited Bayes Works](http://lesswrong.com/lw/h8/tsuyoku_naritai_i_want_to_become_stronger/)!"^[Modified version of [Eliezer Yudkowsky's parody](http://lesswrong.com/lw/ya/normal_ending_last_tears_68/qvc) of the [_Fate/Stay Night_ chant](http://comipress.com/article/2007/07/02/2228).] In October 2009, the site [PredictionBook.com](http://www.predictionbook.com) was [announced on LW](http://lesswrong.com/lw/1bh/predictionbookcom_track_your_calibration/). I signed up in July 2010, as tracking free-form predictions was the logical endpoint of my dabbling in prediction markets, and I had recently withdrawn from Intrade due to [fee changes](Prediction markets#cashing-out). Since then [I have been](http://predictionbook.com/users/gwern) the principal user of PB.com, and a while ago, I registered my 1001^th^ prediction. (I am currently up to >1628 predictions, with >383 judged; PB total has >4258 predictions.) I had to write and research most of them myself and they represent a large time investment. To what use have I put the site, and what have I gotten out of the predictions? ### Using PredictionBook > "Our errors are surely not such awfully solemn things. In a world where we are so certain to incur them in spite of all our caution, a certain lightness of heart seems healthier than this excessive nervousness on their behalf."^[[William James](!Wikipedia), "[The Will to Believe](!Wikipedia)", section VII] Using PredictionBook taught me two things as far as such sites go: 1. Most prosaic, I learned the value of centralizing (and [backing up](Archiving URLs)) predictions of interest to me. I ransacked [LongBets.org](http://longbets.org/), WrongTomorrow.com, [Intrade](!Wikipedia), [FutureTimeline.net](http://futuretimeline.net/), and various collections of predictions like [Arthur C. Clarke's list](http://www.arthurcclarke.net/?scifi=3), LessWrong's own annual prediction threads ([2010](http://lesswrong.com/lw/1la/new_years_predictions_thread/), [2011](http://lesswrong.com/lw/3kz/new_years_predictions_thread_2010/)), or simply [random comments on LW](http://www.google.com/search?q=lesswrong.com%20site%3Apredictionbook.com) (sometimes [Reddit](http://www.google.com/search?q=predictionbook%2Ecom%20site%3Areddit%2Ecom) too). This makes searching for previous predictions easier, graphs all my registered predictions, and makes backups a little simpler. WrongTomorrow promptly vindicated my paranoia by dying without notice. I now have a reply to [David Brin](!Wikipedia)'s oft-repeated plea for a '[predictions registry](http://www.davidbrin.com/predictionsregistry.htm)': no one cares, so if you want one, you need to do it yourself. 2. Second, I realized that using prediction markets had narrowed my appreciation of what predictions are good for. IEM & Intrade had taught me contempt for certain pundits (and respect for [Nate Silver](!Wikipedia)) because they would mammer on about issues where I knew better from the relevant market; but there are very few liquid markets in either site, and so I learned this for only a few things like the US Presidential elections. Prediction markets will be flawed for the foreseeable future, with individual contracts subject to long-shot bias[^longshot] or simply bizarre claims due to illiquidity[^taiwan]; for these things, one must go elsewhere or not go at all. [^longshot]: [Long-shot bias](!Wikipedia "Favourite-longshot bias") is the overvaluing of events in the 0-5% range or so; it plagues even heavily traded markets on Intrade. Ron Paul and Michele Bachmann are 2 cases in point - they are covered by the heavily-traded US Presidential contracts, yet they are priced too high, and this has been noted by many: - - - - - Beyond blog posts, a [2004 Justin Wolfers paper](http://bpp.wharton.upenn.edu/jwolfers/Papers/Predictionmarkets.pdf) finds their presence: > "In fact, the price differences implied a (small) arbitrage opportunity that persisted for most of summer 2003 and has reappeared in 2004. Similar patterns existed for Tradesports securities on other financial variables like crude oil, gold prices and exchange rates. This finding is consistent with the long-shot bias being more pronounced on smaller-scale exchanges." This is apparently due in part to the short-term pressure on prediction market traders; [Robin Hanson](http://lesswrong.com/lw/1ia/arbitrage_of_prediction_markets/1b2l) says: > "Intrade and IEM don't usually pay interest on deposits, so for long term bets you can win the bet and still lose overall. The obvious solution is for them to pay such interest, but then they'd lose a hidden tax many customers don't notice." Another reason to use a free-form site like PB.com - you can (and I have) made predictions about decades or centuries into the far future without worrying about how to earn returns of thousands of percent. [^taiwan]: Going through Intrade to copy over predictions to PB.com, I was struck by how non-liquid markets could be left at hilarious prices, prices that make no rational sense since they can't even represent someone hedging against that outcome because so few shares have been sold; example contracts include: 1. [US attacking North Korea](http://predictionbook.com/predictions/3048) 2. [China attacking Taiwan](http://predictionbook.com/predictions/3052) 3. [Japan acquiring nuclear weapons](http://predictionbook.com/predictions/3073) ### Noted predictions Do any particular sets of predictions come to my mind? Yes: 1. My largest outstanding collection are [the >207 predictions](otaku-predictions) about the unreleased _Evangelion_ movies & manga; I regard their upcoming releases as excellent chances to test my theories about _Evangelion_ interpretation in a way that is usually impossible when it comes to literary interpretation 2. For my personal Adderall double-blind trial, I [recorded 16 predictions about a trial](Nootropics#adderall-blind-testing) (guessing whether it was placebo or Adderall) to try to see how strong an effect I could diagnose, in addition to whether there was one at all. (I also did one for [modafinil](Nootropics##modalert-blind-day-trial).) 3. During the big Bitcoin bubble, I recorded a number of predictions on Reddit & LW and followed up on a number of them; I believe this was educational for those involved - at the least, I think I tempered my own enthusiasm by noting the regular failure of the most optimistic predictions and the very low Outside View probability of a take-off 4. I have made qualitative predictions in [Haskell Summer of Code](), but I've refrained from recording them because I've been accused of being subjective in my evaluations. ### Benefits from making predictions When I do use predictions, I've noticed some direct benefits: - Giving probabilities can make an analysis clearer (how do I know what I think until I see what I predict?); when I speculated on the identity of [Mike Darwin](http://www.chronopause.com)'s patron (above, 'Notes'), the very low probabilities I assigned in the conclusion to any particular billionaire makes clear that I repose no real confidence in any of my guesses and that this is more of a Fermi problem puzzle or exercise than anything else. (And indeed, none of them were correct.) I believe that sharpening my analyses has also made me better at spotting political bloviation and pundits pontifying: > "Don't ask whether predictions are made, ask whether predictions are implied."^[[Steven Kaas](https://twitter.com/#!/stevenkaas/statuses/149616831290818560)] - Going on the record with time-stamps can turn sour-grapes into a small victory. If one read my [Silk Road]() article and saw [a footnote](Silk Road#fn3) to the effect that the Bitcoin forum administrators were censors who removed any discussion of the Silk Road, such an accusation is rather less convincing than a footnote linking to a prediction that a particular thread would be removed and noting that as the reader can verify for themselves, said thread was indeed subsequently deleted. One of the things I hoped would make my site [unusual](About#long-content) was regularly employing prediction; I haven't been able to do it as often as I hoped, but I've still used it in 8 pages: - [About](): projections about finishing writing/research projects - [In Defense Of Inclusionism](): predicting the WMF's half-hearted efforts at editor retention will fail - [Nootropics](): double-blind Adderall trial (above) - [Notes](Notes#the-hidden-library-of-the-long-now): predictions on Steve Jobs's lack of charity, correctness of speculative analysis - [otaku-predictions](): mentioned previously - [plan](): cost of genotyping, to guide decision not to purchase last year - [Prediction markets](): political predictions, Intrade failure predictions, GJP acceptance - [Silk Road](): prediction of censorship on main Bitcoin forums (see above), and of no legal repercussions ### Lessons learned > "We should not be upset that others hide the truth from us, when we hide it so often from ourselves."^[[François de La Rochefoucauld](!Wikipedia), _Maximes_ 11] To sum things up, like the [haunted rationalist](http://lesswrong.com/lw/1l/the_mystery_of_the_haunted_rationalist/), I learned in my gut things that I already supposedly knew - the biases are now [more satisfying](http://lesswrong.com/lw/73r/remind_physicalists_theyre_physicalists/); the following are my subjective impressions: - I knew, to quote Julius Caesar, that "What we wish, we readily believe, and what we ourselves think, we imagine others think also.", but it wasn't until I was sure that George Bush would not be re-elected in 2004, that I knew that I could succumb to that even in abstract issues which I had read enormous quantities of information & speculation on. - while I am weak in areas close to me, in other areas I am underconfident, which is [a sin](http://lesswrong.com/lw/c3/the_sin_of_underconfidence/) and as much to be remedied as overconfidence. (Specifically, it seemed I was initially overconfident on 95%+ predictions and underconfident in the 60-90% regime; I think I've learned my lesson, but by the nature of these things, my recorded calibration will take many predictions to recover in the extreme ranges.) - I am too optimistic and not cynical enough; the cardinal example, personally, would be the five-year [XiXiDu](http://predictionbook.com/predictions/2909) prediction which was falsified in *one month*. The Outside View heavily militated against it, as did my fellow predictors, and if it had been formulated as something socially disapproved of like alcohol or smoking, I would probably have gone with 10 or 20% like JoshuaZ; but because it was a fellow LessWronger trying to get his life straight... - I am considerably more skeptical of op-eds and other punditry, after tracking the rare clear predictions they made. (I was already wary due to Tetlock, and a more recent [study of major pundits](http://www.hamilton.edu/news/polls/pundit/an-analysis-of-the-accuracy-of-forecasts-in-the-political-media.pdf) but not enough, it seems.) The rareness of such predictions has instill in me an appreciation of Hansonian signaling theories of politics - it is *so* hard to get falsifiable predictions out of writings even when they *look* clear; for example, leading up to the 2011 US Federal debt crisis and ratings downgrade, everyone prognosticated furiously - but did they mean any rating agency, or all of them, or just a majority? - I respect fundamental trends more; they are powerful predictors indeed, and like Philip Tetlock's experts, I find that it's hard to out-perform the past in predicting. I no longer expect much of politicians, who are as trapped as the rest of us. This could be seen as more use of base rates as the prior, or as moving towards more of an Outside View. I am frequently reminded of the power of reductionism and analysis - pace _MoR_ Quirrel's question to Harry[^mor], what states of the world would a prediction coming true imply had become more likely? Sometimes when I record predictions, I see someone who has [clearly not considered](http://lesswrong.com/lw/1la/new_years_predictions_thread/2hg3) what his predictions coming true implies about the *current* state of the world; I sigh and reflect on how you just can't get *there* from *here*. - Merely contemplating seriously my predictions over years and decades makes the future much more concrete to me; I will live most of my life there, so I *should* take a longer-term perspective. - Making thousands of predictions has helped me gain detachment from particular positions and ideas (which made it easier for me to write my [Mistakes]() essay and publicly admit them - after so many 'failures' on PB.com, what were a few described in more detail?) To quote [Alain de Botton](https://twitter.com/#!/alaindebotton/status/112772752041181184): > "The best salve for failure -- to have quite a lot else going on." This detachment itself seems to help accuracy; I was struck by a psychology study demonstrating that not only are people better at falsifying theories put forth by other people, they are better at falsifying *when pretending it is held by an imaginary friend*[^imaginaryfriend]! - Raw probabilities are more intuitive; I can't describe this much better than the poker article, ["This is what 5% feels like."](http://rationalpoker.com/2011/04/21/this-is-what-5-feels-like/) - [Planning fallacy](http://lesswrong.com/lw/jg/planning_fallacy/): I knew it perfectly well, but still committed it until I tracked predictions; this is true both of my own mundane activities like writing, and larger more global events (recently, running out the clock on the Palestinian nationhood UN vote) This was interesting because it's so easy to make excuses - 'I would've succeeded if not for X!' The question (in the classic study) is whether students could predict their projects' actual completion time; they're not trying to predict project completion time given a hypothetical version of themselves which didn't procrastinate. If they aren't self-aware enough to know they procrastinate and to take that into account - their predictions are still bad, no matter *why* they're bad. (And someone on the outside who is told that in the past the students had finished -1 days before the due date will just shrug and say: 'regardless of whether they took so long because of procrastination, or because of [Parkinson's law](!Wikipedia), or because of a 3rd reason, I have no reason to believe they'll finish early *this* time.' And they'd be absolutely correct.) It's like a fellow who predicts he won't fall off a cliff, but falls off anyway. 'If only that cliff hadn't been there, I wouldn't've fallen!' Well, duh. But you still fell. How can you correct this until you stop making excuses? - Less [hindsight bias](http://wiki.lesswrong.com/wiki/Hindsight_bias); when I have my previous opinions written down, it's harder to claim I knew it all along (when I didn't), and as [Arkes et al 1988](/docs/1988-arkes.pdf "Eliminating the hindsight bias") indicated, writing down my reasons (even in Twitter-sized comments) helped prevent it. Example: I had put the 2011 S&P downgrade at [5%](http://predictionbook.com/predictions/2030), and reminded of my skepticism, I can see the double-standards being applied by pundits - all of a sudden they remember how the ratings agencies failed in the housing bubble and how the academic literature has proven they are inferior to the [CDS](!Wikipedia "Credit default swap") markets and how they are a bad government-granted monopoly, even though they were happy to cite the AAA rating beforehand and are still happy to cite the *other* ratings agencies... In short, while base rates are powerful indeed, there are still many exogenous events and multiplicities of low probability events. [^mor]: Eliezer Yudkowsky, [chapter 20](http://www.fanfiction.net/s/5782108/20/Harry_Potter_and_the_Methods_of_Rationality), _Harry Potter and the Methods of Rationality_: > "...while I suppose it is barely possible that perfectly good people exist even though I have never met one, it is nonetheless _improbable_ that someone would be beaten for fifteen minutes and then stand up and feel a great surge of kindly forgiveness for his attackers. On the other hand it is _less_ improbable that a young child would imagine this as the _role to play_ in order to convince his teacher and classmates that he is not the next Dark Lord. > > The import of an act lies not in what that act _resembles on the surface_, Mr. Potter, but in the states of mind which make that act more or less probable." [^imaginaryfriend]: ["When falsification is the only path to truth"](http://csjarchive.cogsci.rpi.edu/Proceedings/2005/docs/p512.pdf); abstract: > "Can people consistently attempt to falsify, that is, search for refuting evidence, when testing the truth of hypotheses? Experimental evidence indicates that people tend to search for confirming evidence. We report two novel experiments that show that people can consistently falsify when it is the only helpful strategy. Experiment 1 showed that participants readily falsified somebody else's hypothesis. Their task was to test a hypothesis belonging to an 'imaginary participant' and they knew it was a low quality hypothesis. Experiment 2 showed that participants were able to falsify a low quality hypothesis belonging to an imaginary participant more readily than their own low quality hypothesis. The results have important implications for theories of hypothesis testing and human rationality." One line of thought in [evolutionary psychology](!Wikipedia) is that our minds are *not* evolved for truth-seeking _per se_, but rather are split between heuristics and effective procedures like that, and argumentation to try to deceive & persuade others; eg. ["Why do humans reason? Arguments for an argumentative theory"](http://files.meetup.com/284333/Philosophy-Reason_SSRN-id1698090.pdf) (Mercier & Sperber 2011). This ties in well with why we are better at falsifying the theories of *others* - you don't convince anyone by falsifying your own theories, but you do by falsifying others' theories. I think, but am not sure, that I really have [internalized](http://lesswrong.com/lw/1yq/understanding_your_understanding/) these lessons; they simply seem... obvious to me, now. I was surprised when I looked up my earliest work and saw it was only around 14 months ago - I felt like I'd been recording predictions for far longer. ### Non-benefits > "If people don't want to come to the ballpark how are you going to stop them?"^[[Yogi Berra](!Wikipedia), _The Yogi book: I really didn't say everything I said!_, Workman Publishing, 1997, ISBN 0761110909, p. 36] Making predictions has been personally costly; while some predictions have been total time investments of a score of seconds, other predictions required considerable research, and thinking carefully is no picnic, as we've all noticed. I justify the invested time as a learning experience which would hopefully pay off for others as well, who can free-ride off the many predictions (eg. the [soon-to-expire predictions](http://predictionbook.com/predictions/future)) I have laboriously added to PB.com. (Only a fool learns from his mistakes only.) What I have not noticed? It was suggested that predictions might help me in resolutions based on some experimental evidence[^hbv]; I did not notice anything, but I didn't carefully track it or put in predictions about many routine tasks. Making predictions seems to be largely effective for improving one's *epistemic* rationality; I make no promises or implied warranties as to whether it is *instrumentally* rational. [^hbv]: ["Can self-prediction overcome barriers to hepatitis B vaccination? A randomized controlled trial"](http://www.ncbi.nlm.nih.gov/pubmed/21875205) 2011: > "Half of participants were assigned randomly to a "self-prediction" intervention, asking them to predict their future acceptance of HBV vaccination. The main outcome measure was subsequent vaccination behavior. Other measures included perceived barriers to HBV vaccination, measured prior to the intervention. Results: There was a significant interaction between the intervention and vaccination barriers, indicating the effect of the intervention differed depending on perceived vaccination barriers. Among high-barriers patients, the intervention significantly increased vaccination acceptance. Among low-barriers patients, the intervention did not influence vaccination acceptance. Conclusions: The self-prediction intervention significantly increased vaccination acceptance among "high-barriers" patients, who typically have very low vaccination rates." ### How I make predictions A prediction can be broken up into 3 steps: 1. The specification 2. The due-date 3. The probability The first issue is simply formulating the prediction. The goal is to make a statement on an objective and easily checkable fact; imagine that the other people predicting are yourself if you had been raised in some completely opposite fashion like an evangelical Republican household, and they are quite as suspicious of you as you are of them, and believe you to be suffering from as many partisan and self-serving biases as you believe them to. The prediction should be so clear that they would expose themselves to mockery even among their own kind if they were to seriously disagree about the judgment^[It may help to read the dialogue/examples of "Dr. Malfoy" and "Dr. Potter" in [chapter 22](http://www.fanfiction.net/s/5782108/22/Harry_Potter_and_the_Methods_of_Rationality) ('The Scientific Method') of Eliezer Yudkowsky's _Harry Potter and the Methods of Rationality_.]. For example, 'Obama will be the next President' is perfectly precise - *everyone* knows and understands what it is to be President and how one would decide - and so there's no need to do any more; it would be risible to try to deny it. On the other hand, 'the globe will increase 1 degree Fahrenheit' may initially sound good, but your dark counterpart immediately objects: 'what if it's colder in Russia? When is this increase going to happen? Is this exactly 1 degree or are you going to try to claim as success only 0.9 degrees too? Who's deciding this anyway?' A good resolution might be 'OK, global temperatures will increase >=1.0 degrees Fahrenheit on average according to the next IPCC report'. Deciding the due-date of a prediction is usually trivial and not worth discussing; when making open-ended predictions about people (eg. 'X will receive a Nobel Prize'), I find it helpful to consult [life table](!Wikipedia)s like [Social Security's table](http://www.ssa.gov/oact/STATS/table4c6.html) to figure out their average life expectancy and then set the due-date to that. (This both minimizes the number of changes to the due date and helps calibrate us by pointing out what time spans we're really dealing with.) When we begin deciding what probability to give the prediction, we can employ a number of heuristics (partially drawn from ["Techniques for probability estimates"](http://lesswrong.com/lw/3m6/techniques_for_probability_estimates)): 1. What does the prediction about the future world imply about the present world? Every prediction one makes is also a *retrodiction*: you are claiming that the world is now and in the past on a course towards the future you have picked out of all the possibilities (or not on that course), and on that course to the degree you specified. What does your claim imply about the world as it is now? The world has to be in a state which can progress of its own internal logic to the future state, and so we can work backwards to figure out what that implies about the present or past. (You can think of this as a kind of proof by contradiction: assuming prediction _X_ is true, what can we infer from _X_ about the present world which is absurd?) In our first example, Miller predicted 15% for ["Within ten years either genetic manipulation or embryo selection will have been used on at least 50% of Chinese babies to increase the babies' expected intelligence"](http://predictionbook.com/predictions/1689). This initially seems reasonable: China is a big place with known interests in eugenics. But then we start working backwards - this prediction implies handling >=9 million pregnancies annually, which entails hundreds of thousands of gynecologists, geneticists, lab technicians etc., which all have lead-times measured in years or decades. (It takes a long time to train a doctor even if your standards are low.) And the program must be set up with hundreds of thousands of employees, policies experimented with and implemented, and so on. As matters stand, even in the United States mere [SNP](!Wikipedia "Single-nucleotide polymorphism") genotyping couldn't be done for 9 million people annually, and genetic sequencing is much more expensive & difficult, and genetic modification is even hairier. If we work backwards, we would expect to see such a program already begun and active as it frantically tries to scale up to handle those millions of cases a year in order to hit Miller's deadline. But as far as I knows, all the pieces are absent in China as of the day it was predicted; hence, it's already too late. And then there are the politics; it is a deeply doubtful assertion that the Chinese population would countenance this, given the stress over the [One Child policy](!Wikipedia) and the continuing [selective abortion](!Wikipedia "Sex-selective abortion") crisis. Even if the prediction comes true eventually, it definitely will not come true in time. (The same logic applies to ["Within ten years the SAT testing service will require students to take a blood test to prove they are not on cognitive enhancing drugs."](http://predictionbook.com/predictions/1691); [~1.65 million test-takers](http://press.collegeboard.org/releases/2011/43-percent-2011-college-bound-seniors-met-sat-college-and-career-readiness-benchmark) implies scores of thousands of [phlebotomists](!Wikipedia), who do not exist, although in theory they could be trained in under a year - but whence the trainers?) A second example would be a series of predictions on anti-aging/life-extension. The first and earliest prediction - ["By 2025 there will be at least one confirmed person who has lived to 130"](http://predictionbook.com/predictions/3847) - initially seems at least possible (I am optimistic about the approaches suggested by [SENS](!Wikipedia)), and so I assigned it a reasonable probability of 3%. But I felt troubled - something about it seemed wrong. So I applied this heuristic: what does the existence of an 130 year-old in 2025 imply about people in 2011? Well, if someone is 130 in 2025, then that implies that are now 116 years old ($130 - (2025 - 2011)$). Then I looked up the oldest person in the world: [Besse Cooper](!Wikipedia), aged 11*5* years old. Oops. It's *impossible* for the prediction to come true, but because we didn't think about what it coming true implied about the present world, we made an absurdly high prediction. We can do this for all the other anti-aging predictions; for example ["By 2085 there will be at least one confirmed person who has lived to 150"](http://predictionbook.com/predictions/4431) can be rephrased as 'someone aged 76 now will live to 2085', which seems implausible except with a [technological singularity](!Wikipedia) of some sort ("Hmm, phrased in that context, my estimate has to go down"). This can be applied to financial or economic questions, too, since under even the weakest version of [efficient markets](!Wikipedia), the markets are smarter than you - [Tyler Cowen](!Wikipedia) asks [why we don't](http://marginalrevolution.com/marginalrevolution/2011/11/what-is-the-future-of-solar-power.html) see investor piling into solar power if it's following an exponential curve downwards and is such a great idea ([Robin Hanson](http://www.overcomingbias.com/2011/11/when-see-solar.html) appeals to discount rates and purblind investors). The idea of 'rephrasing' leads directly into the next heuristic. 2. [Base rates](!Wikipedia). Already discussed, but base rates should be your mental starting point for every prediction, before you take into account any other opinion or belief. Base rates are easily expressed in terms of frequencies: "of the last Y years, X happened only once, so I will start with 1/Y%". ("There are 10 candidates for the 2012 Republican nominee, so I will assume 10% until I've looked at each candidate more closely.") Frequencies have a long history in the academic literature of making suboptimal or fallacious performance just disappear, and there's no reason to think that is not true for your predictions as well. This works for personal predictions as well - focus on what sort of person you are, how you've done in similar cases over years, and you'll improve your predictions[^personal-predictions]. An example: ["A Level 7 (Chernobyl/2011 Japan level) nuclear accident will take place by end of 2020"](http://predictionbook.com/predictions/4654). One's gut impression is a very bad place to start because Fukushima and Chernobyl - mentioned in the very prediction! - are such vivid and [mentally available](!Wikipedia "Availability heuristic") examples. 60%? 50%? Read the coverage of Fukushima and many people give every impression of expecting fresh disasters in coming years. (Look at Germany quickly announcing the shutdown of its nuclear reactors, despite tsunamis not being a *frequent* problem in northern Europe.) But if we *start* with base rates and look up nuclear accidents, we realize something interesting: Chernobyl and Fukushima come to mind readily in part because they are literally the *only* such level-7 accidents over the past >40 years. So the frequency would be 1 in ~20 years, which certainly puts a different face on a prediction spanning 9 years. This gives us a base rate more like ~40%. This is our starting point for asking how much does the rate go down because Fukushima has prompted additional safety improvements or closure of older plants (Fukushima's equally-outdated sibling nuclear plants will have a harder time getting delays in their executions) and how much the rate goes up due to global warming or aging nuclear plants. But from here we can hope to arrive at a sensible answer and not be spooked by a recent incident. 3. Breaking predictions down into conjunctions Similar to heuristic #1, we may not realize what a prediction implies *internally* and so wind up giving high probability to [a vivid or interesting scenario](!Wikipedia "Conjunction fallacy"). 'Hillary Clinton will become President in 2012' is specific, easily dateable, implies things about the present world like rumors of Clinton running and strong political connections (as do exist), and yet this prediction is *still easy to mess up*. Why? Because becoming President is actually the outcome of a long series of steps, every one of which must be successful and every one of which is doubtful: Hillary must resign from the White House, she must announce a run, she must become Democratic nominee (out of several candidates), and she must actually win. It's the exceptional nominee who ever has >50% odds, so we start with a coin flip and work our way down to perhaps a few percent. We can see a particularly striking failure to analyze in the prediction ["Obama gets reelected and during that time Hillary Clinton brokers the middle east peace deal between Israel and Palestine for the two state solution. This secures her presidency in 2016."](http://predictionbook.com/predictions/4145), where the predictor gave it a flabbergasting *80%*; before clicking through, the reader is invited to assign probabilities to the following events (and then multiply them to obtain the probability that they will *all* come true): 1. Barack Obama is re-elected 2. A Middle East peace deal is brokered 3. The peace deal is for a two state solution 4. Hillary Clinton runs in 2016 5. Hillary Clinton is the 2016 Democratic nominee 6. Hillary Clinton is elected (Sometimes the examples are [even more extreme](http://predictionbook.com/predictions/5070) than 6 clauses.) This heuristic is not perfect, as it works best on step-by-step processes where every step must happen. If this is not true, the heuristic will be overly pessimistic. Worse, it is possible to lie to ourselves by simply breaking down the steps into ever tinier steps and giving them relatively small probabilities like 99%. [Steven Kaas](https://twitter.com/#!/stevenkaas/status/9852587905912832): > "Walking requires dozens of different muscles working together, so if you think you can walk you're just committing the conjunction fallacy." [^personal-predictions]: From _Principles of Forecasting_: > "[Osberg and Shrauger (1986)](/docs/1986-osberg.pdf "Self-Prediction: Exploring the Parameters of Accuracy") determined prediction accuracy by scoring an item as a hit if the respondents predicted the event definitely or probably would occur and it did, or if the respondent predicted that the event definitely or probably would not occur and it did not. Respondents who were instructed to focus on their own personal dispositions predicted significantly more of the 55 items correctly (74%) than did respondents in the control condition who did not receive instructions (69%). Respondents whose instructions were to focus on personal base rates had higher accuracy (72%) and respondents whose instructions were to focus on population base rates had lower accuracy (66%) than control respondents, although these differences were not statistically significant." # External links - ["Calibrate your self-assessments"](http://lesswrong.com/lw/7o7/calibrate_your_selfassessments/) -(miscalibration of one's capability, performance, personal appearance etc. can cause suffering & stress) - ["Calibrating our Confidence"](http://measureofdoubt.com/2011/08/17/calibrating-our-confidence/) -(on PB.com) - ["Amanda Knox: post mortem"](http://lesswrong.com/lw/84j/amanda_knox_post_mortem/) -(even if we cannot infallibly judge our predictions, our beliefs should still change over time) - ["PredictionBook: A Short Note"](http://lesswrong.com/r/discussion/lw/8dx/predictionbook_a_short_note/)