Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
1290 lines (774 sloc) 143 KB
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="">
<title><![CDATA[Jason Punyon]]></title>
<link href="" rel="self"/>
<link href=""/>
<name><![CDATA[Jason Punyon]]></name>
<generator uri="">Octopress</generator>
<title type="html"><![CDATA[Trip Report: ICML 2015]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>One of the awesome benefits of working at Stack Overflow is the Conferences and Education policy. You get a ballpark budget ($3500) and time (3 days) each year to update your skills and knowledge. In return for all this conferencing and educationing you write up a summary of what you learned and your experience at the conference.</p>
<p>This year I went to the <a href="">International Conference on Machine Learning</a> in Lille, France. ICML is an academic conference broken up into three parts: 1 day of tutorials, 3 days of &ldquo;The Main Conference&rdquo;, and 2 days of Workshops.</p>
<p><strong>TL;DR</strong>: I loved the tutorials, the keynotes, and the workshops. The paper sessions were hit-or-miss but I might&rsquo;ve been doing it wrong (sorry, I&rsquo;m new to the job). It was a great conference, I&rsquo;m gonna go again next year.</p>
<h2>The Tutorials</h2>
<p>The tutorials were my favorite part of the conference. I wish the whole conference were tutorials like the ones I went to on this first day. There were three two-hour-long sessions that started at the surface and tried to dive somewhat deeply into their respective topics.</p>
<h3>Natural Language Understanding: Foundations and State-of-the-Art</h3>
<p>This was my favorite tutorial. It was given by Percy Liang, a computer science professor and natural language researcher at Stanford. One of the presenter&rsquo;s goals was to seed the minds of ML theorists and practitioners with the essential phenomena and sharp corners of natural language in hopes of motivating new ideas. The amount of information he related in such a short time was impressive and there&rsquo;s no way I can do it justice in a paragraph. Read through <a href="">the slides</a> to get an idea. There&rsquo;s video coming in September.</p>
<h3>Advances in Structured Prediction</h3>
<p>The next tutorial was a two-parter on Structured Prediction from Hal Daume III and John Langford. I liked this tutorial because I knew the least about structured prediction going in and I felt like I had at least a tenuous grasp on it by the end. Structured prediction is about answering the question &ldquo;What can you do if you&rsquo;re going to give up the independence assumption and model a joint probability distribution?&rdquo; The tutorial went through what structured prediction was and existing methods, then talked about their method (Learning to Search), empirical results, performance, and advertised their fast/flexible implementation in Vowpal Wabbit. <a href="">slides</a></p>
<h3>Modern Convex Optimization Methods for Large-scale Empirical Risk Minimization</h3>
<p>The final tutorial of the day was another two-parter on convex optimization. Machine learning is frequently formulated as minimizing (optimizing) a loss function. Certain loss functions have mathematical properties that make them efficiently optimizable and we call those functions &ldquo;convex&rdquo;. Two subsets of methods within convex optimization are: &ldquo;primal&rdquo; methods (gradient descent) and &ldquo;dual&rdquo; methods (coordinate descent). The first part of the talk covered primal methods and the second part of the talk covered the dual methods and concerns at-scale like minibatching, parallelization, and distributed implementations.</p>
<p>I liked this talk because it was mathy and about performance characteristics on big problems and we really dig that kind of thing at Stack Overflow, however this seemed like it was the furthest tutorial from me practically. It&rsquo;s nice to know that people are attacking that size of problem but our problems just aren&rsquo;t big enough to warrant some of these techniques. <a href="">part 1 slides</a> <a href="">part 2 slides</a></p>
<h2>The Main Conference</h2>
<p>Each day of the main conference had a keynote address and then 3-4 sessions of 3-5 short talks on submitted papers organized into different tracks. There were also poster sessions where you could meet and greet the researchers and dive into their paper with them in more depth.</p>
<p>Susan Murphy&rsquo;s &ldquo;Learning Treatment Policies in Mobile Health&rdquo; was an interesting tale at the corner of healthcare funding, clinical trial design, machine learning, and mobile development. Smartphones might be able to provide effective, low-cost, personalized, just-in-time treatment for things we don&rsquo;t fund particularly well in the US like drug addiction treatment. How to even study the problem is a can-of-worms factory. Conflating factors abound where you&rsquo;re vying for a subject&rsquo;s attention via their phone in the real world instead of in an in-patient treatment setting. Trying to learn effective treatment policies through the noise is a crazy-ambitious goal. I was really impressed by their devotion to the ethics of the research. Things like refusing to have the phone light up/deliver messages when it was moving above a certain speed. No point in trying to help the person&rsquo;s drug addiction if messaging them causes them to crash their car.</p>
<p>Jon Kleinberg&rsquo;s &ldquo;Social Phenomena in Global Networks&rdquo; talked about how we&rsquo;re recording social activity more granularly than ever and insights that can be gleaned from looking at a person&rsquo;s network neighborhood in a social graph and the interactions the person has with their network over time. He discussed the different features to extract from the graph in order to solve the problem of identifying a person&rsquo;s romantic partner.</p>
<h4>Two High-Stakes Challenges in Machine Learning</h4>
<p>Both of those were super interesting but they were eclipsed by Leon Bottou&rsquo;s talk on <a href="">two high-stakes challenges in machine learning</a>. The first challenge is that machine learning components are fundamentally different from the things we&rsquo;re used to in software engineering. ML components aren&rsquo;t rigid like sorting routines, they&rsquo;re squishy. The guarantees they provide are weak. If the training set data distribution is different from the data distribution in situ for any reason, all your guarantees go out the window. Your component just ambles around drunkenly in an alien world grasping for vestiges of the only reality it knows.</p>
<p>&ldquo;AHA!&rdquo; you think. &ldquo;I ain&rsquo;t no fool. We won&rsquo;t fall victim to the data distribution drift problem with statically trained models because we won&rsquo;t use &lsquo;em! We&rsquo;ll learn <em>online</em>!&rdquo; I, for one, greatly admire the bold leadership it takes to suggest that we actually fight fire with fire. Some would say &ldquo;Maybe we&rsquo;ve got enough learning,&rdquo; but you say &ldquo;Maybe we haven&rsquo;t got enough learning enough!&rdquo; I like you.</p>
<p>The problem with learning online is that you&rsquo;ve created an even shakier foundation for your engineering division, because feedback loops are <em>everywhere</em>. If your organization has more than one online learning ML algorithm in a particular pipeline you&rsquo;ve got a system where one piece can learn from another piece that is learning from the first piece. These feedback loops can manifest themselves at two online algorithms in the pipeline, imagine what happens when you&rsquo;re trying to learn online in many places across a large engineering organization.</p>
<p>The second challenge is that the experimental paradigm that has driven machine learning forward over the last few decades is reaching its limits. Machine learning has a single experimental paradigm: split up the data into a test set and training/validation sets, train on the training/validation sets, and test final performance on the test set. Machine learning is an outlier among experimental sciences in that it has a single paradigm. Other sciences have to work much harder to design their experiments and interpret their results.</p>
<p>One of the problems with this single paradigm and its focus on datasets is that <a href="">large datasets are biased</a>. Training and testing on biased datasets leads to unrealistic expectations of performance. The image below pretty much tells the story. When classifying cars, models trained on one of the large datasets perform worse when tested on the other large datasets almost universally. If cars were represented in these datasets in an unbiased way you&rsquo;d expect much smaller differences in performance.</p>
<p><img src=""></p>
<p>And while we&rsquo;re at it, we aren&rsquo;t even learning the right things if we want to acheive the heights of our increased ambitions. Our solution to classifying is statistical classification. But it&rsquo;s hard to find a purely statistical basis in which to ground some problems like the one brought up in this slide&hellip;</p>
<p><img src=""></p>
<p>These insights were really eye-opening for me coming from a young engineering organization that&rsquo;s only recently become more rigorous about adding ML components to our stack.</p>
<p>Paper sessions were scheduled around the keynotes. Each session had a topic, some topics spanned multiple sessions over the three days. The sessions all went similarly, a presenter would present their paper in ~15 minutes, then there was time for a few questions, then&hellip;next presenter.</p>
<p>This ended up being pretty grueling by the end of the third day. There wasn&rsquo;t really time to just think about any of the things that were presented to you. Trying to put together the-problem-you-didn&rsquo;t-even-know-was-a-problem-3-minutes-ago and the research-at-the-edge-of-human-understanding that was just presented as the solution to that problem was a difficult process to repeat 3 times an hour, 6 hours a day, for 3 days. That&rsquo;s not to mention synthesizing the information from the day&rsquo;s keynote, and the poster sessions. These were long days. Jet lag wasn&rsquo;t helping either.</p>
<p>At any time there were 6 paper sessions going on, so my strategy was to just pick the topic that I was most interested in. This was a bit of a high variance strategy as the sessions had a hit-or-miss quality to them. Other people used a more informed strategy of picking specific papers/presenters they knew to be good and then chose their sessions based on that. There&rsquo;s no &ldquo;Map of the Machine Learning Stars&rdquo; to consult though&hellip;my lack of familiarity with a lot of the field precludes that kind of strategy. Maybe when I&rsquo;ve been around a bit longer I&rsquo;ll have a better handle on that kind of thing.</p>
<h2>The Workshops</h2>
<p>The final piece of the conference was the workshops. These were little mini-conferences organized by groups repesenting subfields of machine learning. I was happy to attend the 2-day workshop on the latest in Deep Learning. Invited speakers gave hour long talks on their research and took questions. The first day was supervised learning, the second was on unsupervised and reinforcement learning.</p>
<p>The highlight of the two days was Jason Weston&rsquo;s presentation on the <a href="">bAbI tasks</a>. Leon Bottou mentioned this work in his keynote as an example of a way to evaluate machine learning systems that wasn&rsquo;t just another instance of the normal training/validation/test set paradigm. The context for the work is in Question Answering. The computer is presented a small story and asked a simple question like this&hellip;</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>Mary moved to the bathroom.
</span><span class='line'>John went to the hallway.
</span><span class='line'>Where is Mary?</span></code></pre></td></tr></table></div></figure>
<p>You can classify the questions asked by the kind of knowledge necessary to answer them. The example above is of task #1 &ldquo;Basic Factoid QA with a Single Supporting Fact&rdquo;. Here&rsquo;s an example of task #14 &ldquo;Time manipulation&rdquo;&hellip;</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class=''><span class='line'>In the afternoon Julie went to the park.
</span><span class='line'>Yesterday Julie was at school.
</span><span class='line'>Julie went to the cinema this evening.
</span><span class='line'>Where did Julie go after the park?</span></code></pre></td></tr></table></div></figure>
<p>There are 18 more of these tasks that all test a different aspect of the machine&rsquo;s &ldquo;intelligence&rdquo;. Counting, Negation, Deduction, Positional reasoning are some of these aspects. The bAbI tasks are interesting because they attempt to probe a specific ability of the program, much like a unit test. Cataloging the abilities of an ML system in this way was something I haven&rsquo;t come across before. I&rsquo;m really drawn to the idea.</p>
<h2>Wrap Up</h2>
<p>ICML 2015 was a great conference. It reminded me of being in the physics research community and rekindled my latent love for academia. I&rsquo;ve been away from it for a long time. ICML 2016 is in NYC next year which is a lot closer to home. I&rsquo;ll definitely be going.</p>
<title type="html"><![CDATA[Providence: Failure Is Always an Option]]></title>
<link href=""/>
<content type="html"><![CDATA[<p><em>This post is part of a series on the Providence project at Stack Exchange. The first post can be found <a href="">here</a>.</em></p>
<p><a href="">The</a> <a href="">last</a> <a href="">five</a> <a href="">blog</a> <a href="">posts</a> have been a highlight reel of Providence’s successes. Don’t be fooled, though, the road to Providence was long and winding. Let’s balance out the highlight reel with a look at some of the bumps in the road.</p>
<p>I said the road was long. Let’s quantify that. The first commit to the Providence repo happened on February 13, 2014 and Providence officially shipped with <a href="">Targeted Job Ads for Stack Overflow</a> on January 27, 2015. We worked on Providence for about a year.</p>
<p>You read that right; we worked on Providence for about a year. I’m telling you that this system&hellip;</p>
<p><img class="center" src=""></p>
<p>&hellip;took a year to build. It’s kind of impressive, actually. The only <em>new</em> things on that diagram are the Daily Process, the Read API and a couple of redis instances. This certainly isn’t a big system by any means. We only sling around a few GB a day.</p>
<h2>So&hellip;What Happened?</h2>
<p>We <strong>failed slowly</strong> while <strong>solving the wrong problems</strong> trying to build a system that was <strong>too big</strong> on <strong>speculatively chosen technologies</strong>.</p>
<h3>Too Big</h3>
<p>When we started building Providence, we started with the idea that we were gonna keep data about persons for the most recent 365 days. Here’s a graph I made today:</p>
<p><img class="center" src=""></p>
<p>That’s showing the distribution of people we saw on February 11, 2015 by the last time we saw them. As you go further back in time the percentage of people who were last seen on that day drops off precipitously (that’s a log scale). Keeping a year of data around makes absolutely no sense if you have that graph. The potential benefit (a hit on an old person) is far outweighed by the operational cost of a system that could keep that much data around. We didn’t have that graph when we sized the system, we just picked a number out of the sky. Our choice of a year was an order of magitude too high.</p>
<h3>Speculative Technology Decisions</h3>
<p>The problem with keeping around a year of data is that&hellip;that’s a lot of data. We decided that SQL Server, the data store we had, was inadequate based on the fact that we were going to have to do key-value lookups on hundreds of millions of keys. We never really questioned that assertion or anything, it just seemed to go without saying.</p>
<p>This led us to a speculative technology decision. We “needed” a new datastore, so the task was to evaluate the ones that seemed promising. I spent about 2 weeks learning Cassandra, did some tiny proofs of concept, and without evaluating other options (aside from reading feature lists) decided that it was the way to go.</p>
<p>The terribleness of this decision would be borne out over the ensuing months, but far more terrible was this way of making decisions. Based on my most cursory of investigations of one technology I, with what can only be described as froth, was able to convince Stack Exchange Engineering and Site Reliability we needed to build a cluster of machines to run a distributed data store that I had all of two weeks of experience with. And we did it.</p>
<p>While I certainly appreciate the confidence, this was madness.</p>
<p>Another decision we made, apropos of nothing, was that we needed to update predictions for people every 15 minutes. This led us to decide on a Windows Service architecture, which wasn’t really our forte. In addition to that, we also were pretty frothy about using C#’s async/await as TheWay™ to do things, and we had <em>some</em> experience there but not a bunch.</p>
<h3>Solving the wrong problems</h3>
<p>Early on we spent a good deal of time on the offline machine learning aspect of Providence. This was one of the few things we got right. Even if we had all our other problems nailed down, Providence still wouldn’t be anything if the underlying models didn’t work. We knew the models had to work and that they were the hardest part of the problem at the beginning, so that’s what we worked on.</p>
<p>The Developer Kinds model was finished with offline testing in June. The next step was to get it tested in the real world. That next step didn’t happen for <em>four</em> months. The linchpin of the system sat on the shelf untested for four months. Why?</p>
<p>Some of the fail of speculative technology decisions is that you don’t realize exactly how much you’re giving up by striking out in a new direction. We put years of work into our regular web application architecture and it shows. Migrations, builds, deployment, client libraries, all that incidental complexity is handled just the way we like. Multiple SRE’s and devs know how most systems work and can troubleshoot independently. I wouldn’t go so far as to say as it’s <em>finely</em> honed, but it’s definitely honed.</p>
<p>It was folly to think that things would go so smoothly with new-and-shiny-everything. The Windows Service + async/await + new data store equation always added up to more incidental work. We needed to make deploying Windows Services work. Cassandra needed migrations. Our regular perf tools like MiniProfiler don’t work right with async/await, so we needed to learn about <a href="">flowing execution context</a>. If Cassandra performance wasn’t what we needed it to be we were in a whole new world of pain, stuck debugging a distributed data store we had little experience with.</p>
<p>And then it happened.</p>
<div style="text-align:center">
<blockquote class="twitter-tweet" lang="en"><p>The day we lost our innocence. <a href=""></a></p>&mdash; Jason Punyon (@JasonPunyon) <a href="">May 16, 2014</a></blockquote>
<script async src="//" charset="utf-8"></script>
<p>The datacenter screwed up and power to our systems was cut unceremoniously. When we came back up Cassandra was in an odd state. We tried repairing, and anything else we thought would fix it but ultimately got nowhere. After a few days we found a <a href="">bug report</a> that exhibited similar behavior. It’d been filed a few weeks earlier, but there was no repro case. The ultimate cause wasn’t found until a month after the original bug report was filed.</p>
<p><img class="center" src=""></p>
<p>This was the nail in the coffin for Cassandra. The fact that we got bitten by a bug in someone else’s software wasn’t hard to swallow. Bugs happen. It was the fact that were we in production, we’d have eaten a mystery outage for a month before someone was able to come up with an answer. It just proved how not ready we were with it and it made us uncomfortable. So we moved on.</p>
<h3>Speculative Technology Decisions Redux</h3>
<p>So what do you do when a bad technology choice bites you in the ass after months of work? You decide to use an existing technology in a completely unproven way and see if that works any better, of course.</p>
<p><img class="center" src=""></p>
<p>We still “knew” our main tools for storing data weren’t going to work. But we’d also just been bitten and we wanted to use something closer to us, not <em>as</em> crazy, something we had more experience with. So we chose elasticsearch.</p>
<h3>Solving The Wrong Problems Redux</h3>
<p>And lo, we repeated all the same wrong problem solving with elasticsearch. There was a smidge less work because elasticsearch was already part of our infrastructure. Our operational issues ended up being just as bad though. We were auditing the system to figure out why our data wasn’t as we expected it and rediscovered a <a href="">more-than-a-year-old bug</a>, that HTTP Pipelining didn’t work. We turned pipelining off and while elasticsearch acted correctly we saw a marked performance drop. We tried to optimize for another few weeks but ultimately threw in the towel.</p>
<p><img class="center" src=""></p>
<h3>Failing slowly</h3>
<p>Bad planning, bad tech decisions, and a propensity for sinking weeks on things only incidental to the problem all adds up to excruciatingly&hellip;slow&hellip;failure.
Failing is only kind of bad. Failing slowly, particularly as slowly as we were failing, and having absolutely nothing to show for it is <em>so</em> much worse. We spent months on datastores without thinking to ourselves “Wow, this hurts a lot. Maybe we shouldn’t do this.” We spent time rewriting client libraries and validating our implementations. We held off throwing in the towel until way too late way too often.</p>
<p>At this point, now mid September 2014, we <em>finally</em> realized we needed to get failing fast on these models in the real world or we weren’t going to have anything to ship in January. We gave up on everything but the models, and focused on how we could get them tested as quickly as possible. We dropped back to the simplest system we could think of that would get us going: Windows task scheduler invoked console applications that write data to files on disk that are loaded into redis. We gave up on the ill-chosen “a year of data” and “updates every 15 minutes” constraints, and backed off to 6 weeks of data updated daily.</p>
<p>Within <em>two weeks</em> we had real-world tests going and we got incredibly lucky. Results of the model tests were nearly universally positive.</p>
<h2>So what’s different now?</h2>
<p>Engineering made some changes in the way we do things to try to keep Providence’s lost year from happening again. Overall we’re much more prickly about technology choices now. Near the end of last year we started doing Requests For Comment (RFCs) to help our decision making process.</p>
<p>RFCs start with a problem statement, a list of people (or placeholders for people) representative of teams that are interested in the problem, and a proposed solution. RFCs publicize the problem within the company, help you to gauge interest, and solicit feedback. They are public artifacts (we just use Google Docs) that help surface the “how” of decisions, not just the “what”. It&rsquo;s still early going, but I like them a lot.</p>
<blockquote class="twitter-tweet" lang="en"><p>I need a stamp that reads &quot;I&#39;d Really Like This Done By Friday&quot;</p>&mdash; Kevin Montrose (@kevinmontrose) <a href="">February 6, 2014</a></blockquote>
<script async src="//" charset="utf-8"></script>
<p>Kevin and I have essentially become allergic to big projects. We attempt to practice &ldquo;What can get done by Friday?&rdquo; driven development. Keeping things small precludes a whole class of errors like &ldquo;We need a new datastore&rdquo;, &lsquo;cause that ain&rsquo;t gettin&rsquo; done by Friday. It&rsquo;s hard to sink a week on something you weren&rsquo;t supposed to be working on when all you have is a week.</p>
<title type="html"><![CDATA[Providence: Architecture and Performance]]></title>
<link href=""/>
<content type="html"><![CDATA[<p><em>This post is part of a series on the Providence project at Stack Exchange. The first post can be found <a href="">here</a>.</em></p>
<p>We’ve talked about how we’re trying to understand our users better at Stack Exchange and seen just how <a href="">big an impact</a> it’s had on our pilot project, the Careers Job Ads. Let’s take a look at the architecture of the system.</p>
<p><img class="center" src=""></p>
<p>This is the easy part, so let’s just get it out of the way. Providence has 2 workhorse servers (where we crunch data and make predictions), and 3 redis servers, configured like this:</p>
<h3>Workhorse Servers</h3>
<li>Dell R620</li>
<li>Dual Intel E5-2697v2 (12 Physical Cores @2.7GHz, 30MB Cache, 3/3/3/3/3/3/&frac34;/5/6/7/8 Turbo)</li>
<li>384GB RAM (24x 16GB @ 1600MHz)</li>
<li>2x 2.5&#8221; 500GB 7200rpm (RAID 1 - OS)</li>
<li>2x 2.5&#8221; 512GB Samsung 840 Pro (RAID 1 - Data)</li>
<h3>Redis Servers</h3>
<li>Dell R720XD</li>
<li>Dual Intel E5-2650v2 (8 Physical Cores @2.6GHz, 10MB Cache, 5/5/5/5/5/6/7/8 Turbo)</li>
<li>384GB RAM (24x 16GB @ 1600MHz)</li>
<li>2x 2.5&#8221; 500GB 7200rpm (RAID 1 - OS, in Flex Bays)</li>
<li>4x 2.5&#8221; 512GB Samsung 840 Pro (RAID 10 - Data)</li>
<p>If you’re a hardware aficionado you might notice that’s a weird configuration for a redis box. More on that (and the rest of our failures) in my next post.</p>
<p>Providence has 3 main parts: The Web Apps/HAProxy, The Daily Process, and the Read API.</p>
<h3>The Web Apps and HAProxy</h3>
<p><img class="center" src=""></p>
<p>Providence’s architecture starts with the problem of identity. In order to make predictions about people we have to know who’s who. We handle identity in the Stack Exchange web application layer. When a person comes to Stack Exchange they’re either logged in to an account or not. If they’re logged in we use their Stack Exchange AccountId to identify them. If they’re not logged in, we set their “Providence Cookie” to a random GUID with a far future expiration and we use that to identify the person. We don’t do panopticlicky evercookies or anything, because ew. We add the person’s identifier to a header in every response the web application generates.</p>
<p>HAProxy sits in front of all our web apps and strips the identity header off of the response before we return it to the user. HAProxy also produces a log entry for every request/response pair that includes things like the date, request URI and query string, and various timing information. Because we added the person’s identity to the response headers in the web application layer, the log also includes our identifier for the person (the AccountId or the Providence Cookie).</p>
<p>We take all these log entries and put them into our HAProxyLogs database, which has a table for each day. So for each day we have a record of who came to the sites and what URIs they hit. This is the data that we crunch in the Daily Process.</p>
<h3>The Daily Process</h3>
<p>At the end of each broadcast day we update Providence’s predictions. The Daily Process is a console application written in C# that gets run by the Windows Task Scheduler at 12:05AM UTC every morning. There are 6 parts to the Daily Process.</p>
<h4>Identity Maintenance</h4>
<p>Whenever a person logs in to Stack Exchange they will have 2 identifiers in the HAProxyLogs that belong to them: the ProvidenceCookie GUID (from when they were logged out) and their AccountId (from after they logged in). We need to make sure that we know these two identifiers map to the same person. To do that we have the web app output both identifiers to the HAProxyLogs whenever someone successfully logs in. Then we run over the logs, pull out those rows, and produce the Identifier Merge Map. This tells us which Providence Cookie GUIDs go with which AccountIds. We store the Identifier Merge Map in protocol buffers format on disk.</p>
<p>Identity maintenance takes about 20 minutes of the Daily Process. The merge map takes up about 4GB on disk.</p>
<h4>Tag Views Maintenance</h4>
<p>The second step of the Daily Process is to maintain Tag View data for each person. A person’s Tag View data is, for each tag, the number of times that person has viewed a question with that tag. The first thing we do is load up the Identifier Merge Map and merge all the existing Tag View data to account for people who logged in. Then we run over the day’s HAProxyLogs table pulling out requests to our Questions/Show route. We regex the question id out of the request URI, lookup that question’s tags and add them to the person’s Tag View data.</p>
<p>We keep all Tag View data in one of our redis instances. It’s stored in hashes keyed by a person’s identifer. The field names of the hashes are integers we assign to each tag, the values are the number of times the person has viewed a question with the associated tag.</p>
<p><img class="center" src=""></p>
<p>For each person we also maintain when their Tag View data was last updated, once in each person’s TagViews hash, and once again in a redis sorted set called “UpdatedDates” (more on this in a moment).</p>
<p>Tag Views maintenance takes about 20 minutes of the Daily Process. The Tag View data set + UpdatedDates fits in about 80GB of memory, and 35GB when saved to disk in redis’ RDB format.</p>
<h4>Making Predictions</h4>
<p><img class="center" src="" width="512"></p>
<p>The third step of the Daily Process is to update our predictions. Our models are static, and the only input is a person’s Tag View data. That means we only need to update predictions for people whose Tag View data was updated. This is where we use the UpdatedDates sorted set as it tells us exactly who was updated on a particular day. If we didn’t have it, we’d have to walk the entire redis keyspace (<a href="">670MM keys at one point</a>) to find the data to update. This way we only get the people we need to do work on.</p>
<p>We walk over these people, pull out their Tag View data and run it through each of our predictors: Major, Web, Mobile, and Other Developer Kinds, and Tech Stacks. We dump the predictions we’ve made onto disk in dated folders in protocol buffers format. We keep them on disk for safety. If we hose the data in redis for some reason and we don’t notice for a few days we can rebuild redis as it should be from the files on disk.</p>
<p>Making predictions takes 5 or 6 hours of the Daily Process. The predictions take up about 1.5GB on disk per day.</p>
<h4>Loading Predictions Into Redis</h4>
<p>The fourth step of the Daily Process is to load all the predictions we just made into redis. We deserialize the predictions from disk and load them into our other redis server. It’s a similar setup to the Tag View data. There’s a hash for each person keyed by identifier, the fields of the hash are integers that map to particular features (1 = MajorDevKind.Web, 2 = MajorDevKind.Mobile, etc), and the values are the predictions (just doubles) for each feature.</p>
<p><img class="center" src=""></p>
<p>Loading predictions takes about 10 minutes of the Daily Process. The predictions dataset takes up 85GB in memory and 73GB on disk in redis’ RDB format.</p>
<h4>Six Week Cull</h4>
<p>Providence’s SLA is split by whether you’re anonymous or not. We keep and maintain predictions for any anonymous person we’ve seen in the last 6 weeks (the vast vast majority of people). We keep and maintain predictions for logged in people (the vast vast minority of people) indefinitely as of 9/1/2014. So that means once an anonymous person hasn’t been updated in 6 weeks we’re free to delete them, so we do. We use the UpdatedDates set again here to keep us from walking hundreds of millions of keys to evict the 1% or so that expire.</p>
<p>The Six Week Cull runs in about 10 minutes.</p>
<p>The last part of the Daily Process is getting all the data out to Oregon in case we have an outage in New York. We backup our two redis instances, rsync the backups to Oregon, and restore them to two Oregon redis instances.</p>
<p>The time it takes to do Backup/Restore to Oregon varies based on how well our connection to Oregon is doing and how much it’s being used by other stuff. We notify ourselves at 5 hours if it hasn’t completed just to make sure someone looks at it and we’ve tripped that notification a few times. Most nights it takes a couple hours.</p>
<h3>The Read API</h3>
<p>Step four in the Daily Process is to load all our shiny new predictions into redis. We put them in redis so that they can be queried by the Read API. The Read API is an ASP.NET MVC app that turns the predictions we have in redis hashes into JSON to be consumed by other projects at Stack Exchange. It runs on our existing service box infrastructure.</p>
<p>The ad server project calls into the Read API to get Providence data to make better decisions about which jobs to serve an incoming person. That means the Read API needs to get the data out as quickly as possible so the ad renders as quickly as possible. If it takes 50ms to pull data from providence we’ve just eaten the entire ad server’s response time budget. To make this fast Read API takes advantage of the serious performance work that’s gone into our redis library <a href="">StackExchange.Redis</a>, and <a href="">Kevin&rsquo;s</a> JSON serializer <a href="">Jil</a>.</p>
<h3>The Workhorses</h3>
<p><img class="center" src=""></p>
<p>Excluding backups, the daily process takes about 7 hours a day, the bottleneck of which is cranking out predictions. One workhorse box works and then sits idle for the rest of the day. The second workhorse box sits completely idle, so we have plenty of room to grow CPU-hours wise.</p>
<p><img class="center" src=""></p>
<p>With redis we care about watching CPU load and memory. We can see there’s two humps in CPU load each day. The first hump is the merging of Tag View data, the second is when redis is saving backups to disk. Other than that, the redii sit essentially idle, only being bothered to do reads for the Read API. We aren&rsquo;t even using the third redis box right now. After we implemented the Six Week Cull, the Tag Views and Predictions data sets fit comfortably in RAM on one box, so we may combine the two we&rsquo;re using down to just one.</p>
<h3>The Read API</h3>
<p><img class="center" src=""></p>
<p>The big thing to watch in the Read API is latency. The Read API has a pretty decent 95th-percentile response time since launch, but we did have some perf stuff to work out. The Read API is inextricably tied to how well the redii are doing, if we’re cranking the TagViews redis core too hard, Read API will start timing out because it can’t get the data it needs.</p>
<p><img class="center" src=""></p>
<p>At peak the Read API serves about 200 requests/s.</p>
<h2>Failure is Always an Option</h2>
<p>That’s it for the architecture and performance of Providence. Up next we’ll talk about our failures in designing and building Providence.</p>
<title type="html"><![CDATA[Providence: Testing and Results]]></title>
<link href=""/>
<content type="html"><![CDATA[<p><em>This post is part of a series on the Providence project at Stack Exchange. The first post can be found <a href="">here</a>.</em></p>
<p>The Providence project was motivated by our desire to better understand our users at Stack Exchange. So we think we’ve figured out what kind of <a href="">developers come to our sites</a>, and <a href="">what technologies they’re using</a>. Then we figured out a way to combine all our features into the <a href="">Value Function</a>. How did we test these features?</p>
<p>After each feature worked in the clean room (it passed the muster on the training, validation, and test sets we&rsquo;d labeled) we had to drop it into some real-world mud to try and move a needle. Please attempt to contain your surprise as I tell you we chose to use ads as our mud puddle, and click-through rate as our needle.</p>
<p><img class="center" src=""></p>
<p>The particular ads we wanted to use were the Careers Job Ads that run in the sidebar of Stack Overflow. Last we left Careers Job Ads, we knew that <a href="">geography</a> was by far the most important feature we could use to select jobs to show you in the ads. We’ve done some visual updates since then, but the job selection algorithm has remained unchanged.</p>
<p style="text-align:center">
<img src=""> <img src="">
<p>We had a number of features in the pipeline at this point that were untested in real-world conditions. Major Dev Kinds, Minor Dev Kinds, Tech Stacks and Job Tags were the individual features we had to test. We did those first and then the final test was for the Value Function that combined each of those individual features and geography. Each test had these phases:</p>
<h3>Labeling Party: The Jobening</h3>
<p>We had a model for users, but we weren’t going to model jobs at the same time because then we wouldn’t know whether we were testing the user model or the job model. We also needed the cleanest labels on jobs that we could get which meant good ol’ fashioned home grown fresh squeezed human labels. Gotta give a big shout out to the developers at Stack Exchange for giving us their time to label each data set.</p>
<h3>Pre-analysis and Experiment Design</h3>
<p>This is a thing we’re getting more and more into here. We’ve wasted a lot of cycles on tests and ideas that would’ve been avoidable if we’d just looked at the data ahead of time. We needed to know approximately how many impressions we were going to get in the test groups so we would know how long we’d have to wait for the A/B results to converge. If it was going to take months for the results to converge for one of these experiments, we’d be better off just doing a different experiment. We had to know when we were going to have to give up controlling for important factors like geography in order to get enough data points in a bucket.</p>
<h3>Turning it up</h3>
<p>Once we knew how long we’d have to wait, we did some easing in. We’d run the test at 1% for a day to work out any bugs, monitor the early impression distributions and click-through rates to make sure they weren’t getting completely hosed and that we were also measuring correctly. Then we’d slowly ramp it up during the second day to 100% of usable impressions.</p>
<p><img class="center" src="" title="Just watching the hours tick byyyyyyyyyyyyy..." ></p>
<h3>Post Analysis</h3>
<p>At this point we’d have click-through rates that had supposedly converged, but it wasn’t enough that CTR just went up. We’d also have to make sure that gains on one subgroup weren’t at an unreasonable expense to the other groups. We called these checks “Degeneracy Passes”.</p>
<p>By the end, we were able to get through about a test per week. By the luck o’ the data scientist we even got some decent results.</p>
<p>The Value Function performs pretty admirably in comparison to the geography based targeting we were using before. We saw ~27% lift.</p>
<p>Looking at CTR improvement by geography, we found that jobs closer to the user improved less, confirming our previous experiments that geography is pretty important. Inside 60 miles we saw ~17% lift picking jobs using Value Function. Outside 60 miles we saw even larger gains.</p>
<p><img class="center" src=""></p>
<p>Looking at jobs by some popular dev kinds we saw the following gains:</p>
<p><img class="center" src=""></p>
<p>It wasn’t all kitten whispers and tickle fights. Desktop OSX jobs lost about 6% of their clicks, but we only had 9 of those jobs in the test. Similarly, SharePoint Integration jobs lost 100% of their clicks in the test, but there were only 3 of those jobs. I guess we’ll just have to live with those terrible blemishes for now.</p>
<p>Looking at jobs with a smattering of popular tags we saw these gains:</p>
<p><img class="center" src=""></p>
<p>Node really stands out there (I think we just got lucky), but we were pretty happy with all those gains. We didn’t find any losers among tags used on more than 50 jobs.</p>
<p><em>Next up: <a href="">The Architecture of Providence</a></em></p>
<title type="html"><![CDATA[A Wild Anomaly Appears! Part 2: The Anomaling]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>After all the rave reviews of my last post I knew you were just on the edge of your seat waiting to hear more about my little unsupervised text anomaly detector.</p>
<p><a href=""><img class="center" src="" /></a>
<a href=""><img class="center" src="" /></a>
<a href=""><img class="center" src="" /></a></p>
<p>So, we&rsquo;ve got some <a href="">working ML</a>! Our job&rsquo;s done right?</p>
<p><a href=""><img class="center" src="" /></a></p>
<p>&lsquo;Round these parts, it ain&rsquo;t shipped &#8216;til it&rsquo;s fast <em>and</em> it&rsquo;s got it&rsquo;s own chat bot. We spend all day in chat and there&rsquo;s a cast of characters we&rsquo;ve come to know, and love, and hate.</p>
<p><a href=""><img class="center" src="" /></a></p>
<p>Pinbot helps us not take each other&rsquo;s database migrations. You can say &ldquo;taking ###&rdquo; and Pinbot removes the last taken migration from this room&rsquo;s pins and replaces it with this one. So we always know what migration we&rsquo;re up to, even if someone hasn&rsquo;t pushed it yet. Also pictured: Me calling myself a dumbass in the git logs. Which gets broadcast by our TeamCity bot. Someone starred it.</p>
<h2>Hair on Fire</h2>
<p><a href=""><img class="center" src="" /></a></p>
<p>Hair on fire bot helps Careers keep making money. Hair on fire pops up every now and again to tell us a seriously critical error that affects the bottom line has happened on the site. If someone is buying something or on their way to buying something and gets an exception Hair On Fire dumps the details directly to chat so someone can get to work immediately.</p>
<p><a href=""><img class="center" src="" /></a></p>
<p>Here&rsquo;s another little Careers helper. We have a policy that any advertising images that go up on Stack Overflow <em>must</em> be reviewed by a real live person. When display advertising is purchased from our ad sales/ad operations teams, they do it. When we introduced <a href="">Company Page Ads</a> we automated the process of getting images on Stack Overflow and no one had to talk to a sales person. This begat LogoBot who reminds us to do our job and make sure no one&rsquo;s putting up animated gifs or other such tawdriness.</p>
<h2>Malfunctioning Eddie</h2>
<p><a href=""><img class="center" src="" /></a></p>
<p>Malfunctioning Eddie&rsquo;s&hellip;got problems.</p>
<h2>Anomaly Bot</h2>
<p>Which brings me to the Anomaly bot. We need to make sure that all these anomalous jobs I&rsquo;m detecting get in front of the sales support team. They are the human layer of detectors I alluded to in my last post who used to have to check over every single job posted to the board.</p>
<p><a href=""><img class="center" src="" /></a></p>
<p>There it is. Anomaly bot. Where does it lead us puny humans?</p>
<p><a href=""><img class="center" src="" /></a></p>
<p>Welcome to the super secret admin area of Careers. At the top of the page we have a list of the jobs that were posted today. There are 3 columns, the anomaly score (which is based solely on title), the job title, and a few buttons to mark the job anomalous or not. The second section of the page is for all jobs currently on the board.</p>
<p><a href=""><img class="center" src="" /></a></p>
<p>I&rsquo;m hoping the heatmap pops out at you. It runs from Red (pinned to the all-time most anomalous job) to Green (pinned to the all-time most middle-of-the-road job ever). The jobs posted today are light orange at worst, so that&rsquo;s pretty good! On the &ldquo;all jobs&rdquo; list there&rsquo;s a bunch of red that we need to review.</p>
<p>Just to give a reference, here was the first version sans heatmap.</p>
<p><a href=""><img class="center" src="" /></a></p>
<p>So much more information in that tiny extra bit of color. If you want to make simple heatmaps it&rsquo;s really easy to throw together some javascript that uses <a href="">the power of HSL</a>.</p>
<h2>What&rsquo;s Next?</h2>
<p>We&rsquo;re gonna let this marinate for a while to actually test my hypothesis that we only have to look at the top 10% of jobs by anomaly score. The sales support team&rsquo;s gonna clear the backlog in the &ldquo;all jobs&rdquo; section of the report, then use the tool for a little while and then we&rsquo;ll have the data we need to actually set the threshold. Once we do that the Anomaly bot can get a little smarter. Right now Anomaly bot just shows every three hours with that same dumb message. Maybe it&rsquo;ll only show up when there&rsquo;s jobs above our human-trained threshold (modulo a safety factor). Maybe we&rsquo;ll change it to pop up as soon as an anomalous job gets posted on the board.</p>
<h2>Here, have some code</h2>
<p>If you want to use the very code we&rsquo;re using right now to score the job titles it&rsquo;s up on <a href="">Nuget</a>, and the source is on <a href="">Github</a></p>
<p><a href=""><img class="left" src="" /></a></p>
<p>Got experience solving problems like this one? Wanna work at a place like <a href="">Stack Exchange</a>? Head on over to our completely middle-of-the-road <a href="">job listing</a> and get in touch with us.</p>
<title type="html"><![CDATA[A Wild Anomaly Appears!]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>So, I&rsquo;m working on the new Data Team at <a href="">Stack Exchange</a> now. Truth is we have no idea what we&rsquo;re doing (<a href="">WANNA JOIN US?</a>). But every now and then we come across something that works a little too well and wonder why we haven&rsquo;t heard about it before.</p>
<p>We run a <a href="">niche job board for programmers</a> that has about 2900 jobs on it this morning. Quality has been pretty easy to maintain. We have a great spam filter called &ldquo;The $350 Price Tag&rdquo;. Then we have some humans that look over the jobs that get posted looking for problems. Overall the system works well, but at 2900 jobs a month that means someone has to look through about 150 jobs every working day. They&rsquo;re looking for a needle in a haystack as most (>95%) of the jobs posted are perfectly appropriate for the board, so there&rsquo;s a lot of &ldquo;wasted&rdquo; time spent looking for ghosts that aren&rsquo;t there. And it&rsquo;s pretty boring to boot. I&rsquo;m sure that person would rather do other things with their time.</p>
<p>It&rsquo;d be nice if we had an automated way of dealing with this. We have no idea what we&rsquo;re doing, so we frequently just reach into our decidedly meager bag of tricks, pull one out, and try it on a problem. I&rsquo;d done that a few times before on this problem, trying Naive Bayes or Regularized Logistic Regression, but had gotten nowhere. There are a lot of different ways a job can be appropriate for the board and there are a lot of different ways a job could be not appropriate for the board which makes coming up with a representative training set difficult.</p>
<p>Last week while taking another whack at the problem I Googled &ldquo;Text Anomaly&rdquo; and came across David Guthrie&rsquo;s 186 page Ph. D. thesis, <a href="">Unsupervised Detection of Anomalous Text</a>. There&rsquo;s a lot there, but the novel idea was simple enough (and it worked in his experiments and mine) that I&rsquo;m surprised I haven&rsquo;t heard about it until now.</p>
<h3>Distance to the Textual Complement</h3>
<p>Say you have a bunch of documents. You pull one out and want to determine how anomalous it is with respect to all the others. Here&rsquo;s what you do:</p>
<li>Choose some features to characterize the document in question.</li>
<li>Convert the document to its feature representation.</li>
<li>Treat all the other documents as one giant document and convert that to its feature representation.</li>
<li>Calculate the distance between the two.</li>
<p>Do this for every document and sort the results descending by the distance calculated in step 4. The documents at the top of the list are the &ldquo;most anomalous&rdquo;.</p>
<p>That&rsquo;s it. Pretty simple to understand and implement. There are two choices to make: which features, and which distance metric to use.</p>
<h2>Obscurity of Vocabulary, I choose you!</h2>
<p>In any machine learning problem you have to come up with features to characterize the set of things you&rsquo;re trying to work on. This thesis is chock full of features, 166 of them broken up into a few different categories. This part of the paper was a goldmine for me (remember, I have no idea what I&rsquo;m doing). The text features I knew about before this were word counts, frequencies, tf-idf and maybe getting a little into part of speech tags. The kinds of features he talks about are stuff I never would&rsquo;ve come up with on my own. If you&rsquo;re doing similar work and are similarly lost, take a look there for some good examples of feature engineering.</p>
<p>The set of features that stood out to me the most were the ones in a section called &ldquo;Obscurity of Vocabulary Usage&rdquo;. The idea was to look at a giant reference corpus and rank the words in the corpus descending by frequency. Then you make lists of the top 1K, 5K, 10K, 50K, etc. words. Then you characterize a document by calculating the percentages of the document&rsquo;s words that fall into each bucket.</p>
<h2>Manhattan Distance, I choose you!</h2>
<p>Guthrie pits a bunch of distance metrics against eachother and for Distance to the Textual Complement method the <a href="">Manhattan distance</a> got the blue ribbon, so I used that.</p>
<p>When I&rsquo;ve been looking through the jobs before I can pretty much tell by their titles whether they&rsquo;re busted or not, so my documents were just the job titles. There isn&rsquo;t really a good reference corpus from which to build the Top-N word lists, so I just used the job titles themselves. I tried a couple different sets of Ns but ended up on 100, 300, 1000, 3000, and 10000 (Really ~7,000 as that&rsquo;s the number of unique terms in all job titles).</p>
<p><a href=""><img src="" /></a></p>
<p>Here&rsquo;s the sort of all the jobs that were on the board yesterday.</p>
<p><a href=""><img src="" /></a></p>
<p>Basically everything on the right is good and has low scores (distances).</p>
<p>Most of the jobs on the left have something anomalous in their titles. Let&rsquo;s group up the anomalies by the reasons they&rsquo;re broken and look over some of them.</p>
<h4>Stuff that just ain&rsquo;t right</h4>
<p>These jobs just don&rsquo;t belong on the board. We need to follow up with these people and issue refunds.</p>
<li>Supervisor Commercial Administration Fox Networks Group</li>
<li>Sales Executive Risk North Americas</li>
<li>Associate Portfolio Manager Top Down Research</li>
<li>Senior Actuarial Pre-Sales Consultant</li>
<li>Ad Operations Coordinator</li>
<li>NA Customer Service Representative for Sungard Energy</li>
<li>Manager, Curation</li>
<h4>Just Terrible Titles</h4>
<p>These jobs would belong, but the titles chosen were pretty bad. Misspellings, too many abbreviations, etc. We need to follow up with our customers about these titles to improve them.</p>
<li>Javascript Devlopers Gibraltar</li>
<li>Sr Sys Admin Tech Oper Aux Svcs</li>
<li>VR/AR Developer</li>
<h4>Duplicate Information</h4>
<p><img src="" alt="Duplicate Information" /></p>
<p>Anywhere you see the title for a job on <a href="">Stack Overflow</a> or <a href="">Careers</a> we also show you things like the location of the job, whether it&rsquo;s remote or not, and the company&rsquo;s name. These titles duplicate that information. We need to follow up with our customers to improve them.</p>
<li>Delivery Manager, Trade Me, New Zealand</li>
<li>Visualization Developer, Calgary</li>
<li>Technical Expert Hyderabad</li>
<li>Technical Expert Pune</li>
<li>Technical Expert Chennai</li>
<li>Technical Expert Gurgaon</li>
<li>New York Solutions Architect</li>
<li>Sr Fullstack Eng needed for Cargurus We reach over 10MM unique visitors monthly</li>
<li>Sr. Tester, Sky News, West London</li>
<li>Chief Information Officer/CIO</li>
<li>Machine Learning Engineer Part Time Remote Working Available</li>
<h2>What about the false positives?</h2>
<p>A number of false positives are produced (this is just a sample):</p>
<li>Computer Vision Scientist</li>
<li>Mac OSX Guru</li>
<li>Angularjs + .NET + You</li>
<li>Android Developer 100% Boredom Free Zone</li>
<li>Java Developer 100% Boredom Free Zone</li>
<li>DevOps Engineer - Winner, Hottest DC Startups!!! $10M Series A</li>
<li>Jr. Engineer at Drizly</li>
<p>Some of these (Computer Vision, Mac OSX) are just infrequently found on our board. Some of these people are trying to be unique (and are successful, by this analysis) so that their listing stands out.</p>
<p>Guthrie goes into a bit of detail about this in a section on precision and recall in the paper. His conclusion is that this kind of anomaly detection is particularly suited to when you have a human layer of detectors as the last line of defense and want to reduce the work they have to do. An exhaustive exploration of the scores finds that all of the jobs we need to follow up on are in the top 10% when ordered descending by their anomaly scores. Setting that threshold should cut the job our humans have to do by 90%, making them happier and less bored, and improving the quality of the job board.</p>
<title type="html"><![CDATA[So You Want a Zillion Developers&hellip;]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>I work at <a href="">Stack Overflow</a> on <a href="">Careers 2.0</a>. In addition to our job board we have a candidate database where you can search for developers to hire. Our candidate database has 124K+ developers in it right now.</p>
<p>Customers frequently gawk at this number because they&rsquo;ve looked at other products in the dev hiring space that offer <strong>millions</strong> of candidates in their databases. <a href=""></a> claims to have &ldquo;over 4 million developers&rdquo; in their database. <a href="">Gild</a> offers &ldquo;Over 6 Million Developers&rdquo;. <a href="">Entelo</a> will give you access to &ldquo;18+ million candidates indexed from 20+ social sites.&rdquo;</p>
<h3>Yeah man, your numbers stink</h3>
<p>Hey. That hurts.</p>
<p>Let&rsquo;s put those numbers in perspective. The vast majority of the developers &ldquo;in&rdquo; these other databases don&rsquo;t even know they exist. The devs never signed up to be listed or even indicated that they were looking for work. There isn&rsquo;t even a way to opt out. These databases are built by scraping APIs and data dumps from sites developers <em>actually</em> care about like Stack Overflow and GitHub.</p>
<p>On the other hand the <em>only</em> people you&rsquo;ll find in the Careers 2.0 database are ones who made the <em>affirmative choice</em> to be found. They start by getting an invitation to create a profile. They build out a profile with their employment and education history, open source projects, books they read, peer reviewed answers on Stack Overflow, and so on. Then they can choose to be listed as either an active candidate (they&rsquo;re in the market for a job right now) or a passive candidate (they&rsquo;re happy where they are but are willing to hear your offer). After a candidate gets hired they can delist themselves from the database so future employers don&rsquo;t waste any time on them.</p>
<p>So the difference between us and them is that we give you a smaller number of candidates who are already interested in job offers and they give you a giant database filled with hope and built by skeez.</p>
<p>We have some data from Careers that tells us hope is not a recruiting strategy.</p>
<h3>Our Experiment</h3>
<p>Careers 2.0 experimented with the &ldquo;index a bunch of people who don&rsquo;t know they&rsquo;re being indexed&rdquo; model to see if it could possibly work. We created what we called &ldquo;mini-profiles&rdquo; which consisted exclusively of already public information available on Stack Overflow. We would add mini-profiles to the database if the Stack Overflow user provided a location in their profile and had a minimum number of answers with a minimum score. We showed these mini-profiles along with our &ldquo;real&rdquo; candidates in search results. If an employer wanted to contact one of the people behind a mini-profile Careers 2.0 would send them an e-mail asking if they want to open up a conversation with the employer. If the candidate wanted to continue they could authorize us to share their contact information with the employer and they&rsquo;d start working on their love connection.</p>
<h3>Our Results</h3>
<p>We track response rates to employer messages to look out for bad actors and generally make sure the messaging system is healthy. A candidate can respond to a message interested/not interested or they can fail to respond at all. Response rate is defined as Messages Responded To / Messages Sent. When we compared the response rates of messages to mini-profiles to the response rates of messages to &ldquo;real&rdquo; profiles the results were not good for mini-profiles. <strong>Messages to &ldquo;real&rdquo; profiles were 6.5x more likely to get a response than messages to mini-profiles.</strong> That was the last and only straw for mini-profiles. We retired the experiment earlier this year.</p>
<h3>So what about the zillions of programmers?</h3>
<p>All those services I named at the beginning of this post do what we did in our experiment, just a little more extensively by including devs from more places online. I have to believe that the response rates from their unqualified leads are similar to the ones we found in our experiment. I suppose technically the response rates from randodevs on GitHub or Bitbucket could be higher than that of randodevs on Stack Overflow thus invalidating our conclusion, but anecdotal evidence from our customers about those other services suggests not.</p>
<p>&ldquo;Wait a sec there Jason,&rdquo; you&rsquo;re thinking, &ldquo;if their databases are at least 6.5x larger than yours I&rsquo;ll still get more responses to my messages right?&rdquo; Absolutely! That&rsquo;s called <strong>spam</strong>. You are totally allowed to go down the path of the spammer but let me hip you to the two problems there. The first problem with your plan is that devs hate recruiting spam more than they hate PHP, and they hate PHP alot. The word will get out that you&rsquo;re wasting everyone&rsquo;s time. <a href="">People will write about it</a>. The second problem is that spam is supposed to be cheap. This isn&rsquo;t cheap. In this case you&rsquo;ll have to spend at least 6.5x the time wading through these zillions of devs identifying the ones that meet your hiring criteria, messaging them, and waiting for responses. So not only are you wasting their time, you&rsquo;re wasting yours.</p>
<p>We aren&rsquo;t going to build this business off hope and spam and wasting people&rsquo;s time. If a smaller database is the price, so be it.</p>
<title type="html"><![CDATA[Commuting: A Perverse Incentive at Stack Exchange]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>So, we just went through comp review season here at the Stack Exchange. This is pretty much the only time of year we talk about money, because that&rsquo;s the way we want it. We pay people enough to be happy and then shut up about it. You&rsquo;ll probably only ever hear stuff about comp from me around September each year because that&rsquo;s the only time it&rsquo;s on my mind. The system works, and I&rsquo;m generally happy about my financial situation, but we have a comp policy about remote work that subjects me to a bit of a perverse incentive when it comes to commuting.</p>
<p>The policy is that if you work out of the New York office, you get a 10% pay increase relative to what you&rsquo;d be making if you worked remote. The reason for this has always been a little cloudy to me. I&rsquo;ve heard cost of living adjustment. I&rsquo;ve heard we want to incentivize people to be in the office because of &ldquo;accidental&rdquo; innovation from pick-up meetings and conversations in the hall. Regardless of the reason, that&rsquo;s the policy.</p>
<p><img class="right" src="" width="400" title="Elle gettin' her redis on" ></p>
<p>I live in Stamford, CT and have been commuting to the New York Office 3 days a week (down from 5) since my daughter Elle was born in December. My total commute time averages just under 3 hours a day (10 min from my house to the Metro North, 55 minutes to Grand Central, 20 minutes from Grand Central down to the Office). So I end up commuting about 36 hours per month (down from 60).</p>
<p>On the Metro North getting a seat means cramming in next to one or two other people in side-by-side seats leaving little elbow room for typing (or living, FSM forbid they&rsquo;re overweight), sitting in the seats that face each other and knee-knock with people who are drawn from a population with a mean height of 7 feet, or sitting on the floor in a vestibule near the doors. Some days the Metro North crawls because apparently they didn&rsquo;t design this surface rail line to deal with even the slightest amount of rain. The subway is the subway, you get what you get. This commute stinks and it&rsquo;d be my default position to forgo it.</p>
<p>Here&rsquo;s where the perversion comes in. Let&rsquo;s say I make $120K a year (I&rsquo;m using this number because the math works out simply) out of the New York Office and decide to go remote. Every month I&rsquo;ll make $1K less and get 36 hours of my life back. So Stack Exchange thinks my commute is worth $27.78 an hour. 4x minimum wage for no productive output is nice work if you can get it.</p>
<blockquote><p>When done right, it makes people extremely productive. Private office? Check. Flexible hours? Check. Short commute? Check. I’ll let you in on a secret: most of our remote developers work longer hours than our in-office devs. It’s not required, and probably won’t always be the case, but when going to work is as simple as walking upstairs (pants optional, but recommended) people just tend to put in more hours and work more productively.</p><footer><strong>David Fullerton</strong> <cite><a href=''>;</a></cite></footer></blockquote>
<p>Going remote means a large portion of the 36 hours a month I spend commuting would go back to productive work (I won&rsquo;t lie, some of it will be spent enjoying time with my daughter) so Stack Exchange is better off. I&rsquo;d be happier because I get to skip the dreadful commute and work instead so I&rsquo;d be better off. But I don&rsquo;t make nearly enough that I can just drop 10% of my pay and not feel it.</p>
<title type="html"><![CDATA[Fun With RNGs: Calculating &#960;]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>So, calculating &#960; is a fun pastime for people it seems. There are many ways to do it, but this one is mine. It&rsquo;s 12 lines of code, it wastes a lot of electricity and it takes forever to converge.</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
</pre></td><td class='code'><pre><code class='csharp'><span class='line'><span class="k">public</span> <span class="kt">double</span> <span class="nf">EstimatePi</span><span class="p">(</span><span class="kt">int</span> <span class="n">numberOfTrials</span><span class="p">)</span>
</span><span class='line'><span class="p">{</span>
</span><span class='line'> <span class="kt">var</span> <span class="n">r</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Random</span><span class="p">();</span>
</span><span class='line'>
</span><span class='line'> <span class="k">return</span> <span class="m">4</span> <span class="p">*</span> <span class="n">Enumerable</span><span class="p">.</span><span class="n">Range</span><span class="p">(</span><span class="m">1</span><span class="p">,</span> <span class="n">numberOfTrials</span><span class="p">)</span>
</span><span class='line'> <span class="p">.</span><span class="n">Select</span><span class="p">(</span><span class="n">o</span> <span class="p">=&gt;</span> <span class="p">{</span>
</span><span class='line'> <span class="kt">var</span> <span class="n">x</span> <span class="p">=</span> <span class="n">r</span><span class="p">.</span><span class="n">NextDouble</span><span class="p">();</span>
</span><span class='line'> <span class="kt">var</span> <span class="n">y</span> <span class="p">=</span> <span class="n">r</span><span class="p">.</span><span class="n">NextDouble</span><span class="p">();</span>
</span><span class='line'> <span class="k">return</span> <span class="n">Math</span><span class="p">.</span><span class="n">Pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="m">2</span><span class="p">)</span> <span class="p">+</span> <span class="n">Math</span><span class="p">.</span><span class="n">Pow</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="m">2</span><span class="p">)</span> <span class="p">&lt;</span> <span class="m">1</span> <span class="p">?</span> <span class="m">1</span> <span class="p">:</span> <span class="m">0</span><span class="p">;</span>
</span><span class='line'> <span class="p">})</span>
</span><span class='line'> <span class="p">.</span><span class="n">Average</span><span class="p">();</span>
</span><span class='line'><span class="p">}</span>
<p>What&rsquo;s going on here? First we initialize our random number generator. Then for 1 to the number of trials we specify in the argument we do the following:</p>
<li>Generate two random numbers between 0 and 1. We use one for the X coordinate and one for the Y coordinate of a point.</li>
<li>We test if the point (X,Y) is inside the unit circle by using the formula for a circle (x<sup>2</sup> + y<sup>2</sup> = r<sup>2</sup>).</li>
<li>If the point (X,Y) is inside the circle we return a 1 otherwise a zero.</li>
<p>Then we take the average of all those zeros and ones and multiply it by a magic number, 4. We have to multiply by four because the points we generate are all in the upper right quadrant of the xy-plane.</p>
<p>How bad is it? Here&rsquo;s some output:</p>
<pre><code> Number Of Trials Estimate of Pi
10 3.6
100 3.24
1000 3.156
10000 3.1856
100000 3.14064
1000000 3.139544
10000000 3.1426372
100000000 3.14183268
1000000000 3.141593 (Took 2:23 to complete)
<title type="html"><![CDATA[Things That, Were I to Unfortunately Wake Up Tomorrow as a Recruiter, I Would Never Do]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>I would never send e-mails that make potential candidates for a position think I&rsquo;m not effective at finding potential candidates for a position. Giving candidates that impression just makes them think I stink at everything else too.</p>
<p><img src=""></p>
<blockquote><p>Subject: Barbara Nelson in Search of Javascript Expertise</p></blockquote>
<p>Do you mean <strong>the</strong> Barbara Nelson?</p>
<blockquote><p>Hello from Barbara!</p></blockquote>
<p>What a great salutation! Not. Save that one for your next family newsletter.</p>
<blockquote><p>I saw your profile either on github or on stackoverflow</p></blockquote>
<p>Really? WOW! It sounds like you did a lot of research on me and moreover you&rsquo;re the kind of go-getter who keeps the relevant information she needs at her fingertips at all times.</p>
<blockquote><p>I am looking for several strong JavaScript Object-Oriented Engineers (not “just” web developers). These three openings have been especially challenging to fill&hellip;</p></blockquote>
<p>Well alright let me click through and see what these jobs are about. Oh&hellip;no company names? The third one is really a C++ job? And you say you&rsquo;re having trouble filling these positions?</p>
<blockquote><p>Some JavaScript opportunities I am helping to fill are at solid funded start-ups, some are at start-ups already acquired by a well-known global company with solid benefits. We can make your relocation to the beautiful Bay Area happen if there’s a good fit.</p></blockquote>
<p>That&rsquo;s good I guess&hellip;I&rsquo;m not really that interested in moving to the Bay Area.</p>
<blockquote><p>Those who are interested in a brief discussion on the phone: please send a resume or an online profile that reflects your experience, a good time to talk, and a good phone number, and we’ll schedule a quick call.</p></blockquote>
<p>Those who sent this e-mail should learn how to address the recipient directly and singularly instead of giving the impression that this is just another useless e-mail blast from a contingency recruiter.</p>
<blockquote><p>If you never want to hear about career opportunities from me again, just let me know; reply and say so.</p></blockquote>
<p>By the way you almost whited that out I&rsquo;d almost think you didn&rsquo;t want me to actually do that.</p>
<blockquote><p>I love referrals.</p></blockquote>
<p>I love how I almost don&rsquo;t even get the feeling you&rsquo;re trying to get me to do your job for you.</p>
<blockquote><p><img src=""></p></blockquote>
<p>So let&rsquo;s look an e-mail with a similar goal.</p>
<p><img src=""></p>
<blockquote><p>Subject: Facebook Engineering</p></blockquote>
<p>Do you mean <strong>the</strong> Facebook? Let&rsquo;s not be unfair to poor Barbara. Her subject line is much harder to get right than this one.</p>
<blockquote><p>I hope all is well. I had the pleasure of stumbling upon your information online and saw that you have been working on some pretty neat stuff with Stack Overflow and various companies (it wasn&rsquo;t disclosed on your resume) plus you have an awesome academic background from SUNY Geneseo to complement it.</p></blockquote>
<p>This is <em>much</em> better than what Barbara had to say about me. Minimally Jeremy has read my public <a href="">Careers 2.0 profile</a> and noted my current position and where I went to school. He also called out the fact that I don&rsquo;t list the companies I&rsquo;ve worked at before on my profile (mainly so I can write about my experiences there when I want to without anyone getting bent out of shape). This e-mail is about me. It&rsquo;s not a cattle call.</p>
<blockquote><p> I am currently helping grow our engineering team in the NYC office and would love to chat with you about what you&rsquo;ve been up to and perhaps put us on your radar; if nothing else we can have a friendly conversation. Let me know what works for you and we can schedule a time at your convenience. If this isn&rsquo;t the right time, I completely understand and we can stay in touch based on your schedule – no rush. I look forward to hearing from you.</p></blockquote>
<p>Great tone. Sounds like a human. He tells me what he&rsquo;s after while being accomodating and not pushy. He makes me believe that if I respond, <em>he&rsquo;s going to respond back</em>. Jeremy could&rsquo;ve broken some of this down into paragraphs to make it less <strong>WALLOFTEXT</strong> but other than that it was a decent recruiting e-mail.</p>
<title type="html"><![CDATA[Get Your Redis &#79;n on Windows]]></title>
<link href=""/>
<content type="html"><![CDATA[<p><strong>TL;DR</strong>: Want a virtual machine running redis in however long it takes you to download 400MB + a little completely automated install time? Follow the instructions <a href="">here</a> and you&rsquo;ll be on your way.</p>
<p>Well, it only took me a year of having this blog for me to write up something even remotely technical. But here you are, and here I am&hellip;so let&rsquo;s just tone down the celebration a little bit and get on with it already.</p>
<p>So&hellip;it&rsquo;s hard running a dev environment sometimes. We at the <a href="">Stack Exchange</a> will use anything to get the job done, but on the programmer&rsquo;s side we&rsquo;re mainly a windows shop. One piece of software we&rsquo;ve come to know and love is <a href="">Redis</a> though. We love it so much we&rsquo;ve got <a href="">antirez</a> on speed dial. It&rsquo;s really the greatest.</p>
<p>Here&rsquo;s where it isn&rsquo;t quite the greatest though (for us): it&rsquo;s really meant to run on Linux. Some people have made mostly working <a href="">windows builds</a> in the past that were good enough for dev&#8217;ing on but had weird behavior when it came to background operations. They&rsquo;re great and I appreciate the work they d(o|id), but they fall behind when redis bumps stable versions (it&rsquo;s behind 1 stable version right now leaving out features like the Lua scripting engine). Microsoft went through the rigamarole of <a href="">patching redis</a> so that it will run on windows, but that patch isn&rsquo;t getting merged to master&hellip;ever.</p>
<p>So what&rsquo;s a girl to do? When I&rsquo;ve been on a team of one and had this kind of problem I thought to myself, &ldquo;Self! Get VMWare on here, spin up a one off VM with ubuntu and just run it there! Problem Solved!&rdquo; and many internal high-fives were had. But when you&rsquo;re on a team of 6 (the <a href="">Careers</a> team, plug: <a href="">we&rsquo;re hiring</a>) that doesn&rsquo;t really scale well. So what are my choices? Let&rsquo;s go to the big board of options:</p>
<li>Just tell my teammates &ldquo;Hey, spend a couple hours spinning up your own VM and hope the one you have and the one I have match up and behave exactly the same&rdquo;. (HINT: No)</li>
<li>Check in a 10 gig VM into source control and push so the other members of the team can run it too? (HINT: NO. That&rsquo;s an example of what we call the &ldquo;I quit&rdquo; check-in.)</li>
<p>So how do you solve this problem?</p>
<h3>Enter the Hobo</h3>
<p>So it turns out a bunch of other people have this problem too (WEIRD RITE?). A <a href="">smart dude</a> decided to solve it and created <a href="">Vagrant</a>. Vagrant is a super simple yet powerful way to create and manage a reproducible virtual dev environment. Check in a couple kB of config and you get a virtual machine (or a multiple machine environment) your whole team can run. Vagrant wraps around <a href="">Virtual Box</a> for it&rsquo;s virtual machines and it&rsquo;s not just for windows. It runs on Linux and Mac too. Let&rsquo;s run it down.</p>
<h4>Getting in Installed</h4>
<p>Follow the <a href="">startup guide</a> here. It&rsquo;s basically install VirtualBox and install Vagrant.</p>
<h4>Creating a machine</h4>
<p>To create the machine, the first thing we do is create your Vagrantfile. Don&rsquo;t worry&hellip;it isn&rsquo;t a driter fetish. It&rsquo;s just a config file that outlines how your virtual machine is setup. It&rsquo;s also just a bit of ruby. Here&rsquo;s the one we&rsquo;re using:</p>
<p>File /Users/jpunyon/code/ could not be found</p>
<p>So first we tell Vagrant which box to use. A <a href="">box</a> is essentially a map from a key to a file. Box names can be anything you want, in this case I just have a name telling me that it&rsquo;s ubuntu&rsquo;s latest 64-bit release.</p>
<p>Next we have a url that points to a file. As it says in the comment there, this url points to a box file that will be downloaded if the box with the name in <code></code> doesn&rsquo;t exist. This is nice because it means i send the file and when my teammate runs it it will go fetch everything it needs to create the virtual machine. Brilliant. A bunch of base boxes can be found at <a href=""></a>. They have many different guest operating systems and versions and such to use. Very cool.</p>
<p>Next we have some port forwarding settings. Vagrant takes care of setting up the network for you, you just have to tell it what you need. So I&rsquo;m just forwarding to port 6379 on the guest machine (the default port on which redis runs) from port 6379 on my host machine.</p>
<p>Next I <a href="">customize</a> the vm to have a gig of memory instead of whatever the base box has by default.</p>
<p>So that&rsquo;s it for the configuration the box. The last line runs a provisioner which will setup the box once it&rsquo;s running. There are a number of provisioners to choose from including Puppet, Chef and shell. This was the gotcha for me when I was doing it the first time. The docs list the provisioners in this order&hellip;</p>
<p><img class="center" src=""></p>
<p>So I spent an hour or two trying to grok the chef and puppet docs and ended up getting frustrated. Those systems have a bunch of abstractions in them which probably make them great for doing sys-adminny type stuff but in my head I was screaming &ldquo;AAAARRRGH. JUST LET ME RUN A FUCKING SHELL SCRIPT!&rdquo;. Of course I go back to the vagrant docs after that, look an inch or two down and feel like an idiot. I do wish the bullet points there went in order of increasing complexity though.</p>
<p>Long story short, the provisioner just executes the specified shell script on the guest box after it boots up.</p>
<h4>Shellack It</h4>
<p>So once the machine boots up what do we want it to do? Well, this:</p>
<p>File /Users/jpunyon/code/ could not be found</p>
<p>So first we make the directories where redis will live. Then we go to the top level one, download and extract the code for the version of redis we&rsquo;re interested in and build it. Then we copy the resulting executables to their final home in <code>/opt/redis/bin</code>.</p>
<p>Next we copy an init.d script to where it needs to be, then we copy the redis configuration to where it lives. Add a redis user, start redis and we&rsquo;re all finished.</p>
<p>You might be asking &ldquo;How did that init.d script and redis configuration get into the vagrant directory on the guest box?&rdquo;</p>
<p>The way you run vagrant is by going to the directory where the Vagrantfile lives and typing <code>vagrant up</code>. That starts the whole ball rolling. When vagrant starts up your VM, it automatically shares the directory where the Vagrantfile is with the guest box at <code>/vagrant</code> on the guest box. It&rsquo;s a magical default behavior.</p>
<h3>So that&rsquo;s pretty much it</h3>
<p>Well, for now anyway. Vagrant can be used to setup multiple machine environments (which I might do next to test out an elasticsearch cluster for Careers). It has many more bells and whistles to keep your virtual dev environment running lean and mean. I&rsquo;ve been super impressed with just how easy it is to work with (total home grown code to get my redis VM up was 31 lines, 15 of which were the shell script for installing redis) and bonus everyone on my team thinks I&rsquo;m a hero. It&rsquo;s that magical.</p>
<p>+1 Vagrant&hellip;+1.</p>
<p>This is the init.d script I used which I cribbed from <a href="">Ian Lewis</a>.</p>
<p>File /Users/jpunyon/code/ could not be found</p>
<title type="html"><![CDATA[Geography's the Fuck.]]></title>
<link href=""/>
<content type="html"><![CDATA[<p><a href="">Raganwald</a> poses an interesting question. Why do some of the best minds in our industry spend time figuring out how to make people click more on ads? Aren&rsquo;t there more interesting problems for these bright up-and-comers to spend their valuable time and insight on?</p>
<p>One simple answer to the question is <strong>Geography</strong>.</p>
<h3>How do you know this?</h3>
<p>I&rsquo;ve spent a little time figuring out how to make people click more on ads :)</p>
<p>I work at <a href="">Stack Exchange</a> on our <a href="">Careers 2.0</a>. We try to make it easier for programmers to get better jobs. We have two main ways of getting people to the Careers 2.0 website. Our users evangelize on our behalf by inviting their programmer friends to show off their accomplishments on their own Careers profiles (like mine <a href="">here</a>).</p>
<p>The second way looks like this</p>
<p><img class="center" src=""></p>
<p>Me and the team spent a bit of time trying to figure out how to get programmers to click on this ad. It (and it&rsquo;s smaller variant) shows up on the vast majority of the pages of Stack Overflow.</p>
<p>The <strong>number one</strong> way to make people click on this ad is to show them a job from a <strong>place near them</strong>. It&rsquo;s simple as that. Showing the user a job near them outperforms every other way of constructing the ad we could come up with by a factor of between <strong>2 and 5</strong>.</p>
<p>Location is still <em>incredibly</em> important to job seekers (and employers). If our ad analysis is to be believed, they are more interested in having a job near them than having a job that matches their skills. Or a job that is interesting. Or a job where they work on &ldquo;super important&rdquo; problems.</p>
<p>I know this to be true from my own experience. I spent the summers of my college years working in Berkeley, CA for Lawrence Livermore National Laboratory. We did low energy nuclear physics experiments measuring certain reaction <a href="">cross sections</a> on radioactive nuclei. Some of this work gets used downstream by the <a href="">Stockpile Stewardship</a> program. Stockpile Stewardship is the program responsible for maintaining the reliability of the United States nuclear arsenal. The United States doesn&rsquo;t test nuclear weapons anymore and this program ensures that the ones we have continue to work, and ascertain the failure modes of the ones that aren&rsquo;t going to work. I think this would qualify as an important problem to Raganwald (and most other people). Most of the work <em>I</em> did was on software for simulating our particle and gamma ray detectors.</p>
<p>After college was over I ended up back in New York because that&rsquo;s where my <em>life</em> was. I worked at a management consulting company and then a couple hedge funds before I got my head out of my ass and realized I wanted to work on something that mattered a little bit more than the continued aggrandizement of the uber-rich. By then I&rsquo;d been married for a few years. My wife had a job in New York, my family was in New York (and elsewhere on the East coast). If I was going to work on something that mattered, it was going to have to be something that mattered&hellip;in New York.</p>
<h3>Wait&hellip;haven&rsquo;t we solved this?</h3>
<p>But, what about technology? Hasn&rsquo;t technology solved the geography problem? Don&rsquo;t we have Skype, company chat, Google hangouts and <em>shudder</em> the lowly telephone to connect people remotely? All these media come with their own problems. You need people who are exceptionally good at communicating through electronic media for these solutions to work. It is <strong>REALLY HARD</strong>. Stack Exchange was founded remotely. We built our own <a href="">chat system</a> because others were inadequate. We hire a bunch of remote devs, sys admins and others. I do internal support for people in 3 timezones (that&rsquo;s more than any of my international hedge fund jobs). A lot of people aren&rsquo;t cut out for it. I&rsquo;ll admit it, sometimes I&rsquo;m not.</p>
<p>Long story short it&rsquo;s really easy to say &ldquo;Hey we&rsquo;ll just hire the best people remotely.&rdquo; but much harder to do in practice. Your culture has to be just so.</p>
<p>Unfortunately you and the &ldquo;right job for you&rdquo; (for some criteria of right) are star-crossed. At all times there&rsquo;s an ordered list of jobs that are a best match for you based on all factors. Rarely anyone has the Juliet at the top of their list. You and your Romeo pass eachother silently (and not so silently) in the night for myriad reasons. Transaction costs for switching jobs are high. People need certainty. I&rsquo;m harping on geography here, but it&rsquo;s merely one of the reasons people aren&rsquo;t working &ldquo;the right&rdquo; job or fixing &ldquo;the right&rdquo; problems.</p>
<p>Geography is important to programmers. It&rsquo;s probably at the top of the list of factors that goes into deciding whether a job is &ldquo;right&rdquo;. If the company working on &ldquo;important&rdquo; problems doesn&rsquo;t jive with your geography, you&rsquo;re probably going to leave it by the wayside.</p>
<p>Thanks to <a href="">Matt Jibson</a> for an edit.</p>
<title type="html"><![CDATA[A Guy Walks Into an Apple Store...]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>So me and my wife had a babby recently. Unfortunately my wife had some preterm labor around week 33 and we had to spend about 10 days in the hospital. Don&rsquo;t worry, everything turned out all right (see perfection below), but I was pretty burnt out after the 10 days in the hospital. My wife was happy with my performance during our mini-crisis and she told me to go indulge a little. I&rsquo;d been super excited about the iPhone 5 release, so I went and picked one up.</p>
<p><img class="center" src="" width="250" title="&#34;Perfection&#34;" alt="&#34;Perfection&#34;"></p>
<p>Fast forward a few weeks and a couple things started going wrong with the phone. The first thing you can probably see from Elle&rsquo;s picture. Let me blow it up for you&hellip;</p>
<p><img src=""></p>
<p>That&rsquo;s not a birthmark, something was wrong with the camera. It showed up the week my wife was due. I had slightly more important things happening so it just went on the ever growing pile of stuff I needed to do later.</p>
<p>Then right after the babby was born another problem popped up. I was talking to my sister on the phone at CVS and the receiver just cut out. Tried calling back, no dice. The speakerphone worked and if I used the headphones I got sound but the receiver was completely borked. But then a day later I picked up the phone and called someone and voila the receiver was working again. Weird. Cut to a few days later and it was out again. So the problem was intermittent.</p>
<p>I&rsquo;d never had a problem with an Apple product before, so I had no idea how their customer service was in situations like this. I assumed the worst. I worried they were going to tell me it was my fault and they had never seen problems like this and if I want to get it fixed it&rsquo;d cost ${hefty sum}. So as I usually do when I think I&rsquo;m going in against a company that&rsquo;s going to try to screw me I went about building my case.</p>
<p>I started searching around for problems like mine. I found a <a href=";tstart=0">couple</a> <a href="">threads</a> talking about the earpiece problem. So it wasn&rsquo;t just me or my phone. I fired up my iPad and loaded up those bookmarks in safari and headed to the store. I didn&rsquo;t have a plan for the camera discussion, but the earpiece case seemed pretty strong. I figured I&rsquo;d lead with that, then throw the camera problem in later.</p>
<p>I drove to the store, steeling myself to deal with crappy customer service and a maybe dim/maybe dumb rep. Went over the plan in my head a couple times and thought about how to deal with refutations. I arrived, walked up to the genius bar and made an appointment. They told me it&rsquo;d be an hour so I cruised around the mall for a while and came back. I bellied up to the bar to meet my genius with my first rehearsed line on the tip of my tongue ready to start sparring.</p>
<p>She asked me what the problem was and I demonstrated it for her by pressing play on one of my voicemails and holding the phone up to her ear. &ldquo;See, no sound.&rdquo; I said. Then before I was able to say anything else she whisked my phone away into the back and came back a minute later. &ldquo;Yep. I confirmed there was no sound. We&rsquo;re going to go ahead and replace your phone.&rdquo;</p>
<p>But&hellip;but&hellip;I had all my plans to deal with a crappy customer service rep and get outraged about some ridiculous policy. I practiced arguments. I didn&rsquo;t even get to show you my links!</p>
<p>I had dealing-with-bad-customer-service-blue-balls. I was so ready for an argument that just never materialized. So what do you do when you get what you want? YOU SHUT UP and let them give it to you. I almost made it but broke while she was setting up the new phone. I tossed out a &ldquo;Is this kind of thing common? I read about it on blah blah blah&rdquo; nonchalantly, half making conversation but mostly just needing to get out some of that pent up preparation frustration. I picked up my brand new working phone and headed out happy as a school girl.</p>
<h3>Compare and Contrast</h3>
<p>A few days later I picked up <a href=";qid=1358008564&amp;sr=8-1&amp;keywords=Influence">Influence: The Psychology of Persuasion</a>. This isn&rsquo;t usually the kind of thing I read, but I was really impressed with a guy I learned about named Apollo Robbins. It was reported he was enamored with the book so I grabbed it on my kindle and started reading.</p>
<iframe width="420" height="315" src="" frameborder="0" allowfullscreen></iframe>
<p>Chapter 1 of this book talks about a bunch of stuff, but the thing that caught my eye was the contrast principle.</p>
<blockquote><p>There is a principle in human perception, the contrast principle, that affects the way we see the difference between two things that are presented one after another. Simply put, if the second item is fairly different from the first, we will tend to see it as more different than it actually is. So if we lift a light object first and then lift a heavy object, we will estimate the second object to be heavier than if we had lifted it without first trying the light one.</p><footer><strong>Robert B. Cialdini</strong> <cite>Influence: The Psychology of Persuasion</cite></footer></blockquote>
<h3>So What&rsquo;s The Point?</h3>
<p>OK, so I&rsquo;ve had you reading this pedestrian story about increasing the planet&rsquo;s population, things working out splendidly at the Apple store, a cool video on the internet and a 6 year old book for 13 paragraphs now. Here&rsquo;s the point:</p>
<p><strong>Having good customer service is even <em>more powerful</em> than you think</strong>.</p>
<p>I know&hellip;that came out of nowhere, right?</p>
<p>Things go wrong with company&rsquo;s products. This makes customers worry. Not just because the thing they have is broken, but because there&rsquo;s so much bad customer service out there and they now have to go interact with it. Cable companies, utility companies, <a href="">these guys</a>, the MTA, and the DMV are all out there creating bad customer service experiences.</p>
<p>When something goes wrong with your product your customer gets worried. They&rsquo;re primed by all the bad customer service experiences they&rsquo;ve ever had and they call you up or walk into your store. At this point you have a tremendous opportunity to take advantage of the contrast principle. They&rsquo;ve presented themselves with all the bad service they&rsquo;ve ever gotten worrying if you&rsquo;ll be the same. They&rsquo;re aiding and abetting your success before they even talk to you. Present them with just adequate service and because of the amplifying effect of the contrast principle they&rsquo;ll think you&rsquo;re the greatest thing since that last thing that was so great.</p>
<p>Now imagine how over the moon your customers will be when you empower your customer service team to go the <em>extra</em> mile and literally do everything it takes to make them happy&hellip;</p>
<title type="html"><![CDATA[Rock Stars Went Where?]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>So I read <a href="">this</a> article today about how the tech industry is too elitist and thinks everyone who&rsquo;s good is already rich and can follow their bliss until the cows come home waiting for the olympian software companies of the world to come hire them. I just wanted to provide a data point from Stack Exchange. I&rsquo;m not gonna bloviate that we&rsquo;re all rock stars at Stack Exchange (I wouldn&rsquo;t even describe myself like that) but here&rsquo;s the list of schools attended by all the programmers and sysadmins at Stack Exchange:</p>
<p>Georgia State University<br/>
Arcadia University<br/>
Metro State College of Denver<br/>
Technische Universität Clausthal<br/>
University of Texas<br/>
University of Exeter<br/>
Boston University<br/>
University of New South Wales<br/>
Rensselaer Polytechnic Institute<br/>
State University of New York at Geneseo (Plug: The <a href="">physics</a> department is awesome.)<br/>
University of Pittsburgh<br/>
Colorado State University<br/>
New England Conservatory of Music<br/>
Cleveland Conservatory of Music<br/>
North Carolina State University (2)<br/>
Washington University, St Louis.<br/>
Montgomery County Community College<br/>
Northern Territory University</p>
<p>You might notice a significant dearth of Ivy there. (And the one Ivy Leaguer we <em>do</em> have has been promoted to management so he doesn&rsquo;t even code anymore. We still love him though :))</p>
<p>An underlying premise of Stack Overflow and Stack Exchange is that there&rsquo;s tremendous amounts of knowledge tied up in &ldquo;ordinary&rdquo; people, and given an easy way to show it off and a little incentive they can make this knowledge available and help as many people as possible. Given Stack Exchange&rsquo;s belief in the common (wo)?man it&rsquo;s not surprising that the educational background of our programmers and system administrators is pretty&hellip;well, ordinary.</p>
<p>Locke setup up a straw man straw man by invoking Joel. I know he said the words &ldquo;Ivy League&rdquo; but Joel wasn&rsquo;t really talking about the Ivy League. He was talking about <em>selectivity</em>. Specifically that as a sorting criteria a resume showed that the associated applicant had successfully navigated some process that was highly selective:</p>
<blockquote><p><strong>Selectivity</strong>. Another thing we look for on resumes is evidence that someone has gone through some highly selective process in the past. Not everyone at Ivy League schools is worth hiring, and not everyone at community college is worth avoiding, but getting into a very selective school does at least mean that someone, somewhere judged you using some kind of selection process and decided that you were pretty smart. Our company criterion for selectivity is usually getting into a school or program that accepts less than 30% of its applicants (there are about 60 schools in the US that meet this standard), or working for a company which is known to have a difficult application process, like a whole day of interviews. Highly selective branches of the military like officer’s training or pilot’s courses, or even just getting into the Marines indicates someone that has made it through some kind of difficult application/selection procedure and all in all this is a positive sign.</p></blockquote>
<p>Tech companies <em>are</em> looking outside the Ivy League and the Dan Shippers of the world for their &ldquo;rock star&rdquo; developers. I could list the previous positions of all the Stack Exchange programmers and sysadmins but I&rsquo;ll save you the suspense: none of us were paying our bills running our own startups just waiting for something awesome like Stack Exchange to come along. In my case <a href=""><em>I</em> sought <em>them</em> out</a> and quit my cushy yet crappy hedge fund job.</p>
<p>Complaining about how it&rsquo;s not fair because they went to an Ivy League school and you didn&rsquo;t or they started a company and you didn&rsquo;t and these things makes their resume look better than yours isn&rsquo;t going to get you very far, though. You have to make it easy for employers to choose you from the pile of resumes. You know there&rsquo;s competition out there for these jobs. If you want to work at a start up you have know you&rsquo;re going up against some of the best in the industry and you have to make yourself stand out.</p>
<p>Your resume (or your <a href="">Careers 2.0 profile</a>) has the single purpose of <em>showing</em> them you&rsquo;re worth it. It has to <em>make</em> them believe you&rsquo;re worth calling for an interview. Provide some <strong>evidence</strong>. Don&rsquo;t have any? You&rsquo;re a programmer&hellip;manufacture some! Make a blog. Write a ruby gem. Answer some questions on Stack Overflow. Write some code for a charity. Make a website for your band. This stuff won&rsquo;t take forever to do. It isn&rsquo;t like the old days where you had to apprentice with a master furniture maker for a decade to get some cred. The medium you work in allows you to make things in days, not years. It doesn&rsquo;t have to change the world, it just has to give the person looking at your resume a reason to choose you instead of the other g(al|uy).</p>
<p>Just in case you&rsquo;re in the market and looking to stand out&hellip;we built <a href="">Careers 2.0</a> to help you stand out by enabling you to show off your programmery stuff. It&rsquo;s invitation only so if you&rsquo;d like an invite just tweet me <a href="">@JasonPunyon</a> and I&rsquo;ll hook you up. You may not have gone to an Ivy League school but if you can show off some evidence that you&rsquo;ve got skills you rank up there with the best of the Yalees and Browners. You&rsquo;ve got a section for open source projects hosted at Github, Bitbucket, Code Plex or Source Forge. There&rsquo;s a section for Apps you&rsquo;ve written. Write a blog or read any good books lately? Choose your favorites and show them off. When employers search the Careers database for candidates with particular skills we take all of this stuff into account and sort you accordingly.</p>
<p>Also! We&rsquo;re always looking for great developers to come work with us at Stack Exchange. Don&rsquo;t let what school you went to keep you from <a href="">applying today</a> (we&rsquo;ll even take you Ivy Leaguers).</p>
<p>Oh and again, if you&rsquo;re into physics&hellip;GO TO GENESEO&hellip;if only to watch Dr. Fletcher&rsquo;s valiant yearly attempts to demonstrate quantum tunneling by running full speed towards the classroom wall. He hasn&rsquo;t succeeded yet, but if he tries long enough probability says&hellip;</p>
<title type="html"><![CDATA[Don't Be Represented by Contingency Recruiters]]></title>
<link href=""/>
<content type="html"><![CDATA[<p><img class="right" src="">
Before I faded a <a href="">crazy counter offer</a> and got my <a href="">dream job</a> I worked in finance. I was going on interviews in 2009 having been laid off from my second job after the 2008 collapse. This was back in the days when I was still using contingency recruiters to find jobs. They&rsquo;re a natural fit for finding finance jobs because banks and hedge funds have money they&rsquo;re willing to spend on recruiters when they aren&rsquo;t swimming in it (artists rendering on the right), or spinning it into thread for their golden parachute side business.</p>
<h3>A Counter&hellip;</h3>
<p>I went on an interview and it turned out they liked me, a dubious honor in retrospect, so we continued through the process. One day my recruiter, let&rsquo;s call her Nancy, phoned me up and had <em>exciting news</em>. Nancy had always been pulling for me. She was on my side. We&rsquo;d gone through interviews at a couple of companies together already. This one liked me enough that they wanted to make me an offer at $X a year. &ldquo;Great!&rdquo; I thought.</p>
<p>I took a pause and rewound to what happened the last time I was made an offer. I didn&rsquo;t have a particular reason for doing so, but after I heard the offer of $Y I said &ldquo;It&rsquo;d be really nice if you could make that $Y+5K a year.&rdquo; The guy I was talking to was the one who would go on to hire me and without a thought he said &ldquo;Sure, no problem.&rdquo; So going off my statistically significant set of one data point I said the same thing to Nancy.</p>
<p>I was expecting &ldquo;Hey, let me check on that for you&rdquo; or even a hedged &ldquo;I&rsquo;m not sure where they are on the range, but I&rsquo;ll find out&rdquo;. Instead Nancy&rsquo;s tone dropped from celebratory to agitated. &ldquo;Jason, I don&rsquo;t know <em>what</em> you&rsquo;re thinking&hellip;they aren&rsquo;t going to move on this. You need to take it.&rdquo; This was not the Nancy who was on my side. This was <em>Nasty</em> Nancy, a voice and a force not to be trifled with. The exact same question provoked a totally different response. Why would she have turned on me so quickly for my inquiry, when the last person I asked had no problem with it?</p>
<p>There could be a million reasons, but the incentives for contingency recruiters best explain the evidence. Contingency recruiters are paid on commission when they successfully place a candidate in a job and the candidate stays for a honeymoon period of 3-6 months. The payment is a handsome ransom at 25%-35% of the candidate&rsquo;s first year salary. So for a $100K job they might see $25,000 (No doubt a kingly sum for what is essentially an introduction, but that&rsquo;s neither here nor there. I don&rsquo;t understand it, but the market pays what the market pays).</p>
<p>So my attempt at a counteroffer put a little wrinkle in her reward. She felt like she had a done deal and I was jeopardizing her jackpot. &ldquo;The company made an offer! It&rsquo;s so close and this dunderhead is screwing it up!&rdquo;. She had $ChunkOfChange in her hand. If the company was fine with the counteroffer she stood to make an extra ~$1K. If they weren&rsquo;t, she could end up making nothing. I&rsquo;m sure it didn&rsquo;t help that business was down because everyone was getting laid off after the collapse in 2008 and there was less hiring than ever going on. Faced with this situation Nancy attempted to &ldquo;Close the deal&rdquo; with me. She took the hard line to try and get me to just accept the offer so she could book the commission.</p>
<p>A very similar situation comes up when you use a real estate agent to sell your house. They may not take the hard line like Nancy, but they will try to persuade you to take the deal if offerred. Real estate agents are paid the same way as recruiters, commissions for done deals. The percentages are lower (around 3% per agent) which magnifies the effect of them wanting to close the deal. The difference between $475K for your house and $500K for your house is $25K to you, an amount not to be sneezed at. The agent&rsquo;s commision only goes up a mere $750 bucks though, from $14,250 to $15,000. Of course they&rsquo;d <em>much</em> rather you take the deal than try and bargain your way up to a better price if it means they could lose everything.</p>
<h3>Turns out this was a bad idea from the start&hellip;</h3>
<p>When you&rsquo;re the candidate and a recruiter is involved, my experience has been that you don&rsquo;t talk to someone from the company very much aside from interviews. If you get rejected, you get rejected by the recruiter. When you get an offer you get it from the recruiter. So when it comes to a counter offer, most likely the recruiter is going to be in the middle. The problem with having the recruiter in the middle if you&rsquo;re going to make a counter offer is Dual Agency.</p>
<p>Dual Agency is a term that comes from real estate. One situation where dual agency comes up is when one entity (a single agent or two agents that work at the same broker) represents both the seller and buyer of a house in a transaction. The problem is that a real estate agent has a fiduciary responsibility to the party they represent. They can&rsquo;t fulfill this obligation to two parties on opposite sides of the same transaction. One side wants to sell high, the other wants to buy low, the agent is stuck in the middle unable to fulfill their duty.</p>
<p>Nancy doesn&rsquo;t have a fiduciary obligation to anyone written in law like real estate agents do but the essence of the dual agency problem still exists. When Nancy&rsquo;s services were employed by the company, Nancy became their agent. By trying to route a counter offer through her I was trying to engage her in dual agency. The employer wants to buy low, I want to sell high and Nancy&rsquo;s stuck in the middle unable to effectively represent both sides. It was a bad idea from the outset. Should you not heed my advice (really&hellip;you should) and use a contingency recruiter anyway, make sure you don&rsquo;t try to make a counter offer through them.</p>
<h3>I&rsquo;m taking my ball&hellip;</h3>
<p>Because of the incentives at play and the problems of dual agency the goals of a contingency recruiter (close the deal at any price) simply aren&rsquo;t aligned with those of a candidate seeking a job (sell my labor for the highest price). I don&rsquo;t have any data to support it (I&rsquo;d love to see some if you&rsquo;ve got it) but I wouldn&rsquo;t be surprised if candidates who choose (yes&hellip;it&rsquo;s your choice) to be represented to companies by contingency recruiters end up with lower salaries than they could&rsquo;ve had because of this misalignment (not to mention that giant fee the company pays).</p>
<p>Ultimately I don&rsquo;t think Nancy&rsquo;s a bad person who was nefariously trying to screw me out of anything, or that she wanted to be mean to me. Getting a stack of money dangled in front of you and being threatened with getting it yanked away at the last minute is going to have an effect on someone&rsquo;s behavior.</p>
<p><strong>Incentives matter</strong> and I don&rsquo;t agree with the incentive scheme that&rsquo;s been setup, so I&rsquo;m just not going to play ever again.</p>
<p>Don&rsquo;t be represented by contingency recruiters.</p>
<title type="html"><![CDATA[On Counter Offers]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>Last I left you, dear readers, I had just been given an offer to work my dream job at <a href="">Stack Exchange</a> by Joel. I accepted the offer on the spot, and went home walking on air. What do you do when you get the offer for your dream job? You go home, crack open a bottle of wine and write your resignation letter.</p>
<p>Achieving a dream is really liberating. I&rsquo;d written a resignation letter before, but it wasn&rsquo;t like this. Last time I was leaving one garbage job for one with promise but there was ambivalence instead of the uncontainable excitement I was feeling. I knew things were getting &ldquo;better&rdquo;, but still didn&rsquo;t know what to expect. This time I was typing with purpose and a carefree attitude. Things were gonna be great! I know this sounds silly because resignation letters are generally a one line affair. &ldquo;This is my notice that I will be leaving the company effective [notice period] from today.&rdquo; I printed out three copies, signed &lsquo;em, enveloped &#8216;em and I was ready to go in in the morning and start my new adventure.</p>
<p>I had two bosses. So naturally I printed out three copies. One for each of my bosses and one for the #2 guy at the 1,000 person company that needed one according to the employee handbook (this would be my first interaction with him). I went in to work and boss #1 was already there. I checked the error log and made some quick fix commits while waiting for boss #2 to arrive.</p>
<p>Boss #2 finally got there and I called them both into boss #2&rsquo;s office. We sat down and I broke the news to them and the inevitable happened. They started bargaining.</p>
<blockquote><p>Boss #1: &ldquo;Well, is there anything we can do to keep you around?&rdquo;</p>
<p>Me: &ldquo;No. This is my dream job.&rdquo;</p>
<p>Boss #2: &ldquo;What about $[2X my current salary]&rdquo;</p></blockquote>
<p>BOOM. The counter offer and a <strong>doozy</strong> at that. The employer&rsquo;s last resort in the face of losing a good employee.</p>
<p>Boss #1&rsquo;s eyes went a little wide, remembering that we went through bonus and raise season a few months earlier where he and I had it out over a pittance relative to the current offer on the table. B&amp;R season at hedge funds is a harrowing time for everyone at the company. Almost all productivity stops for a month while secret incantations are spoken in back rooms by big wigs and pointy-headed bosses. The results are never transparent and rarely equitable and good people end up unhappy.</p>
<p>This counter offer reeked of skeeze. He was trying to buy <em>my dreams</em> from me. When someone tells you &ldquo;I&rsquo;ve been given the opportunity to live my dreams, I&rsquo;m leaving&rdquo; the <strike>decent</strike> human thing to do is to congratulate them and send them on their way. (With a security escort out the building, of course. You don&rsquo;t know what these dream heads are gonna do on the way out.) What you <em>don&rsquo;t</em> do is tell them you think so little of them, their lives and their goals that you think can buy them off.</p>
<p>The only thing the counter offer does is open up the possibility that the employee might take it. In my case I had a really positive reason about another company for leaving. It usually goes the other way though. The employee can have a really negative feeling about their current job, which led them to seek out new employment. Either way, the worst thing that can happen at this point is for the employee to take the counter offer.</p>
<p>If I were to accept the completely out of whack offer on the table, the company would have successfully bought my dreams&hellip;for a little while. Eventually, the hard truth of &ldquo;Holy shit, did I really just sell me and my dreams down the river for a couple extra benjamins?&rdquo; would get the better of me, and I&rsquo;d end up really <em>really</em> unhappy. Bad for me. Over time I&rsquo;d become less and less productive and more and more depressed at work and now I cost them minimum twice as much. Bad for the company.</p>
<p>The other case, where the employee is leaving because the company she&rsquo;s at sucks is pretty much just as bad. The employee is so unhappy with the company that they&rsquo;ve gone to the trouble of updating their resume, getting in the candidate pipeline at at least a few companies, phoners, in person interviews, second in person interviews. They&rsquo;ve invested an inordinate amount of time and effort which should tell you, the company, one simple thing. It&rsquo;s not them, <em>it&rsquo;s you</em>. Again, money might paper things over for a while but the ultimate truth is, they&rsquo;re not going to be the awesome employee you want. They&rsquo;re going to end up just as unhappy and unproductive after the money wears off as I would&rsquo;ve been had I taken it.</p>
<p>The other thing employees have to think about with an outsize counter offer is the fact that it may just simply be a pipe dream. Companies don&rsquo;t just allow their managers to get slapped around and make outrageous counter offers to people. Budgets and HR minions can&rsquo;t stand that. The manager gets dinged by the higher ups because she&rsquo;s &ldquo;lost control&rdquo;. So some of these offers are just plain &ldquo;keep them in the building while we find a replacement who&rsquo;ll do the job at the price we were paying.&rdquo; As soon as they do that, you&rsquo;ll find yourself on your ass without the cushion of the comfy offer you declined to take earlier.</p>
<p>I didn&rsquo;t miss a beat when I heard the offer, told Boss #2 &ldquo;No,&rdquo; and got up and left the room.</p>
<p>There was no way I was going to let this skeezy jerk buy my dreams.</p>
<title type="html"><![CDATA[How I Got My Dream Job at Stack Exchange]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>Matt recently wrote his <a href="">story</a> about how he got his job at Stack Exchange and I thought it was such a good idea that I&rsquo;d go ahead and write my own. This is my long rambling account of my path to my dream job.</p>
<h3>Getting bit by the bug</h3>
<p>I was a physics major. I spent the summers of my college years at the 88&#8221; Cyclotron at UC Berkeley helping in the preparation, execution and analysis of medium energy nuclear physics experiments. The summer after your freshman year, you don&rsquo;t know jack about <strike>nuclear</strike> physics.</p>
<p>So what do they do with a kid who knows nothing about nuclear physics at a nuclear physics lab? Stick him in front of a computer, of course. Give him a C library that knows the physics so that he just needs to learn to configure it correctly to spit out the needed answer. So I did that. I built a 3D model of Gammasphere and our specialized particle detector using this library and it was awesome. I got my name on some papers and I presented my work at a posterboard session at a conference for undergraduate researchers. It was great. Most of all I got bit by the programming bug. I learned that I loved the control, the empowerment. I was building a little universe inside my computer and creating something of value.</p>
<h3>Off the path&hellip;</h3>
<p>After college ended the best possible thing in the world happened to me: I got rejected for graduate school. I was devastated at the time because I really wanted to be a physicist but in hindsight not spending most of my twenties grinding out a Ph. D. was so much better. I dicked around for a year playing poker and waiting for my then-future-wife to graduate. Her graduation neared and I realized I needed to GTFO of my parents house so I went looking for a job.</p>
<p>I did what it seems most people with physics degrees do nowadays; I went into finance. I started programming for a reinsurance and management consulting company (read: bullshit artist outfit) on an impressingly boring time tracking tool conversion project. They lost my then-profoundly-bad programming &ldquo;skills&rdquo; when they ratcheted benefits down to non-existent levels and I got up and moved into the big time.</p>
<p>HEDGE FUNDS (read: bullshit artist outfit)</p>
<p>This is where it was at man. Money central. The world was your oyster. I got way better at programming during this period of my career because I had people screaming in my face when shit got ill. People who care about money <em>really</em> care about money and when their programs screw up they will humiliate you in front of 200 people on a trading floor.</p>
<p>The other reason I got better is because I started interacting with people who knew what they were talking about. People spent time <em>learning</em> how to do this? People wrote <em>books</em> about this? Software development as <em>craft</em>? I&rsquo;d been doing this &ldquo;professionally&rdquo; for six months and I was just hacking it together as it came. A friend of mine introduced me to Jeff and Joel&rsquo;s blogs and I was in heaven. All their stuff was brand new to me, and they opened windows and passageways to more awesome stuff. I couldn&rsquo;t have asked for a better education. After I read what they had to offer I started devouring books.</p>
<p>2008 happened and everything got really really dismal. All my friends were getting laid off. Due to my association with the industry the world hated everything about my very existence, and rightly so. The <em>insane</em> focus on money and everyone around you trying to figure out how to screw people out of it was sickening. It was a really depressing thing to be a part of. And what did I do after I got laid off in 2009? I re-upped for round 2, OF COURSE.</p>
<p>The money was good and I felt like I didn&rsquo;t have anywhere else to go. Even in the middle of the crisis the recruitment industry in New York would have you believe there isn&rsquo;t a job outside finance anywhere (really, they just don&rsquo;t care about jobs outside finance because the companies with those jobs won&rsquo;t pay the ridiculously gargantuan fees). During this time contingency recruiters became a big fat thorn in my side. It&rsquo;s not that they&rsquo;re bad people, their incentives are just completely at odds with employers and job seekers so they end up doing some shady things.</p>
<p>So I ended up at another hedge fund doing real-time work for the fixed income trading desk.</p>
<h3>A Glimmer</h3>
<p>You know what else happened in 2008? Stack Overflow was launched. I was looney tunes for this site. It hit every endorphin releasing part of my brain. The game mechanics were great and I was easily hooked. I spent time there each day (even if I wasn&rsquo;t answering) just taking things in. I could do this because while very high pressure, the shops I worked at could get really paralyzed about decisions. This gave me all the underemployment I needed to play my new favorite game.</p>
<p>At the beginning of April 2010 Stack Overflow threw a little shindig at the Fog Creek office. They opened it up to the public so I just had to go. When I met Jeff I introduced myself, &ldquo;Hi I&rsquo;m Jason&hellip;The Big Green J on Stack Overflow (referring to my gravatar)&rdquo; AND HE KNEW OF ME. It was insane. We talked for a while and he mentioned they&rsquo;d be hiring for the Careers team and I threw an off the cuff &ldquo;I&rsquo;d love to work on that.&rdquo; I thought (and still do) that we could do &ldquo;getting a job&rdquo; much better without all the shady recruiter practices I dealt with getting my first three jobs.</p>
<p>He called Joel over and introduced us at which point they told me to keep my ears open, they&rsquo;d be announcing the jobs in a month and they&rsquo;d start reviewing applications. Rock Band ensued, the party wound down and everyone was leaving. I rode the 6 train back to Grand Central with Jeff, Geoff and Jarrod. They wanted to know my story so I told them what I could before they got off at Bleecker.</p>
<p>A month later they posted their Series A blog post with the cattle call and it took me 2 fraking days to respond. What the hell was wrong with me? I figured the job was in such high demand that the game was already over. I threw together my application and sent it in anyway&hellip;</p>
<blockquote><p>Hi Jeff and Jarrod and Geoff and Joel and whoever else is in there,</p>
<p>I would consider it an honor for you to consider it an honor to consider me
for a job with Stackoverflow (was that enough layers of indirection?).
Hopefully it&rsquo;s obvious that I&rsquo;m a programmer and that&rsquo;s the job I&rsquo;m looking
for but I could deal with &ldquo;What&rsquo;s big and grey and proves the uncountability
of the reals?&rdquo; for about a year too :)</p>
<p><a href=""></a></p>
<p>Thanks, <br />
Jason Punyon aka &ldquo;The Big Green J&rdquo;</p></blockquote>
<p>It was the longest 5 days of my life before I got a response asking me for a phone interview. The phoner was with Joel and was pretty standard. We talked about me, something I did that I was proud of, a simple data structures question and that was it. I got an e-mail an hour later asking me to come in for an in person interview. I was in the middle of moving so we scheduled it for a week later.</p>
<p>I was scared shitless of this interview. I thought it was going to be the hardest thing since quantum mechanics. I hit the books hard and studied. I hadn&rsquo;t done C in a while and I knew Fog Creek did their interviews in C and liked to ask questions about pointers and data structures. Remember, I didn&rsquo;t have a CS degree, so I crammed like there was no tomorrow. And interview day came.</p>
<h3>The Big Day</h3>
<p>I didn&rsquo;t know what to wear (they didn&rsquo;t clue me in that they didn&rsquo;t care how interviewees dressed), so I suited and tied up and got on the train. It had an emerald city type of feel; this is where the wizards were. I went in and met David and we started the interview.</p>
<p>David opened with a FizzBuzz type question but it was much simpler. Without the method signature and curly braces the solution was 1 line. Felt good. Then the other shoe dropped. On the second question we worked on I froze. I&rsquo;d gotten the naive solution to the problem and David asked me &ldquo;Can we make it faster?&rdquo;. I knew the concept to the answer to the question, I just couldn&rsquo;t connect it for some reason. I stared at the sheet of paper we were working on for minutes on end quietly panicking and it struck me to just start talking. This was the key to me getting through the question. Thinking just doesn&rsquo;t do it sometimes. Eventually I talked my way through sucessively faster solutions until we got to the optimal one. I got there but I figured the interview was over.</p>
<p>We went to lunch.</p>
<p>I met a couple more people in turn and did fine on their questions. Then I was told to go sit on the red couch in the reception area. This is where you sit to sweat it out while people are talking about you. I figured I was a coinflip at best because of how down in flames I went on David&rsquo;s second question.</p>
<p>Joel came out and wanted to talk to me. SQUEEEE. We went into his office and just talked. There wasn&rsquo;t any assessing of skills going on, we were just chatting about Stack Overflow stuff in general and then it happened.</p>
<p>&ldquo;We&rsquo;d love it if you came and worked for us here at Stack Overflow.&rdquo;</p>
<p>Disbelief. Shock. Amazement. Relief. Excitement.</p>
<h3>What did I learn?</h3>
<p>Specifically about interviewing: Don&rsquo;t. Freeze. Going into the tank and clamming up tells the interviewer nothing about you. Talking is much <em>much</em> better than not talking. Thinking the wrong stuff out loud doesn&rsquo;t make you look stupid, it shows the thought process behind getting to the solution. It&rsquo;s just like a written test. Getting the right answer is good, but showing your work still gets you partial credit. Things that are far away from you always seem more daunting than they are. Each step you take to stack the deck in your favor gets you a little closer and makes things a little less scary.</p>
<p>You aren&rsquo;t that far away from your dreams if you&rsquo;re willing to do a little work and take a little risk to make it happen. Taking a vacation day and going to the party to meet the team and show my interest, riding the train home with them to get an extra chance to talk and get remembered, following up when they announced their openings (even though I thought I was late), cracking the books to make sure I knew what I needed to know to pass the test were all extra work I did to give myself the best possible chance to get where I wanted to go.</p>
<p>If you think Stack Exchange is the place for you please don&rsquo;t hesitate to apply, you might surprise yourself. We&rsquo;re currently looking for a <a href="">few sharp devs</a> to join the team in New York and a <a href="">search architect</a>.</p>
<title type="html"><![CDATA[Employee Referral Bonuses Are Pants on Head Stupid: Follow Up]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>In my <a href="">previous post</a> I outlined one reason why I think employee referral bonuses aren&rsquo;t good: They distort the incentives for referring someone into the company in a way that can&rsquo;t possibly be good. First off, my thanks go out to everyone who read that post. Some people refuted my argument in various ways and I thought I&rsquo;d take a look at a couple of them.</p>
<h3>Refutation The First: Bonus Size Matters</h3>
<p>This is correct. I was working under the scenario of the <a href="">miso article</a> where the bonus was $10,000. The size of the bonus absolutely matters. The bonus can&rsquo;t distort any incentives if it&rsquo;s too small for people to care about. But that&rsquo;s the problem then. It&rsquo;s too small for people to care about. The promise of $500 ($350 after taxes?) six months from hire date is not enough to divert your employee&rsquo;s attention from their primary responsibilities to do this recruiting work for you.</p>
<p>If you also use contingency recruiters (you shouldn&rsquo;t, but I realize companies still do) another problem that might exist with relatively small referral bonuses is you can&rsquo;t simultaneously have GIANT contingency recruiter fees that your employees know about. If a contingency recruiter gets $25K on a $100K salaried worker when your employee gets $500 or $1000 for the same kind of referral your employee is going to see the dissonance there and be unhappy.</p>
<p>Another dissonance your employees might see is that the company says things like &ldquo;We are 100% dedicated to finding great talent. It is #1 with a bullet on our list of priorities&rdquo; and then you pay out $500 for a referral. It just doesn&rsquo;t jive.</p>
<h3>Refutation The Second: Risk vs. Reward</h3>
<p>Another popular thought was that referring someone to the company carries with it the inherent risk that the hire doesn&rsquo;t work out. When the hire doesn&rsquo;t work out it&rsquo;s the referrer that would be &ldquo;blamed&rdquo; for the failure and that the bonus compensates them for that risk. This was interesting and I have to admit that I didn&rsquo;t consider framing the issue like this at all.</p>
<p>Your employee who refers a potential hire doesn&rsquo;t get the bonus until the potential hire is hired and stays for a honeymoon period of 3 or 6 months. The honeymoon period is there to protect the company from paying out a bonus and then having the new hire quit after 3 weeks. So when the new hire doesn&rsquo;t work out during the honeymoon period, the referrer gets both the &ldquo;blame&rdquo; <strong>and</strong> no bonus. The bonus can&rsquo;t possibly compensate them for any risk during the honeymoon period.</p>
<p>The honeymoon period simultaneously sends the message &ldquo;Once the new hire makes it out of the honeymoon period, we&rsquo;re happy with your referral performance. Let&rsquo;s celebrate with some rewards.&rdquo; If the new hire makes it out of the period, and then something doesn&rsquo;t work out, the company <em>can&rsquo;t</em> with a straight face blame the referrer for anything. It&rsquo;s simply schizophrenic. If you&rsquo;re doing it right, there isn&rsquo;t any blame risk to compensate them for after the honeymoon period is over.</p>
<p>So during the honeymoon period the referrer can&rsquo;t possibly be compensated for the risk, and after the honeymoon period there isn&rsquo;t any risk for which to compensate the referrer if the company isn&rsquo;t being schizophrenic.</p>
<p>Side note: This analysis was done under the pretense that there&rsquo;s this blame game being played. If there&rsquo;s any of this going on at all there is a <strong>serious</strong> problem that needs to be remedied immediately. The company can&rsquo;t attach this nebulous liability to the referrer for a person <em>the company</em> hired. The company should still put the referred candidate through their paces in the interview process and prove to themselves that the recommendation was justified. In doing so the company can&rsquo;t possibly place blame at the foot of the referrer for providing what is essentially an introduction.</p>
<title type="html"><![CDATA[Employee Referral Bonuses Are Pants on Head Stupid]]></title>
<link href=""/>
<content type="html"><![CDATA[<p>So, I read <a href="">this</a> story yesterday about the employee who referred a friend and got screwed out of his referral bonus. People on Hacker News argued about &ldquo;Name and Shame&rdquo; and who was right and wrong but the first thought that popped into my head was &ldquo;employee referral bonuses are pants on head stupid.&rdquo;</p>
<p>Forget about this particular case and the collateral problems it&rsquo;s causing. The employee referral bonus is just an intrinsically bad idea. Ostensibly the bonus exists to reward an employee for referring someone into a job. All it really does is distort the incentives in a way that can&rsquo;t possibly be good.</p>
<p>Following is a list of reasons I want an employee of mine referring someone to my company:</p>
<li>The person being referred is beyond awesome at what they do and we&rsquo;d be fools not to hire them.</li>
<li>Did you not read #1?</li>
<p>When I hold a $10,000 gun to the referrer&rsquo;s head I&rsquo;m distracting them from that <strong>one</strong> simple reason. When there&rsquo;s no bonus involved there are people the employee will refer and people they won&rsquo;t. With the money hanging out there a third category of people opens up: people the employee will refer if they can get $10,000. This third category is made up completely from the people the employee wouldn&rsquo;t have referred if the bonus didn&rsquo;t exist. I don&rsquo;t want any of these people in the third category.</p>
<p>The absolute <strong>last</strong> thing I want to do is turn my employees into contingency recruiters and that&rsquo;s exactly what the referral bonus does.</p>
<p>So if referral bonuses aren&rsquo;t right how <em>do</em> you get your employees to refer great talent to the company and the company to great talent? Make your company such an awesome place to be and work that your employees naturally want to do that. Figure out what your employees want and then <strong>give it to them</strong>. Great health care, free lunch, 20% time&hellip;whatever. They&rsquo;ll be so happy with you that they won&rsquo;t be able to contain themselves when they find out a friend is looking for a job.</p>
<p>If you want to hear my thoughts on some of the responses to this post, read my follow up <a href="">here</a></p>