-
Notifications
You must be signed in to change notification settings - Fork 0
/
atom.xml
1703 lines (1288 loc) · 204 KB
/
atom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title><![CDATA[Will You Harry Me]]></title>
<link href="http://harry.me/atom.xml" rel="self"/>
<link href="http://harry.me/"/>
<updated>2015-11-25T23:11:14-05:00</updated>
<id>http://harry.me/</id>
<author>
<name><![CDATA[Harry Brundage]]></name>
</author>
<generator uri="http://octopress.org/">Octopress</generator>
<entry>
<title type="html"><![CDATA[Draw a Circle Around It]]></title>
<link href="http://harry.me/blog/2015/11/24/draw-a-circle-around-it/"/>
<updated>2015-11-24T19:36:00-05:00</updated>
<id>http://harry.me/blog/2015/11/24/draw-a-circle-around-it</id>
<content type="html"><![CDATA[<p>In my world as an engineer, the objective is to minimize the cost of creating and maintaining a thing, while maximizing the effectiveness of the thing towards some cool goal. <a href="http://www.shopify.com">Shopify</a> is one such thing that experiences this tension: its goal of making selling stuff really really easy is well on it’s way to accomplished, but it has taken a sizable army of software developers to get and keep it there. The main cost we’re afraid of is that as the system gets bigger and more complicated we will be immobilized by all the complexity we’ve introduced, which means we couldn’t stay competitive as the markets around us change. So, I’d like to share a principle that serves our value of building maintainable, low-service-cost things at Shopify. It’s something I call <em>drawing circles</em>.</p>
<!--more-->
<p>The most annoyingly complex requirements very rarely add that much value to the actual goal of the system. Much cost often centers around compliance or security: without these “annoyances”, we could make things work in a much simpler, easier way. Satisfying these requirements is necessary in that I get to keep my job, and our product doesn’t get shut down by the big bad government man, but these things aren’t the most important aspects of the system. It’s just a cost of building.</p>
<p>Implementing PCI compliance for credit card data is an example: we need to adhere to strict processes around how we store sensitive credit card data. We need to encrypt the data in transmission and at rest, implement cumbersome (read: effective) procedures around data access and code deployment, and prepare for all sorts of audits and attacks. Storing a credit card number would be much easier if we didn’t care about any of this, but we do because we want to protect our customers. Another example is implementing <a href="https://en.wikipedia.org/wiki/Right_to_be_forgotten">The Right To Be Forgotten</a> in a data warehouse. It’d be easiest and most valuable to a business to keep all of everyone’s data around forever, but that’s a violation of people’s privacy as defined by law, so purge functionality for personally identifiable information is required. Warehouses often process raw data in multiple, ever expanding layers of transformations, so if you want to remove something at the source you also need to propagate purges through any downstream transforms that used the now-tainted information. This is a terribly complicating design constraint that results in a whole lot of extra implementation and maintenance once more.</p>
<p>It is very easy to accidentally let these complexity-generating requirements dictate a complex design of the whole system. In the case of PCI Level 3 certification, the cumbersome requirements it imposes are reasonable requirements for something storing credit card data, but Shopify is a full featured content management system for much less sensitive data as well! It powers a simple blogging system, a theme engine and a product catalog. PCI compliance dictates that every code change be audited and documented by two other developers that the author, and the deployed by yet another party. Do we really need each developer to write a document for every single code change request and have it audited by a third party before shipping said change? Do we really need someone other than the author to deploy the code every single time?</p>
<p>If we foolishly built only one system to store all of Shopify’s data regardless of sensitivity, then every Shopify developer would be required to conform to the most restrictive set of requirements of any piece of data we capture. What we should instead do is not let the most complex requirements govern the design of the entire system, but draw a circle around this complexity. We should spin complexity off it into it’s own system where that complexity is contained, and the system remaining can be simple. The part of the product that necessitates the complexity PCI imposes should be the <em>only</em> part burdened by it.</p>
<p>The same principle applies to the data warehousing example. Shopify uses Hadoop to store its data which is much less amenable to deletes than one might hope<sup id="fnref:1"><a href="#fn:1" rel="footnote">1</a></sup>, but because we’re required to implement deletes in order to not break the law, we are presented with a design challenge. If we were goons and added this requirement to every data pipeline anyone wrote, everyone would need to teach the system how to purge data long after it hoped to have “finally” processed it. Every subsequent pipeline stage would need to comply with upstream purge demands and track which data that it produced depends on what from the upstream so that it can implement this propagation. This does not sound like minimizing the cost.</p>
<p>Again, this requirement of our system doesn’t really add much value to the business use case for it: analysts and data scientists want to make the business smarter, not spend all day deleting data. Instead, we should create a second system off to the side to contain this complexity with a nice little circle around it. We map out the data that could conceivably need to be purged from our system, store it exclusively over in this second area, and then make a rule that data pipelines may not be built on top of this sensitive data. Users of this sensitive data must instead depend on a token which references it, and propagate that sucker to the end of any pipeline. Then, in our “final destination” analytics database we make the sensitive information available from the side location, and join it in at the very last moment using the token. Now, only the tiny few tables in the second system need to implement any kind of purging or deleting, which is way less work and might even allow for a different implementation<sup id="fnref:2"><a href="#fn:2" rel="footnote">2</a></sup>. We’re also able to swap whatever pieces of the design we need to to make that easy over there, while still reaping the benefits of the original design for the vast majority of the other impact we want to make.</p>
<p>The main drawback I hear to this technique is that there are now two systems instead of one, and if the one just did what we needed, you wouldn’t have to maintain two. I say ballyhoo: the idea that one system should do it all is misguided because the overall complexity add is rarely worth it. Nor is what I am suggesting actually just moving and obscuring the maintenance cost with semantics. The complexity that should be in a circle has the opportunity to make <em>everything</em> worse, not just some things. The generated complexity will seep into every process and every bit of code if not managed effectively: why pay the cost for the 90% of code that doesn’t need to care?</p>
<p>The boundary between the system inside the circle and the system beside it also has benefits. If it turns out the two different problems are in fact more different than you thought, maybe completely different tech stacks, or teams, or approaches make sense! It’s exciting to service-ize such that the implementation of in-circle requirements is decoupled from the rest.</p>
<p>If you see a particular requirement generating a whole lot of complexity, try to not let it govern the whole system, and instead, draw a circle around it such that the rest of the system doesn’t have to care.</p>
<div class="footnotes">
<hr/>
<ol>
<li id="fn:1">
<p>Hadoop stores stuff in big, immutable blocks of data, and block operations aren’t as cheap as a standard filesystem. Excising individual rows from each block is incredibly annoying and non-performan because the target record (usually just a line in the file) needs to be found among all the blocks, and then the entire block needs to be rewritten to get rid of it.<a href="#fnref:1" rev="footnote">↩</a></p></li>
<li id="fn:2">
<p>The second side system is likely has a whole order of magnitude lower data volume too, which means we might be able to get away with not even doing any purge on it at all, and instead just copy whatever source data we need from scratch every day! Horray for circles being different!<a href="#fnref:2" rev="footnote">↩</a></p></li>
</ol>
</div>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Neat Algorithms - Paxos]]></title>
<link href="http://harry.me/blog/2014/12/27/neat-algorithms-paxos/"/>
<updated>2014-12-27T12:42:00-05:00</updated>
<id>http://harry.me/blog/2014/12/27/neat-algorithms-paxos</id>
<content type="html"><![CDATA[<p><link rel="stylesheet" href="http://harry.me/assets/paxos-b8344ea2a0bc2491f6b6331d1df9c108.css"></p>
<p>This is an explanation and demonstration of an extraordinarily neat algorithm called Paxos. Paxos is a family of algorithms for teaching a whole bunch of decidedly unreliable processes to reliably decide on stuff. More formally: it allows a group of unreliable processors to deterministically and safely reach consensus if some certain conditions can be met, while ensuring the group remains consistent if the conditions can’t be met.</p>
<!--more-->
<p>The demo below demonstrates a live version of Paxos implemented in Javascript. Two different clients propose new values to random nodes in the system, and depending on what nodes are contacted and in what order, consensus is reached on one of the two proposed values and the clients informed.</p>
<div id="main_demo"></div>
<h1>Paxos: something we can agree on.</h1>
<p>Paxos is an algorithm to solve the <a href="http://harry.me/blog/2013/07/07/id-like-to-have-an-argument-a-primer-on-consensus/">consensus problem</a>. Honest-to-goodness real-life implementations of Paxos can be found at the heart of world class software like Cassandra, Google’s magnificent Spanner database, and also their distributed locking service Chubby. A system governed by Paxos is usually talked about in terms of the value, or state, it tracks. The system is built to allow many processes to store and report this value even if some fail, which is handy for building highly available and strongly consistent systems. To restate, a majority of the members of the system must agree that a particular value is in fact “the one true” value to then report it as such. Conversely, it means that one rogue process which has an out of date idea of the world can’t report something that isn’t “the one true” thing.</p>
<p>Let’s get some definitions out of the way for upcoming explanation:</p>
<ul>
<li>A <code>process</code> is one of the computers in the system. Lots of people use the word replica or node for this as well.</li>
<li>A <code>client</code> is a computer who isn’t a member of the system, but is asking the system what the value is, or asking the system to take on a new value.</li>
</ul>
<p>Paxos is only a small piece of building a distributed database: it only implements the process to write exactly one new thing to the system. Processes governed by an instance of Paxos can either fail, and not learn anything, or by the end of it have a majority having learned the same value such that there is consensus. Paxos doesn’t really tell us how to use this to build a database or anything like that, it is just the process which governs the individual communications between nodes as they execute one instance of deciding on one new value. So, for our purposes here, the thing we build with Paxos is a datumbase which can store exactly one value, and only once, such that you can’t change it after you’ve set it the first time.</p>
<h1>The read guts</h1>
<div id="read_demo"></div>
<p>To read a value from the basic Paxos system, a client asks all the processes in the system what they have stored for the current value, and then takes the value that the majority of the processes in the system hold. If there is no majority or if not enough processes respond, the read fails. To the left you can see a client asking the nodes what their value is, and them returning the value to the client. When the client gets a majority of responses agreeing on a value, it has successfully read it and keeps it handy.</p>
<p><br class="break" /></p>
<p>This is weird compared to single node systems. In both places, the client needs to make an observation on the system to determine the state, but in non-distributed systems like MySQL or one memcached process, the software only needs to ask the one canonical place where that state is stored. In simple Paxos, the client needs to observe the state the same way, but there is no canonical place where it is stored. It needs to ask all the members, so that it can be sure that there is actually only one value reported, and that it is in fact held by a majority of nodes. If the client just asked one node, it could be asking a process which is out of date, and get the “wrong” value. Processes can be out of date for all sorts of reasons: messages to them might have been dropped by unreliable networks, they might have failed and recovered with an out of date state, or the algorithm could still be underway and the process could have just not gotten it’s messages quite yet. It is important to note that this is “naïve” Paxos: there are much better ways of doing reads when implementing a system using Paxos that don’t require contacting every node for every read, but they extend beyond the original Paxos algorithm.</p>
<h1>The write guts</h1>
<p>Let’s examine what Paxos makes our cluster of processes do when a client asks that a new value be written. The following procedure is all to get only one value written. Eventually we can use this process as a primitive to allow many values to be set one after another, but the basic Paxos algorithm governs the flow for the writing of just one new value, which is then repeated to make the thing actually useful.</p>
<div id="client_demo"></div>
<p>The process starts with a client of the Paxos governed system asks that a new value be set. The client here shows up as the red circle, and the processes show up as the teal circles. Paxos makes a guarantee that clients can send their write requests to any member of the Paxos cluster, so for the demos here the client picks one of the processes at random. This property is important and neat: it means that there is no single point of failure, which means our Paxos governed system can continue to be online (and useful) when <em>any</em> node goes down for whatever unfortunate yet unavoidable reason. If we designated one particular node as “the proposer”, or “the master” or what have you, then the whole system would grind to a halt if that node failed.</p>
<p>When this write request is received, the Paxos process that receives the write request “proposes” this new value to the system. “Proposition” is in fact a formalized idea in Paxos: proposals to a system governed by Paxos can succeed or fail, and are a required step to ensure consensus is maintained. This proposal is sent to the whole system by way of a <code>prepare</code> message from the process the client contacted to all the other processes it knows of.</p>
<h3>Sequence Numbers</h3>
<p>This <code>prepare</code> message holds inside it the value being proposed, as well as what’s called a <em>sequence number</em> inside it. The sequence number is generated by the proposing process, and it declares that the receiving process should prepare to accept a proposal with that sequence number. This sequence number is key: it allows processes to differentiate between newer and older proposals. If two processes are trying to get a value set, Paxos says that value proposed last should take precedence, so this lets processes figure out which one is last, and thus who is trying to set the most recent value.</p>
<div id="prepare_demo"></div>
<p>These receiving processes are able to make a critical check in the system: is the sequence number on an incoming <code>prepare</code> message the highest I’ve ever seen? If it is, then cool, I can prepare to accept this incoming value, and disregard any others I have heard of before. You can see this happening to in the demo on the right: the client proposes a new value every so often to one process, that process sends <code>prepare</code> messages to the other processes, and then those processes note that these successively higher sequence numbers trump the older ones, and let go of those old proposals.</p>
<p>This little ordering idea is what lets any member of the system issue proposal to avoid the single point of failure associated with a designated “proposer” node mentioned above. Without this ordering, members of the Paxos system would have no way to figure out which proposal is the one they should prepare to accept with confidence.</p>
<p>We could imagine a different consensus algorithm which didn’t do this step of sending a first message to ask the other processes to make sure the value trying to be set is the most recent one. Although being way simpler, this would no longer satisfy the consensus algorithm safety requirements. If two processes started proposing different values right around the same time (like in the demos below), the universe could conspire against us and align the packets such that each dueling proposer convinces one half the processes to accept their own maybe-right-maybe-wrong value. The system could end up in a stalemate! There would exist two evenly sized groups having staged different value, which would lead to no value being accepted by a majority group. This stalemate is avoided by the first Paxos message exchange with sequence numbers that allow the processes to all resolve which proposal they should accept. With Paxos' sequence numbers, one of the dueling proposals would have a lower number than the other, and thus upon proposal receipt processes will have a way to unambiguously pick the most recent one. They’d either get the higher number one first, and later receive the lower number one and reject it, or they’d get higher numbered one second and thus replace the lower numbered one with it. Paxos solves the problem of consensus over time by taking hold of time itself with sequence numbers to apply temporal precedence.</p>
<div id="prepare_wrong_demo">
<div class="viz"></div>
<i>The above demo uses processes which just accept the most recent message as the "truth", instead of using sequence numbers. Because the clients send at the same time, we end up with a split brain where some process get one message last, and others get a different one. Consensus can't be reached!</i>
</div>
<div id="prepare_right_demo">
<div class="viz"></div>
<i>The above demo uses proper Paxos processes which examine the sequence number of the incoming proposal to figure out whether or not to actually prepare to accept the new value in tow. All the processes disambiguate properly, and consensus could be reached!</i>
</div>
<p><br class="break" /></p>
<p>Side note: it’s important that no two proposers ever use the same sequence number, and that they are sortable, so that they truly reference only one proposal, and precedence between proposals can be decided using a simple comparison. When implementing Paxos, these globally unique and sortable sequence numbers are usually derivatives of precise system time and node number in the cluster so they grow over time and are never the same.</p>
<h3>Promises</h3>
<div id="promise_demo"></div>
<p>So, after the proposing process has sent out it’s proposal, the processes check the proposal’s sequence number against the highest they’ve ever seen, and if it is the highest, they can make a promise to not accept any proposals older than this new threshold sequence number. This promise is returned as a message sent from the promising process to the one that is proposing a new value, as thing a <code>promise</code> message. This gives the proposing process the information it needs to count how many processes have sent their promises, and thus the basis to establish if it has reached a majority or not. If a majority of processes have agreed to accept this proposal or a higher sequenced one, the proposing process can know it “has the floor”, so to speak, and that progress is possible in the algorithm. If for whatever reason the proposer can’t extract a majority of promises from the other processes, progress is impossible, since consensus couldn’t be reached, so the proposal is aborted and the client is informed that the write failed.</p>
<p>To decide if a proposal has extracted enough promises, proposers simply count the number of <code>promise</code> messages they receive and compare against the total number of processes in the system. “Enough” promises here means promises from a majority (N/2 + 1) of processes in the system are received before a certain timeout. The simplest reason for this might be that more than half of the processes in the system have failed completely, so they wouldn’t return <code>promise</code> messages ever. This means Paxos could never get the proposed value committed a majority of processes, and thus could never satisfy the majority requirements in the read algorithm described above, and thus couldn’t reach consensus, so the proposal should be aborted. Other failure modes which would prevent a majority of promises being returned include things like a network partition preventing a proposer from contacting enough nodes, or more interestingly, a competing proposal having extracted <code>promise</code>s with a higher sequence number already.</p>
<h2>Acceptance</h2>
<div id="accept_demo"></div>
<p>Once a proposer has extracted promises from a majority of other processes, it asks the promising processes to “accept” the value they promised to before. This is the “commit” phase of the algorithm where progress is actually made. If there are no dueling proposals, failures, or partitions, then the proposal will be accepted by all nodes, and Paxos is complete! You can see this is the demo to the right when the second round of messages from the proposer, called the <code>accept</code> messages, cause the promised values to be taken on (sucked in) by all processes.</p>
<p>Acceptance of a particular process can fail however: if enough processes fail right after replying with a <code>promise</code> message, but before they receive the <code>accept</code> message, then the acceptance could only happen on a minority of nodes instead of a majority. In this case, the Paxos round is now in a weird state where some processes have accepted a value, but not all. This state, while undesirable, is actually “consistent” due to the read logic described far above: a client trying to read from the system must receive agreement from a majority of nodes on what the value actually is, so if it managed to contact all the nodes, different, conflicting values would be reported by different minorities of nodes. This would cause the read to fail, which sucks, but Paxos has remained consistent, and hasn’t allowed a write to take place without consensus. This bad state is often corrected in real implementations by either repeating the accept phase to get more nodes and eventually a majority.</p>
<h3>Dueling Proposals</h3>
<p>Acceptance can also fail because of dueling proposals: the promise that promisers reply with is a contract to accept proposals with that proposal’s sequence number, or higher. This means that a second proposal could come around after a first with a higher sequence number, and extract new promises from all the processes where they would no longer accept the first, earlier proposal. The first proposer however might not find out about this second proposal, and continue happily along in the Paxos algorithm, and send out it’s <code>accept</code> messages. Upon receipt of those <code>accept</code> messages, the promising processes would note the lower sequence number than their second promises to the second proposer, and simply reject the <code>accept</code> messages. This is correct: the first proposal sadly hasn’t made progress, but no value was accepted without consensus, and Paxos remains consistent. This kind of situation could easily arise if clients cause two different processes to start proposals at the same time, which you can see above.</p>
<p>The failure case here gets even more complicated if the second proposal comes around <em>after</em> acceptance has happened on some nodes for an earlier proposal. This is a Danger Zone for Paxos: if different processes have accepted different values, and especially if the groups of processes change which value they have accepted over time, reads of the system could return different values at different times for the same round of Paxos! That violates the consensus algorithm safety property that only one value can be reported by the system, so let’s examine how Paxos handles successive proposals and acceptances.</p>
<p>Let’s say that a second proposer recovers from a network partition and tries to propose a new value after a first proposer has already proposed a value, and already had that value accepted by a majority of processes. Paxos has “completed”, in that if the read algorithm was run, the first proposer’s value would be reported as the value for the system. That said, this second proposer is allowed by Paxos despite the completed state, because it declares all stages repeatable to allow for failure correction. So, this second proposer can run, but it must not change what value has been accepted, to keep consensus in tact as described above.</p>
<p>To prevent these later proposals from changing the accepted value, Paxos adds a little doohickey to prevents proposal from having a different value than the first one. If any processes have already accepted a value, Paxos forces <em>any</em> proposal that comes after that to also have that same already-accepted value. Freaky, but this maintains consensus, because now the value can never change after acceptance has started. The way this is implemented is that the <code>promise</code> messages returned by processes which have already accepted a value also carry along what value has already been accepted, and can tell the proposer about old values that have been accepted. The proposer can then detect if a majority of nodes have already accepted an old value, and change it’s proposal to match that value, or not run it at all. In a way, Paxos piggybacks a read operation on top of the promise phase to make sure that a proposal is in fact free to change the value of the system.</p>
<h3>Datumbase</h3>
<p>All this procedure accomplishes one thing: one durable write. Paxos itself has many variants that make it faster, introduce the ideas of masters, sacrifice pure fault tolerance for more speed, and tonnes of layers built on top which use it as a primitive to implement an actual database. An extremely interesting description of how to do this can be found in the Paxos Made Live paper listed below, but for us, this is the end. Thanks for reading! Please send any suggested edits, corrections, or feedback my way at <a href="mailto:harry@harry.me">harry@harry.me</a>.</p>
<h3>More Resources</h3>
<ul>
<li>The Part Time Parliment (the original Paxos paper) by Leslie Lamport: <a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/lamport-paxos.pdf">http://research.microsoft.com/en-us/um/people/lamport/pubs/lamport-paxos.pdf</a></li>
<li>Paxos Made Simple: another attempt at explaining Paxos by the original author by “deriving” the algorithm using the invariants of the consensus problem <a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf">http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf</a></li>
<li>Paxos Made Live: an amazing paper from Google describing the challenges of their Paxos implementation in Chubby: <a href="http://static.googleusercontent.com/media/research.google.com/en//archive/paxos_made_live.pdf">http://static.googleusercontent.com/media/research.google.com/en//archive/paxos_made_live.pdf</a></li>
<li>A Quora thread explaining Paxos in a few different ways: <a href="https://www.quora.com/Distributed-Systems/What-is-a-simple-explanation-of-the-Paxos-algorithm">https://www.quora.com/Distributed-Systems/What-is-a-simple-explanation-of-the-Paxos-algorithm</a></li>
<li>Raft - An Understandable Consensus Algorithm. Raft is another conensus algorithm designed for humans to understand, which if you can tell from the above wall of text might be a problem with Paxos. <a href="https://ramcloud.stanford.edu/raft.pdf">https://ramcloud.stanford.edu/raft.pdf</a></li>
</ul>
<script src="http://harry.me/assets/paxos/paxos-0a2fdb0709bc3b199adb503478fb2f56.js"></script>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[On Software Development Metrics]]></title>
<link href="http://harry.me/blog/2014/08/16/on-software-development-metrics/"/>
<updated>2014-08-16T12:57:00-04:00</updated>
<id>http://harry.me/blog/2014/08/16/on-software-development-metrics</id>
<content type="html"><![CDATA[<p>In which I try to justify data driven software development, just not for performance management.</p>
<!--more-->
<p>Shopify, where I work, has a business unit whose performance measurement and goals are all completely data driven. We know with a good degree of accuracy if the group is hitting its goals, we know exactly who in the group is excelling and who could use some help, and we know exactly how happy the clients of the group are. We sign contracts with business partners guaranteeing this group’s performance because we are confident in it, and the data powering these measurements. These measurements are quantifiable data, which is amazing because we can slice and dice it to learn more about the nature of the group’s performance and goals. We can ask valuable operational questions like “when during the week does the work load mean we need to schedule more people”, or “how many people do we need to hire next quarter to keep our customers happy”. We can ask valuable strategic questions as well, like “does this change to the product affect outcomes”, or “should we switch everyone over to this new potentially more productive tool”. Hard data powers better insight.</p>
<p>This group is, unfortunately, not comprised of software developers like me, but of sales and support staff at Shopify. They’re measured using metrics: how many people did they talk to today, how long did they talk to each of them for, which of those people said the experience was good or bad, etc, which powers the above decisions. For all the concerns the support group has down pat we developers have little to no analog. We have no objective benchmark which tells us if we are meeting all our obligations, we have no objective measure of individual performance for accolades or accusations, and we only have murky, through the grapevine indications of how satisfied our development group’s clients are. We can’t really predict demand for developers with anything other than a loose survey of the team leads, and we struggle to run experiments concerning techniques or tooling using data to actually make it a bona fide experiment. This upsets me, because I believe that this lack of data inhibits effective decision making for my business group. I’d really like to be able to run experiments, or to give long term hiring estimates to finance, or to understand internal customer satisfaction with our deliverables, but we just don’t have the data to power these insights.</p>
<p>So, how could we measure developers and the software development process to try to drive answers to the above questions using data? Well, the industry consensus, and the ideology inside Shopify, is that you can’t.</p>
<p>A mantra often repeated inside Shopify is “if you want a number to go up, put it on a dashboard”, and I’ve found this to be true many times over. A metric gives us a clear goal and a clear report on our progress towards it, so we start getting rewarding feedback cycles as we accomplish things that push that metric in the right direction. We make changes to the product or the code, we see the metric on the dashboard change for the better, and we get our dopamine or our promotion or whatever. This said, every metric has a dual nature: it encourages those who care about it to figure out how to push it in the right direction, but at the cost of that metric potentially forcing people to care about the wrong thing. For the metric to encourage the correct behaviour, it must accurately capture the true goals of the business. If it doesn’t, as soon as anyone or anything’s performance is tied to that metric, they are likely going to start working towards improving it above serving the underlying business goals. Aligning people with a metric only serves the business if the metric captures the business' values completely, lest the metric be gamed.</p>
<p>Take, for example “average customer satisfaction as measured by a short survey”. If we decide to reward service staff based on this metric (among others), we will likely have happier customers, because our service staff is encouraged to satisfy customers. This aligns with the business goal of making more money by keeping customers around, so it is a good metric to stick on dashboards.</p>
<p>Take, for a counter example, a metric like “lines of code added or removed this week” as a way to compare developers. If we started paying developers on a per line basis, we’d start seeing people making gigantic, overly verbose pull requests full of needless code and comments, because they’d get paid more! This does not align with the business goal of developing product faster than our competitors, because developers will be busy writing useless comments and hard-to-maintain complex code. This is thus a bad metric, and not suitable for dashboarding or performance management.</p>
<p>This conundrum of capturing the business goals with a metric is the oft-touted reason that software developers often go without quantitative measurement, at least in a performance management context. No one has really thought of a good metric or combination thereof that really encapsulates all the competing goals during software development. The most frequently pondered metrics are things like lines of code added or removed, automated code complexity reporting, test coverage, test run time and run frequency, code churn / change frequency, or defect discovery or fix rate, which are all really elementary, shortsighted observations about the happenings with the code. These metrics don’t bake in much understanding of true causality, long term maintainability, performance, security, among many other competing concerns good software developers spend time caring about.</p>
<p>The fact that we can’t come up with a suitable performance measurement scheme does not mean we shouldn’t measure the process though. Lines of code added or removed this period isn’t suitable for a feedback system in a dashboard, but it is still an interesting measurement to have on a report. If it grows like crazy all of a sudden, don’t you think it is worth investigating why? I’ve only ever heard of people not caring about this metric, or taking a casual glance at it in Github Pulse, but it really is correlated with important things. If a new developer starts and the rate spikes, that developer could likely use some feedback about simplicity and brevity. If it doesn’t change at all when a developer leaves, perhaps it is a good thing that developer has left, as the absence of their contributions should have at least been felt in the metric. The data that we do have is not useful for holistically measuring developers for performance review purposes, but it is useful for other insight. We correctly hesitate to practice <em>data driven</em> decision making using metrics like lines of code, but we forget that you can still make <em>data informed</em> decisions using these metrics as indicators.</p>
<p>For more examples: if test coverage plummets over the course of a few weeks, I’d love to have a dashboard which tells me where and who authored the new, uncovered code. If one particular area of the code is changing over and over, it’s likely a good candidate for the next refactor to try to make this change easier. If we had a report about the most frequently failed tests on local developer’s full suite runs, we should probably look at the top failures to see if they are easy to understand or perhaps overly brittle.</p>
<p>The benefits of data warehousing apply just as well: by mixing and matching this data with itself, and other data from the organization, we are able to do incredible stuff we couldn’t do before. We could join the lines added / removed history with the list of security incidents to see how old previously insecure code was, and then prompt an audit of code in the same age range to spot security issues before anyone nefarious beats us to it. We could correlate areas of code change with the aforementioned customer satisfaction surveys to see if we can tease out previously unknown relationships between changes to the product and changes in how customers perceive it. We could build data products for ourselves as well: we could make a bot which comments on Github when someone changes a particularly defective piece of code warning them to be extra careful, or we could optimize the order our tests run in so that those most likely to fail run first to give us fast feedback. So far at Shopify we’ve had success reporting on which sections of our codebase need the most love by counting Github issues opened and closed segmented by label, as well as reporting on production exceptions and which areas they have occurred in.</p>
<p>In summary, don’t let the fear of imperfect metrics for performance management stop you from gathering data, and doing some analysis on the software development process. Data driven organizations are more successful, and software development should be no exception.</p>
<h3>Further reading:</h3>
<ul>
<li>Measuring Performance Management in Organizations book by Robert D. Austin: <a href="http://www.amazon.com/gp/product/0932633366">http://www.amazon.com/gp/product/0932633366</a></li>
<li>A Stack Overflow discussion on measuring developer performance using data: <a href="http://pm.stackexchange.com/questions/5289/how-do-i-measure-employee-software-developer-performance-based-on-bugs-created">http://pm.stackexchange.com/questions/5289/how-do-i-measure-employee-software-developer-performance-based-on-bugs-created</a></li>
<li>IBM whitepaper on developer performance measurement using data: <a href="https://jazz.net/library/content/articles/insight/performance-management.pdf">https://jazz.net/library/content/articles/insight/performance-management.pdf</a> , and an article on the implementation of this: <a href="http://www.networkworld.com/article/2182958/software/how-ibm-started-grading-its-developers--productivity.html">http://www.networkworld.com/article/2182958/software/how-ibm-started-grading-its-developers--productivity.html</a></li>
</ul>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[I'd like to have an argument: A primer on consensus]]></title>
<link href="http://harry.me/blog/2013/07/07/id-like-to-have-an-argument-a-primer-on-consensus/"/>
<updated>2013-07-07T14:32:00-04:00</updated>
<id>http://harry.me/blog/2013/07/07/id-like-to-have-an-argument-a-primer-on-consensus</id>
<content type="html"><![CDATA[<p>If you are, say, a piece of e-commerce software, and, say, you want a record of when your customers buy stuff, how might you ensure that you get a record of each transaction every single time one happens? Money is changing hands, and people aren’t very fond of losing money without any gain in return, so having a correct ledger for transactions sure is important.</p>
<!--more-->
<p>You certainly can’t just leave this record on one computer, since that computer’s disk might die and leave you with none of your data. You could put it on two computers so that if one died you’d still have the record on the other computer, but you’d have to make sure that you write the information to both computers every time. Worse is that you must make sure that your cherished record gets written to both computers before you move on and accept more transactions, because if it doesn’t assuredly make it to both places, there’s a small chance you might only have one copy, and thus again risk losing that data.</p>
<p>If, say, your data set grew to be so important that simple dual redundancy was inadequate, you could network some computers who would all be responsible for storing the data. Now that this has happened you are beginning to have an interesting problem: how does your system behave when one of these computers fails? Ideally, if only one of ten of the machines gets its power cord tripped over, you should still be able to add more stuff to the other computers. After all, the more computers we add to increase redundancy, the more likely any failure at all is to occur, since we now have ten things that can fail instead of just one. We still want to make sure that when we write some data to this cluster, it is assuredly written to some bunch of boxes, but ideally it doesn’t need to be all ten so that the system can sustain inevitable failures.</p>
<h3>This isn’t an argument, it’s just contradiction!</h3>
<p>A possible strategy would be to designate one computer as the “master”, whose responsibilities would be to manage all the incoming write requests from clients of the system by doling them out to the other computers which it knows are online. Designating a master sounds good since we now have one computer who can decide if the system is ready to accept writes. This is to say that if enough computers fail, our beloved transaction ledger thing should enter an “unwritable” state, where no transactions can occur because we can’t safely store them. For this transaction log, we’d rather go down than lose data, again because people sure do love their money.</p>
<p>So, going with this strategy for a moment longer, we could program our master node to watch for node deaths, and decide if there is still enough online to continue accepting writes. There is one major glaring problem however: the master itself might fail. We’d need a new master, and lickity split. Then you might think, well, I’ll just have some other computer detect that the master computer has failed, and designate another one as the master! Easy peasy.</p>
<p>As simple as that you have stumbled upon a tough computer science problem. Whichever computers remain after a master failure need to somehow arrive at an agreement on who is going to be the next master. If all the computers leapt up and declared themselves the master, we could start having two different data sets, where depending on who you ask the same person has different amounts of money! If no computers declare themselves the master, the system stops working, and no one can buy stuff, which is also less than ideal. The process these computers should follow to designate one and only one new master is called reaching consensus.</p>
<p>The consensus problem is one of the quintessential building blocks of distributed systems, and seems to be regarded as one of the tougher ones from both a conceptual and software engineering point of view. Depending on what subset of the problems you look at, the aim is to define a rigorous process for submitting a value to a cluster of machines who will try reach consensus in the face of the expected failures, or unexpected ones like buggy software, or even goodness gracious holy macaroni <em>malicious agents</em> participating in the cluster. The cluster can agree to not accept a new value when one is submitted, or it can take a significant amount of time to accept it, but the key is that by the end of the process, the cluster “agrees” on what the “true” value is. This true value could be the one the cluster held before anything happened, or the newly submitted value, but the idea is that there is only one. The “true” value here is a convenient yet misleading metaphor, since again, depending on who or how many people you ask, the answer is different. That said, the role of a consensus algorithm is to define both how to submit a new value to the system, and also how to retrieve the “true” value the system has adopted. A handy definition of the “true” value read algorithm is just to ask everyone and see what value the majority of the cluster thinks the value is.</p>
<p>The reasons this problem is challenging arise from the simple fact that both processes and humans are unreliable. Disks fail, cords get unplugged, engineers write bugs, and yet all the while we still want to buy stuff. It wouldn’t be too tough to write a goofy consensus algorithm I shall enjoy titling “dunnolol” which just rejects any new incoming values in the event of any of these failures. Due to these failures' inevitability “dunnolol”, despite being simple, is relatively useless. The consensus problem holds us engineers to a higher standard of coming up with a way for a cluster of processes with some errors to remain resilient and still accept new values for data.</p>
<h3>Argument is an intellectual process</h3>
<p>Consensus problem solvers enjoy a number of horrid subproblems stemming from the fact that they must admit that there is such a thing as time. Many clients might try to propose a new value to the system around the same time, so problem solvers have to decide if they are going to impose an ordering on the operations the system takes. Messages between processes might arrive slowly, or even out of order as well, which means state has to be very carefully tracked by all actors in the show. A correct implementation of a solution to the problem must guarantee that one and only one value is agreed upon as the true value by the system at one instant. This means it must be completely resilient to conflicting clients proposing conflicting values, and bake in some sort of prevention of different factions of the system trying to pick one of the clients as the correct one.</p>
<p>All this boils <del>down</del> over into a few decades of research. As best I can tell, the state of the art consensus algorithm is one called <a href="http://en.wikipedia.org/wiki/Paxos_(computer_science">Paxos</a>, so if you are looking to see how things relying on consensus are actually built, I’d say start there. Interestingly very recently a new consensus algorithm has risen to prominence in the zeitgeist: <a href="https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf">Raft</a>. Raft interests me because it’s been designed for understandability as well as correctness, so it may be worth investigating as well. There’s also a number of resources describing concrete implementations of Paxos and the myriad of challenges associated with it which are simultaneously horrifying and interesting.</p>
<h3>More resources:</h3>
<ul>
<li>Paxos author’s list of papers: <a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html">http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html</a></li>
<li>Paxos author’s simplest explanation of Paxos: <a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf">http://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf</a></li>
<li>Seminal paper on Raft: <a href="https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf">https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf</a></li>
<li>Raft’s parent project, RAMCloud: <a href="https://ramcloud.stanford.edu/wiki/display/ramcloud/RAMCloud">https://ramcloud.stanford.edu/wiki/display/ramcloud/RAMCloud</a></li>
<li>Google’s report on implementing Paxos: <a href="http://www.read.seas.harvard.edu/~kohler/class/08w-dsi/chandra07paxos.pdf">http://www.read.seas.harvard.edu/~kohler/class/08w-dsi/chandra07paxos.pdf</a></li>
<li>Microsoft’s Will Portnoy’s blog on implementing Paxos: <a href="http://blog.willportnoy.com/2012/06/lessons-learned-from-paxos.html">http://blog.willportnoy.com/2012/06/lessons-learned-from-paxos.html</a></li>
<li>Monty Python’s “Argument Clinic”: <a href="http://www.youtube.com/watch?v=kQFKtI6gn9Y">http://www.youtube.com/watch?v=kQFKtI6gn9Y</a></li>
</ul>
<p> Thanks to <a href="https://twitter.com/camilolopez">@camilo</a> and <a href="https://twitter.com/dellsystem">@dellsystem</a> for helping edit.</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Reinvigoration: Really just a new theme]]></title>
<link href="http://harry.me/blog/2013/05/12/reinvigoration-really-just-a-new-theme/"/>
<updated>2013-05-12T12:41:00-04:00</updated>
<id>http://harry.me/blog/2013/05/12/reinvigoration-really-just-a-new-theme</id>
<content type="html"><![CDATA[<p>Yep. That’s all. This is a test post, and secretly an attempt at making this blog look a bit more loved, whoopee!</p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[JavaScript Documentation Sucks]]></title>
<link href="http://harry.me/blog/2012/01/22/javascript-documentation-sucks/"/>
<updated>2012-01-22T12:41:00-05:00</updated>
<id>http://harry.me/blog/2012/01/22/javascript-documentation-sucks</id>
<content type="html"><![CDATA[<p>I want to use your JavaScript library. I really do.</p>
<p>Unfortunately, a lot of you are bad at documenting it, so I can’t figure out how to use it before I give up and find something easier to learn.</p>
<!--more-->
<p>If you use <a href="http://jashkenas.github.com/docco/">docco</a>, this is because you write your documentation in the same way you write your code. Like a good team hacker you write your code with the principal design goal of making it easy for the next person working on the project, and only after satisfying that do you write for the other decidedly more mechanical audience. You strive to structure your code so that new functionality can be bolted on, or bits and bobs twiddled with to make the whole thing faster, and only serving the goal of getting the computer to do work in as elegant a manner as possible. This is awesome. Maintainable and readable code is awesome. In an effort to make it even more awesome, you start leaving some comments around. You say to yourself, well golly gee shucks, it would be nice if I had a reminder that this function’s boolean argument indicates this or that; or maybe it would be nice to remember what all these <code>RegExp</code> back-references which I’ve rarely used before actually mean.</p>
<p>The problem is this: these comments are oriented at someone trying to understand the code, not just use it, so when you pull them out, they document how the code works, not how to use it. Someone looking to know how to use a function cares not that it calls out to something which takes a boolean parameter. Someone looking to instantiate a class to get work done cares not that some part of it uses a <code>RegExp</code> to accomplish its tasks. What someone does care about is the API that class or function exposes, and what work it does that is helpful. The whole point of abstracting something useful into a library is that I don’t have to write that code myself when I use it, instead I can understand only the surface of it, and rely on you to have done a good job implementing the thing. I don’t want to have to read through the whole piece of code to understand how to call into it. You’ve structured it for a developer looking to change it, not to use it, so I get lost, and your explanatory comments usually don’t tell me function signatures or typical ways I might call them. It doesn’t make sense to make someone who wants to consume your library have to traverse your one monolithic HTML page to find the one stupid function signature they are looking for.</p>
<p>So, for things that aren’t quick and dirty, or aren’t exhibitions, I don’t like docco. If you want to generate awesome docs meant for consumption by people hacking on the project, by all means, use docco, à la <a href="http://pow.cx/docs/">pow</a>. I really don’t like people thinking docco or dox can be used for API documentation however. I believe this was never its intended purpose, but it <a href="https://github.com/echonest/nestjs">certainly</a> <a href="https://github.com/fd/chains.js/tree/50dce330d9f7f588643c016fec28cf1e6bdb34c6/docs">has</a> <a href="https://github.com/jgallen23/fidel">been</a> <a href="https://github.com/podviaznikov/fanfeedr">repurposed</a> <a href="https://github.com/waterfield/redeye">for</a> <a href="https://github.com/killdream/latte/tree/75ecff7a07b88d98aaed672ccc1408a753da88d7">this</a>. Perhaps developers feel that docco and company are suitable for creating API documentation because they, along with JSDoc, are the high visibility tools available in the JavaScript ecosystem. The misappropriation of docco isn’t rampant but the problem is even more insidious than just this. There are some who go the opposite route and orient their comments for consumers using tools like JSDoc. I desire to have JavaScript libraries documented so I can use them, and the idea of a common tool like RDoc (and YARD) giving us something like <a href="http://rdoc.info">rdoc.info</a> would be amazing but for me, JSDoc is not the answer.</p>
<p>The problem again, is this: when you orient your comments at consumers, development becomes simultaneous development and documentation, which is just plain unpleasant. I think it’s absolutely insane to have a 10 to 1 comment to code ratio in a file. It is outrageous to expect developers to work in files where each function has been meticulously explained, with all the available options listed and explained, as well as numerous examples of usage, all in a comment block just above the definition. Class bodies end up being enormous, and you can’t fit a function body and the body of those called in the first body on the same screen. I find scrolling through the grey molasses terribly inefficient when trying to reason about or work on the code at hand. I cite the Rails source, or at least the portions of it which expose APIs aimed at developers, as an example of how challenging it is to navigate code with inline documentation. You just can’t write clear, concise, readable code, at the same time as writing exhaustive, useful documentation when they co-exist in the same file.</p>
<p>You may say, Harry, these comments are wonderful, because we will change our comments when the code changes; they are right there, they will be in sync! I absolutely concede this: the closer the words for humans are to the words for the computer the easier a job we will have of keeping them accurate during change. Harry, you might say, now I can declare the types I expect my arguments to have and people can provide them. Harry, smart IDEs could even verify these types! I can almost pretend its a statically typed language! Again, you have a point, this information is valuable both to the library hacker and the library consumer, and maybe some IDEs I don’t use are smart about the <code>@param</code> declarations. Harry, you say, use an editor where you can ignore comments or fold them. I say back at you: vim folds are hard, and I am lazy, and I’m not always in my own editor where I know how to fold.</p>
<p>To me, the clutter is simply not worth the advantages. We should design code such that the next man or woman coming along can edit it and succeed. We should not sacrifice their chances of success by making them work in and around the screen filling comment blocks. The code is the uniquely canonical specification for how it can be used. It implicitly declares all of its use cases, but it doesn’t have to go ahead and be explicit about all those use cases by talking about them at length inline. It is essential that these examples and signatures and use cases are found somewhere, but I make the conscious decision to move my consumer documentation out of the code such that the code can be clear and unencumbered. Some project owners create gargantuan READMEs, or HTML guide style documentation, or GitHub wikis that these signatures and examples and whatnot look amazing inside. The shining examples from my world are the Django docs, the Rails Guides (and recently the SproutCore guides), and inside the JS world the Mongoose, Express, and Testling documentation sites. These are all curated sets of words for humans as well as code examples for humans showing how to consume a piece of software in another one. The problem comes down to actually creating these words, and continuing to maintain them after they’ve been released. APIs inevitably change, so docs inevitably go out of date. The worst aspect is that developers may not make changes because it is twice as much work as it might be normally. You must write the test, write the code, and now find all the places that code you changed is referenced in the docs and change them. There are of course notable exceptions, but in the JavaScript community I find myself learning how to use something from its README, and then spelunking in the code to find out the nuances.</p>
<p>This situation may not be as dire as I make it out to be, because if it were better tools would have evolved. It often seems to suffice in our community to ask people to read your tests or Google every second thing to figure out how to do it. Those amazing library authors who have the self control to document seem to either publish annotated source, write enormous comments, or pour their hearts into markdown’d READMEs and static sites. The annotated source isn’t a good reference for consumers, JSDoc block comments impede development, and the README’d source can fall out of sync easily, as it has with my principle project, Batman.</p>
<p>So, I ask you, what is a JavaScript developer to do? How do I let people use my project, while developing unencumbered?</p>
<p>I would like to know, but I also have my own storm a'brewin. The solution is underway but can be found here: <a href="https://github.com/hornairs/percolate.">https://github.com/hornairs/percolate.</a></p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Why Batman?]]></title>
<link href="http://harry.me/blog/2011/09/19/why-batman/"/>
<updated>2011-09-19T12:41:00-04:00</updated>
<id>http://harry.me/blog/2011/09/19/why-batman</id>
<content type="html"><![CDATA[<p><a href="http://batmanjs.org/">Batman.js</a> is <a href="http://shopify.com">Shopify</a>’s new open source CoffeeScript framework, and I’m absolutely elated to introduce it to the world after spending so much time on it. Find Batman on GitHub <a href="https://github.com/Shopify/batman">here</a>.</p>
<p>Batman emerges into a world populated with extraordinary frameworks being used to great effect. With the incredible stuff being pushed out in projects like <a href="http://www.sproutcore.com/about/">Sproutcore 2.0</a> and <a href="http://documentcloud.github.com/backbone/">Backbone.js</a>, how is a developer to know what to use when? There’s only so much time to play with cool new stuff, so I’d like to give a quick tour of what makes Batman different and why you might want to use it instead of the other amazing frameworks available today.</p>
<!--more-->
<h2>Batman makes building apps easy</h2>
<p>Batman is a framework for building single page applications. It’s not a progressive enhancement or a single purpose DOM or AJAX library. It’s built from the ground up to make building awesome single page apps by implementing all the lame parts of development like cross browser compatibility, data transport, validation, custom events, and a whole lot more. We provide handy helpers for development to generate and serve code, a recommended app structure for helping you organize code and call it when necessary, a full MVC stack, and a bunch of extras, all while remaining less than 18k when gzipped. Batman doesn’t provide the basics, or the whole kitchen sink, but a fluid API that allows you to write the important code for your app and none of the boilerplate.</p>
<h2>A super duper runtime</h2>
<p>At the heart of Batman is a runtime layer used for manipulating data from objects and subscribing to events objects may emit. Batman’s runtime is used similarly to SproutCore’s or Backbone’s in that all property access and assignment on Batman objects must be done through <code>someObject.get</code> and <code>someObject.set</code>, instead of using standard dot notation like you might in vanilla JavaScript. Adhering to this property system allows you to:</p>
<ul>
<li>transparently access “deep” properties which may be simple data or computed by a function,</li>
<li>inherit said computed properties from objects in the prototype chain,</li>
<li>subscribe to events like <code>change</code> or <code>ready</code> on other objects at “deep” keypaths,</li>
<li>and most importantly, dependencies can be tracked between said properties, so chained observers can be fired and computations can be cached while guaranteed to be up-to-date.</li>
</ul>
<p>All this comes free with every Batman object, and they still play nice with vanilla JavaScript objects. Let’s explore some of the things you can do with the runtime. Properties on objects can be observed using <code>Batman.Object::observe</code>:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="nv">crimeReport = </span><span class="k">new</span> <span class="nx">Batman</span><span class="p">.</span><span class="nb">Object</span>
</span><span class='line'><span class="nx">crimeReport</span><span class="p">.</span><span class="nx">observe</span> <span class="s">'address'</span><span class="p">,</span> <span class="nf">(newValue) -></span>
</span><span class='line'> <span class="k">if</span> <span class="nx">DangerTracker</span><span class="p">.</span><span class="nx">isDangerous</span><span class="p">(</span><span class="nx">newValue</span><span class="p">)</span>
</span><span class='line'> <span class="nx">crimeReport</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">'currentTeam'</span><span class="p">).</span><span class="nx">warnOfDanger</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>
<p>This kind of stuff is available in Backbone and SproutCore both, however we’ve tried to bring something we missed in those frameworks to Batman: “deep” keypaths. In Batman, any keypath you supply can traverse a chain of objects by separating the keys by a <code>.</code> (dot). For example:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="nv">batWatch = </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">currentCrimeReport: </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">address: </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">number: </span><span class="s">"123"</span>
</span><span class='line'> <span class="nv">street: </span><span class="s">"Easy St"</span>
</span><span class='line'> <span class="nv">city: </span><span class="s">"Gotham"</span>
</span><span class='line'>
</span><span class='line'><span class="nx">batWatch</span><span class="p">.</span><span class="nx">get</span> <span class="s">'currentCrimeReport.address.number'</span> <span class="c1">#=> "123"</span>
</span><span class='line'><span class="nx">batWatch</span><span class="p">.</span><span class="nx">set</span> <span class="s">'currentCrimeReport.address.number'</span><span class="p">,</span> <span class="s">"461A"</span>
</span><span class='line'><span class="nx">batWatch</span><span class="p">.</span><span class="nx">get</span> <span class="s">'currentCrimeReport.address.number'</span> <span class="c1">#=> "461A"</span>
</span></code></pre></td></tr></table></div></figure>
<p>This works for observation too:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="nx">batWatch</span><span class="p">.</span><span class="nx">observe</span> <span class="s">'currentCrimeReport.address.street'</span><span class="p">,</span> <span class="nf">(newStreet, oldStreet) -></span>
</span><span class='line'> <span class="k">if</span> <span class="nx">DistanceCalculator</span><span class="p">.</span><span class="nx">travelTime</span><span class="p">(</span><span class="nx">newStreet</span><span class="p">,</span> <span class="nx">oldStreet</span><span class="p">)</span> <span class="o">></span> <span class="mi">100000</span>
</span><span class='line'> <span class="nx">BatMobile</span><span class="p">.</span><span class="nx">bringTo</span><span class="p">(</span><span class="nx">batWatch</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">'currentLocation'</span><span class="p">))</span>
</span></code></pre></td></tr></table></div></figure>
<p>The craziest part of the whole thing is that these observers will always fire with the value of whatever is at that keypath, even if intermediate parts of the keypath change.</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="nv">crimeReportA = </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">address: </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">number: </span><span class="s">"123"</span>
</span><span class='line'> <span class="nv">street: </span><span class="s">"Easy St"</span>
</span><span class='line'> <span class="nv">city: </span><span class="s">"Gotham"</span>
</span><span class='line'>
</span><span class='line'><span class="nv">crimeReportB = </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">address: </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">number: </span><span class="s">"72"</span>
</span><span class='line'> <span class="nv">street: </span><span class="s">"Jolly Ln"</span>
</span><span class='line'> <span class="nv">city: </span><span class="s">"Gotham"</span>
</span><span class='line'>
</span><span class='line'><span class="nv">batWatch = </span><span class="k">new</span> <span class="nx">Batman</span><span class="p">.</span><span class="nb">Object</span><span class="p">({</span><span class="nv">currentCrimeReport: </span><span class="nx">crimeReportA</span><span class="p">})</span>
</span><span class='line'>
</span><span class='line'><span class="nx">batWatch</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">'currentCrimeReport.address.street'</span><span class="p">)</span> <span class="c1">#=> "East St"</span>
</span><span class='line'><span class="nx">batWatch</span><span class="p">.</span><span class="nx">observe</span> <span class="s">'currentCrimeReport.address.street'</span><span class="p">,</span> <span class="nf">(newStreet) -></span>
</span><span class='line'> <span class="nx">MuggingWatcher</span><span class="p">.</span><span class="nx">checkStreet</span><span class="p">(</span><span class="nx">newStreet</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="nx">batWatch</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s">'currentCrimeReport'</span><span class="p">,</span> <span class="nx">crimeReportB</span><span class="p">)</span>
</span><span class='line'><span class="c1"># the "MuggingWatcher" callback above will have been called with "Jolly Ln"</span>
</span></code></pre></td></tr></table></div></figure>
<p>Notice what happened? Even though the middle segment of the keypath changed (a whole new <code>crimeReport</code> object was introduced), the observer fires with the new deep value. This works with arbitrary length keypaths as well as intermingled <code>undefined</code> values.</p>
<p>The second neat part of the runtime is that because all access is done through <code>get</code> and <code>set</code>, we can track dependencies between object properties which need to be computed. Batman calls these functions <code>accessors</code>, and using the CoffeeScript executable class bodies they are really easy to define:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="k">class</span> <span class="nx">BatWatch</span> <span class="k">extends</span> <span class="nx">Batman</span><span class="p">.</span><span class="nb">Object</span>
</span><span class='line'> <span class="c1"># Define an accessor for the `currentDestination` key on instances of the BatWatch class.</span>
</span><span class='line'> <span class="nx">@accessor</span> <span class="s">'currentDestination'</span><span class="p">,</span> <span class="nf">-></span>
</span><span class='line'> <span class="nv">address = </span><span class="nx">@get</span> <span class="s">'currentCrimeReport.address'</span>
</span><span class='line'> <span class="k">return</span> <span class="s">"</span><span class="si">#{</span><span class="nx">address</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">'number'</span><span class="p">)</span><span class="si">}</span><span class="s"> </span><span class="si">#{</span><span class="nx">address</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">'street'</span><span class="p">)</span><span class="si">}</span><span class="s">, </span><span class="si">#{</span><span class="nx">address</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">'city'</span><span class="p">)</span><span class="si">}</span><span class="s">"</span>
</span><span class='line'>
</span><span class='line'><span class="nv">crimeReport = </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">address: </span><span class="nx">Batman</span>
</span><span class='line'> <span class="nv">number: </span><span class="s">"123"</span>
</span><span class='line'> <span class="nx">street</span> <span class="s">"Easy St"</span>
</span><span class='line'> <span class="nv">city: </span><span class="s">"Gotham"</span>
</span><span class='line'>
</span><span class='line'><span class="nv">watch = </span><span class="k">new</span> <span class="nx">BatWatch</span><span class="p">(</span><span class="nv">currentCrimeReport: </span><span class="nx">crimeReport</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'><span class="nx">watch</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">'currentDestination'</span><span class="p">)</span> <span class="c1">#=> "123 Easy St, Gotham"</span>
</span></code></pre></td></tr></table></div></figure>
<p>Importantly, the observers you may attach to these computed properties will fire as soon as you update their dependencies:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="nx">watch</span><span class="p">.</span><span class="nx">observe</span> <span class="s">'currentDestination'</span><span class="p">,</span> <span class="nf">(newDestination) -></span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span> <span class="nx">newDestination</span>
</span><span class='line'><span class="nx">crimeReport</span><span class="p">.</span><span class="nx">set</span><span class="p">(</span><span class="s">'address.number'</span><span class="p">,</span> <span class="s">"124"</span><span class="p">)</span>
</span><span class='line'><span class="c1"># "124 Easy St, Gotham" will have been logged to the console</span>
</span></code></pre></td></tr></table></div></figure>
<p>You can also define the default accessors which the runtime will fall back on if an object doesn’t already have an accessor defined for the key being <code>get</code>ted or <code>set</code>ted.</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="nv">jokerSimulator = </span><span class="k">new</span> <span class="nx">Batman</span><span class="p">.</span><span class="nb">Object</span>
</span><span class='line'><span class="nx">jokerSimulator</span><span class="p">.</span><span class="nx">accessor</span> <span class="nf">(key) -></span> <span class="s">"</span><span class="si">#{</span><span class="nx">key</span><span class="p">.</span><span class="nx">toUpperCase</span><span class="p">()</span><span class="si">}</span><span class="s">, HA HA HA!"</span>
</span><span class='line'>
</span><span class='line'><span class="nx">jokerSimulator</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s">"why so serious"</span><span class="p">)</span> <span class="c1">#=> "WHY SO SERIOUS, HA HA HA!"</span>
</span></code></pre></td></tr></table></div></figure>
<p>This feature is useful when you want to present a standard interface to an object, but work with the data in nontrivial ways underneath. For example, <code>Batman.Hash</code> uses this to present an API similar to a standard JavaScript object, while emitting events and allowing objects to be used as keys.</p>
<h2>What’s it useful for?</h2>
<p>The core of Batman as explained above makes it possible to know when data changes as soon as it happens. This is ideal for something like client side views. They’re no longer static bundles of HTML that get cobbled together as a long string and sent to the client, they are long lived representations of data which need to change as the data does. Batman comes bundled with a view system which leverages the abilities of the property system.</p>
<p>A simplified version of the view for <a href="http://batmanjs.org/examples/alfred.html">Alfred</a>, Batman’s todo manager example application, lies below:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='html'><span class='line'><span class="nt"><h1></span>Alfred<span class="nt"></h1></span>
</span><span class='line'>
</span><span class='line'><span class="nt"><ul</span> <span class="na">id=</span><span class="s">"items"</span><span class="nt">></span>
</span><span class='line'> <span class="nt"><li</span> <span class="na">data-foreach-todo=</span><span class="s">"Todo.all"</span> <span class="na">data-mixin=</span><span class="s">"animation"</span><span class="nt">></span>
</span><span class='line'> <span class="nt"><input</span> <span class="na">type=</span><span class="s">"checkbox"</span> <span class="na">data-bind=</span><span class="s">"todo.isDone"</span> <span class="na">data-event-change=</span><span class="s">"todo.save"</span> <span class="nt">/></span>
</span><span class='line'> <span class="nt"><label</span> <span class="na">data-bind=</span><span class="s">"todo.body"</span> <span class="na">data-addclass-done=</span><span class="s">"todo.isDone"</span> <span class="na">data-mixin=</span><span class="s">"editable"</span><span class="nt">></label></span>
</span><span class='line'> <span class="nt"><a</span> <span class="na">data-event-click=</span><span class="s">"todo.destroy"</span><span class="nt">></span>delete<span class="nt"></a></span>
</span><span class='line'> <span class="nt"></li></span>
</span><span class='line'> <span class="nt"><li><span</span> <span class="na">data-bind=</span><span class="s">"Todo.all.length"</span><span class="nt">></span></span> <span class="nt"><span</span> <span class="na">data-bind=</span><span class="s">"'item' | pluralize Todo.all.length"</span><span class="nt">></span></li></span>
</span><span class='line'><span class="nt"></ul></span>
</span><span class='line'><span class="nt"><form</span> <span class="na">data-formfor-todo=</span><span class="s">"controllers.todos.emptyTodo"</span> <span class="na">data-event-submit=</span><span class="s">"controllers.todos.create"</span><span class="nt">></span>
</span><span class='line'> <span class="nt"><input</span> <span class="na">class=</span><span class="s">"new-item"</span> <span class="na">placeholder=</span><span class="s">"add a todo item"</span> <span class="na">data-bind=</span><span class="s">"todo.body"</span> <span class="nt">/></span>
</span><span class='line'><span class="nt"></form></span>
</span></code></pre></td></tr></table></div></figure>
<p>We sacrifice any sort of transpiler layer (no HAML), and any sort of template layer (no Eco, jade, or mustache). Our views are valid HTML5, rendered by the browser as soon as they have been downloaded. They aren’t JavaScript strings, they are valid DOM trees which Batman traverses and populates with data without any compilation or string manipulation involved. The best part is that Batman “binds” a node’s value by observing the value using the runtime as presented above. When the value changes in JavaScript land, the corresponding node attribute(s) bound to it update automatically, and the user sees the change. Vice versa remains true: when a user types into an input or checks a checkbox, the string or boolean is set on the bound object in JavaScript. The concept of bindings isn’t new, as you may have seen it in things like Cocoa, or in <a href="http://knockoutjs.com/">Knockout</a> or Sproutcore in JS land.</p>
<p>We chose to use bindings because we a) don’t want to have to manually check for changes to our data, and b) don’t want to have to re-render a whole template every time one piece of data changes. With mustache or <code>jQuery.tmpl</code> and company, I end up doing both those things surprisingly often. It seems wasteful to re-render every element in a loop and pay the penalty for appending all those nodes, when only one key on one element changes, and we could just update that one node. SproutCore’s ‘SC.TemplateView’ with Yehuda Katz' <a href="http://www.handlebarsjs.com/">Handlebars.js</a> do a good job of mitigating this, but we still didn’t want to do all the string ops in the browser, and so we opted for the surgical precision of binding all the data in the view to exactly the properties we want.</p>
<p>What you end up with is a fast render with no initial loading screen, at the expense of the usual level of complex logic in your views. Batman’s view engine provides conditional branching, looping, context, and simple transforms, but thats about it. It forces you to write any complex interaction code in a packaged and reusable <code>Batman.View</code> subclass, and leave the HTML rendering to the thing that does it the best: the browser.</p>
<h2>More?</h2>
<p>Batman does more than this fancy deep keypath stuff and these weird HTML views-but-not-templates. We have a routing system for linking from quasi-page to quasi-page, complete with named segments and GET variables. We have a <code>Batman.Model</code> layer for retrieving and sending data to and from a server which works out of the box with storage backends like Rails and <code>localStorage</code>. We have other handy mixins for use in your own objects like <code>Batman.StateMachine</code> and <code>Batman.EventEmitter</code>. And, we have a lot more on the ay. I strongly encourage you to check out the <a href="http://batmanjs.org/">project website</a>, the <a href="https://github.com/Shopify/batman">source on GitHub</a>, or visit us in <a href="irc://freenode.net/batmanjs">#batmanjs on freenode</a>. Any questions, feedback, or patches will be super welcome, and we’re always open to suggestions on how we can make Batman better for you.</p>
<p>Until next time….</p>
<iframe width="560" height="345" src="http://www.youtube.com/embed/X0UJaprpxrk" frameborder="0" allowfullscreen></iframe>
<p><br/>
<em>This article has been crossposted to the <a href="http://www.shopify.com/">Shopify</a> Technology blog <a href="http://www.shopify.com/technology/4080092-how-batman-can-help-you-build-app">here</a>.</em></p>
]]></content>
</entry>
<entry>
<title type="html"><![CDATA[Neat Algorithms - Harmony Search]]></title>
<link href="http://harry.me/blog/2011/07/05/neat-algorithms-harmony-search/"/>
<updated>2011-07-05T12:41:00-04:00</updated>
<id>http://harry.me/blog/2011/07/05/neat-algorithms-harmony-search</id>
<content type="html"><![CDATA[<p>Here I’ll try and demonstrate a neat optimization algorithm based on the principles of performing jazz musicians by applying it to solve Sudoku puzzles. <strong>Update Sept 28th 2015</strong>: Turns out this algorithm is ballyhoo and I don’t like it any more, use something else. Kind of a fun idea though. See <a href="http://www.dennisweyland.net/blog/?p=12">http://www.dennisweyland.net/blog/?p=12</a>.</p>
<p>Harmony Search (often abbreviated HS) is a <a href="http://en.wikipedia.org/wiki/Metaheuristic">metaheuristic optimization</a> algorithm pioneered by <a href="https://sites.google.com/a/hydroteq.com/www/">Dr Zong Woo Geem</a>. Metaheuristic algorithms like harmony search attempt to find the optimal input to some objecting measure of quality, or in other words, find the “best” solution to a given problem. Harmony search has been successfully applied to a vast array of such problems, such as the Travelling Salesman problem, water network design, and actual algorithmic music generation.</p>
<!--more-->
<p>See the algorithm in action:</p>
<div id="searchVis"></div>
<h1>Table of Contents</h1>
<ol>
<li><a href="#searchVis">Demo</a></li>
<li><a href="#heuristics">Intro to Metaheuristics</a></li>
<li><a href="#harmony_search">Harmony Search</a></li>
<li><a href="#exam_example">Exam Mark Demo</a></li>
<li><a href="#sudoku_example">Sudoku Demo and Analysis</a></li>
</ol>
<h1>About this page</h1>
<p>This page features interactive demos and code examples, all written in <a href="http://coffeescript.org/">Coffeescript</a>. It shouldn’t be too hard to pick up if you haven’t seen it before, but visit that page if you want a quick primer on the syntax. If thats too much to ask, know that variables prefixed with <code>@</code> symbols signify instance variables, and that the last value of a block is the implicit return value, and you should be good. The example code you see in the post is also a distilled, unoptimized, nuance-lacking version which gets rid of the boring stuff for your benefit, so don’t make fun if it seems slow.</p>
<p>Also, the computationally intense demo above has an intensity setting you can pick. Pick <code>poutine</code> mode only if you run Chrome or want to watch your browser get crushed. The first three settings defer to the UI thread often enough to stay responsive, but <code>poutine</code> mode uses web workers to destroy FF3, FF4, and Opera on my machine faster than you can say “higgitus figgitus”. <code>Poutine</code> mode is called as such because the browser gobbles up CPU power like I gobble up the aforementioned artery clogger at 3 AM on a Saturday night. Very quickly.</p>
<h1 id="heuristics">Heuristics and Fitness</h1>
<p>Harmony search as well as the above mentioned algorithms are useful for solving a very wide class of problems. Below we’ll apply it to one problem in particular, but first lets examine the role of a metaheuristic algorithm.</p>
<p>The central idea is that when trying to solve some given optimization problem, you have some set of input variables that can be evaluated for their quality, and you want to know what inputs produce the best quality. Metaheuristic algorithms try to find this global optimum using some strategy which is better than brute force. For problems where it is hard to decipher why changing an input changes the quality (and thus the optimal solution isn’t very obvious), these algorithms are extremely useful. Harmony search and its siblings in this category do not guarantee that the globally optimal solution will be found, but often they do find it, and they are often much more efficient than an exhaustive brute force search of all input combinations.</p>
<h2>A Basic Example</h2>
<p>Say I have a killer exam tomorrow, and I have function which represents what mark I’ll get depending on how much time I spend studying and how much time I spend sleeping. For the sake of the example, we’ll say that I can spend a maximum of 10 hours doing either activity, and any time I don’t spend doing either activity will be filled by normal day to day activities. The problem is I’ll get burned out if I study too much, but I won’t pass if I don’t study enough. I could also be groggy during the exam I sleep too much, or be weary and slow if I don’t sleep enough. How do I balance the time before the exam appropriately, given that I have this magical function which predicts the future?</p>
<figure class="big">
<figcaption>The problem space shown as a heat map.</figcaption>
<div id="sleepMap"></div>
</figure>
<p>To the left is a heat map showing where the best marks are earned. You’ll find the hours spent studying on the X axis and the hours spent sleeping on the Y axis, and the mark achieved encoded in the color at that point on the map. A white color represents 100%, and a black color represents a failing grade. You can see that around the edges of the map the colors are darker, indicating a poorer mark. There also appears to be a hotspot right in the middle at about 5 hours spent studying and 8 hours spent sleeping. This is easy for us to see because the data is laid out in such a way we can see the whole problem space at once, and see the exact range of marks earned just by looking at the colors. Us humans can identify a pattern of increasing marks by watching the colors get closer to white as the inputs approach that sweet spot.</p>
<p>The task of an optimization algorithm is to do exactly what we do with our eyes on the heat map. It can also search non differentiable functions, or functions which few assumptions can be made. Also note that this exam example is a tad silly, because every input combination is represented in that heat map, so we could write a brute force program to just try them all and find the max pretty easily and quickly. To make it even worse, the source code for the relatively simple and continuous quality function is also in this page, so just applying some first year calculus we can find the global maximum. For computationally complex functions of many more variables, or non differentiable functions, these brute force and calculus approaches aren’t feasible, and we are left to find a better strategy.</p>
<h1 id="harmony_search">Enter Harmony Search</h1>
<p>Harmony search is one such strategy for finding an optimal set of inputs to an often complicated quality function, among others like random search, simulated annealing, genetic algorithms, and tabu search. It works by imitating the activity of musicians while improvising. The choice of which note to play next while playing as part of a trio or quartet is something which takes years to learn to do effectively, because its hard to know what notes your accompaniment might play, and its hard to know what notes might sound good or great in tandem with the others. Musicians can be seen as trying to play some set of notes simultaneously to produce a <em>fantastic harmony</em>, although this is a somewhat naive take on the whole thing, so don’t let me ruin the magic for you.</p>
<p>Each musician in the ensemble is often faced with the problem of picking the next note. To do so they can reference their knowledge of the notes in the key they are playing in (what notes sound good in the context of the song), as well as the notes they’ve played previously (what notes sound good in the recent context). The notes they played recently most likely sounded alright, so often these are a good choice. Also, it can be wise to pick a particular note that the audience might expect and adjust the pitch of it away from the expected note to create an artistic effect and a new, potentially better, harmony.</p>
<p>These decisions that said bland hypothetical musician makes correspond exactly to how harmony search works. Harmony search seeks an optimal combination of inputs, just as a musician seeks a fantastic harmony. Harmony search generates “harmonies” of inputs which it then evaluates for quality, and iterates this process until it finds the best one possible. The quality of a musical harmony is analogous to the quality of a particular solution, so you might say that harmony search is trying to achieve a <em>fantastic</em> combination of inputs, or that musicians are trying to <em>optimize</em> the note selection problem using their own heuristics.</p>
<p>Each input to the problem is seen as a different instrument in an ensemble, each potential note one of those instruments could play corresponds to each potential value on of the inputs of the function might adopt. The musical harmony of notes is modeled as a programmatic harmony of values. We evaluate the former’s quality using our ears, and the latter’s using a quality function describing the problem.</p>
<h2>Improvising New Solutions</h2>
<p>Harmony search continues to use the musician metaphor to iteratively improve its solution. Each search iteration of the algorithm generates a new harmony to evaluate for quality. Using the note selection strategies mentioned previously, notes for each instrument, or values for each input, are selected. These inputs are fed to the quality function to determine the harmony’s quality. The notes are selected for each instrument by either randomly selecting a note from the range of playable notes, selecting a note from the set of recently played ones, and/or occasionally adjusting the pitch of a note up or down.</p>
<h2>Getting better</h2>
<p>Each iteration a new harmony is generated, its quality is calculated, and if it makes the cut it’s “included” in the musician’s memory. This way, iteration by iteration, old, poor quality harmonies are kicked out and replaced by better ones. The average quality of the set of harmonies in this memory as a whole gradually increases as these new harmonies replace poor ones. The brilliance of the algorithm comes from this: the new harmonies that are generated, which you may recall often reference notes from the memory, start to use notes belonging to known high-quality harmonies. Thus, the newly generated harmonies use good notes, and often have higher qualities because of it. This process repeats, where the increasing the quality of individual harmonies generated increases the average quality of the memory, which increases the quality of the individual generated harmonies, and so on and so forth. At some point (it is hoped), the algorithm generates a harmony which meets the “fantastic” quality hoped for.</p>
<p>Thats it! Harmony search isn’t too complicated, but its a neat algorithm inspired by some everyday natural phenomena. Read on for the code and an example application.</p>
<h1>The Code</h1>
<p>First, lets more formally define some terms.</p>
<ul>
<li><strong>Instrument</strong>: One of the inputs to the quality function.</li>
<li><strong>Note</strong>: One of the possible values of an input.</li>
<li><strong>Harmony</strong>: A combination of each instrument playing a particular note, or in reality the set of chosen inputs for each argument to the quality function.</li>
<li><strong>Quality</strong>: A quantitative measure of a harmony’s desirability, how close or far it is from the <em>fantastic</em> harmony, or optimal solution.</li>
<li><strong>Harmony Memory</strong>: The collection of good harmonies, stored in memory for examination by the harmony generation algorithm.</li>
<li><strong>Harmony Memory Consideration</strong>: The process of generating a new harmony using random notes from the harmony memory.</li>
<li><strong>Pitch Adjustment</strong>: The process of moving a particular instrument’s note up or down</li>
</ul>
<h2>Pseudo code for the actual algorithm</h2>
<ol>
<li>Initialize the Parameters for Problem and Algorithm.</li>
<li>Initialize the Harmony Memory (HM).</li>
<li>Improvise a New Harmony.</li>
<li>Update the Harmony Memory if the new harmony is better than the worst harmony in the memory.</li>
<li>Check the stopping criterion, and if we can continue, go back to 3.</li>
</ol>
<h2>The Parts</h2>
<p>The algorithm, once applied to a problem, is composed of 3 main parts:</p>
<ol>
<li><strong>The harmony generator</strong>, which spits out new, potentially good harmonies based on the contents of the harmony memory and the set of possible notes</li>
<li><strong>The quality function</strong>, which evaluates a particular harmony for its quality.</li>
<li><strong>The search algorithm</strong>, which moves harmonies in and out of the memory based on their quality.</li>
</ol>
<p>I chose to encapsulate the generator and the search algorithm into a <code>HarmonySearch</code> class, and to make the whole thing reusable by making a <code>Harmony</code> class which in a particular problem would be extended to implement the quality function.</p>
<p>Next, we’ll define the formal parameters for the algorithm:</p>
<ul>
<li><strong>Harmony Memory Consideration Rate</strong> or HMCR: the probability that when generating notes for a new harmony, a note from the harmony memory will be picked, instead of just picking a random one out of the possible notes</li>
<li><strong>Pitch Adjustment Rate</strong> or PAR: the probability of randomly shifting a note up or down once it has been chosen</li>
</ul>
<h2>The skeleton</h2>
<p>Here’s the skeleton for the <code>HarmonySearch</code> class:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="k">class</span> <span class="nx">HarmonySearch</span>
</span><span class='line'> <span class="vi">@defaults:</span>
</span><span class='line'> <span class="nv">maxTries: </span><span class="mi">100</span>
</span><span class='line'> <span class="nv">targetQuality: </span><span class="kc">Infinity</span>
</span><span class='line'> <span class="nv">harmonyMemorySize: </span><span class="kc">false</span>
</span><span class='line'> <span class="nv">harmonyMemoryConsiderationRate: </span><span class="p">.</span><span class="mi">95</span>
</span><span class='line'> <span class="nv">pitchAdjustmentRate: </span><span class="p">.</span><span class="mi">1</span>
</span><span class='line'> <span class="nv">instruments: </span><span class="mi">10</span>
</span><span class='line'> <span class="nv">notes: </span><span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">9</span> <span class="p">]</span>
</span><span class='line'> <span class="nv">harmonyMemorySize: </span><span class="mi">10</span>
</span><span class='line'>
</span><span class='line'> <span class="nv">constructor: </span><span class="nf">(options) -></span>
</span><span class='line'> <span class="vi">@options = </span><span class="nx">_</span><span class="p">.</span><span class="nx">extend</span> <span class="p">{},</span> <span class="nx">HarmonySearch</span><span class="p">.</span><span class="nx">defaults</span><span class="p">,</span> <span class="nx">options</span>
</span></code></pre></td></tr></table></div></figure>
<p>All this does is define the defaults for the algorithm.</p>
<p>Here’s the basic, extendable <code>Harmony</code> class:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="k">class</span> <span class="nx">Harmony</span>
</span><span class='line'> <span class="c1"># Pull out the note and index of the note from the chord, passed in</span>
</span><span class='line'> <span class="c1"># in the [[note_1, index_1], [note_2, index_2], ...] format</span>
</span><span class='line'> <span class="nv">constructor: </span><span class="nf">(chord) -></span>
</span><span class='line'> <span class="vi">@notes = </span><span class="p">[]</span>
</span><span class='line'> <span class="vi">@noteIndicies = </span><span class="p">[]</span>
</span><span class='line'> <span class="k">for</span> <span class="nx">i</span><span class="p">,</span> <span class="nx">info</span> <span class="k">of</span> <span class="nx">chord</span>
</span><span class='line'> <span class="nx">@notes</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">=</span> <span class="nx">info</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span><span class='line'> <span class="nx">@noteIndicies</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="o">=</span> <span class="nx">info</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Cache the quality calculation</span>
</span><span class='line'> <span class="nv">quality: </span><span class="nf">-></span>
</span><span class='line'> <span class="nx">@_quality</span> <span class="o">?=</span> <span class="k">this</span><span class="p">.</span><span class="nx">calculateQuality</span><span class="p">()</span>
</span><span class='line'> <span class="nx">@_quality</span>
</span><span class='line'>
</span><span class='line'> <span class="nv">calculateQuality: </span><span class="nf">-></span>
</span><span class='line'> <span class="k">throw</span> <span class="s">"Extend this class to define how a harmony's quality is evaluated"</span>
</span></code></pre></td></tr></table></div></figure>
<p>The above class manages the generic parts of the search. To apply it to a particular optimisation problem, we subclass <code>Harmony</code> and define a quality calculation which suits the problem at hand. Below we’ll apply it to the exam mark problem mentioned above, and then after a less trivial sudoku problem.</p>
<h2>The Harmony Generator</h2>
<p>This is the first component of the <code>HarmonySearch</code> class, responsible for spitting out new harmonies based upon those stored in the harmony memory, as well as the HMCR and the PAR.</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="k">class</span> <span class="nx">HarmonySearch</span>
</span><span class='line'> <span class="c1"># Generate a totally random harmony</span>
</span><span class='line'> <span class="nv">getRandomHarmony: </span><span class="nf">-></span>
</span><span class='line'> <span class="nv">chord = </span><span class="k">for</span> <span class="nx">i</span> <span class="k">in</span> <span class="p">[</span><span class="mi">0</span><span class="p">..</span><span class="nx">@options</span><span class="p">.</span><span class="nx">instruments</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
</span><span class='line'> <span class="nv">index = </span><span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">(</span><span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o">*</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">notesLength</span><span class="p">)</span>
</span><span class='line'> <span class="p">[</span><span class="nx">@options</span><span class="p">.</span><span class="nx">notes</span><span class="p">[</span><span class="nx">index</span><span class="p">],</span> <span class="nx">index</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="k">new</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">harmonyClass</span><span class="p">(</span><span class="nx">chord</span><span class="p">)</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Generate a new harmony based on the HMCR and the PAR</span>
</span><span class='line'> <span class="nv">getNextHarmony: </span><span class="nf">-></span>
</span><span class='line'> <span class="nv">chord = </span><span class="k">for</span> <span class="nx">i</span> <span class="k">in</span> <span class="p">[</span><span class="mi">0</span><span class="p">..</span><span class="nx">@options</span><span class="p">.</span><span class="nx">instruments</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="k">if</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o"><</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">harmonyMemoryConsiderationRate</span>
</span><span class='line'> <span class="c1"># Consider HM. Pick a random harmony, and sample the note at this position in the chord</span>
</span><span class='line'> <span class="nv">harmonyMemoryIndex = </span><span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">(</span><span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span><span class="o">*</span><span class="nx">@options</span><span class="p">.</span><span class="nx">harmonyMemorySize</span><span class="p">)</span>
</span><span class='line'> <span class="nv">note = </span><span class="nx">@harmonyMemory</span><span class="p">[</span><span class="nx">harmonyMemoryIndex</span><span class="p">].</span><span class="nx">notes</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span> <span class="c1"># Grab note for this instrument</span>
</span><span class='line'> <span class="nv">noteIndex = </span><span class="nx">@harmonyMemory</span><span class="p">[</span><span class="nx">harmonyMemoryIndex</span><span class="p">].</span><span class="nx">noteIndicies</span><span class="p">[</span><span class="nx">i</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Do pitch adjustment</span>
</span><span class='line'> <span class="k">if</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o"><</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">pitchAdjustmentRate</span>
</span><span class='line'> <span class="c1"># Adjust the pitch up or down one</span>
</span><span class='line'> <span class="nv">adjustment = </span><span class="k">if</span> <span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o">></span> <span class="mf">0.5</span> <span class="k">then</span> <span class="mi">1</span> <span class="k">else</span> <span class="o">-</span><span class="mi">1</span>
</span><span class='line'> <span class="nv">noteIndex = </span><span class="p">(</span><span class="nx">noteIndex</span> <span class="o">+</span> <span class="nx">adjustment</span> <span class="o">+</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">notesLength</span><span class="p">)</span> <span class="o">%</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">notesLength</span>
</span><span class='line'> <span class="nv">note = </span><span class="nx">@options</span><span class="p">.</span><span class="nx">notes</span><span class="p">[</span><span class="nx">noteIndex</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="k">else</span>
</span><span class='line'> <span class="c1"># Don't consider the HM. Pick a random note from all possible values.</span>
</span><span class='line'> <span class="nv">noteIndex = </span><span class="nb">Math</span><span class="p">.</span><span class="nx">floor</span><span class="p">(</span><span class="nb">Math</span><span class="p">.</span><span class="nx">random</span><span class="p">()</span> <span class="o">*</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">notesLength</span><span class="p">)</span>
</span><span class='line'> <span class="nv">note = </span><span class="nx">@options</span><span class="p">.</span><span class="nx">notes</span><span class="p">[</span><span class="nx">noteIndex</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Return chosen note for the chord</span>
</span><span class='line'> <span class="p">[</span><span class="nx">note</span><span class="p">,</span> <span class="nx">noteIndex</span><span class="p">]</span>
</span><span class='line'>
</span><span class='line'> <span class="k">new</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">harmonyClass</span><span class="p">(</span><span class="nx">chord</span><span class="p">)</span>
</span></code></pre></td></tr></table></div></figure>
<p>Hopefully all this secondary stuff isn’t too confusing, but if it is, the next section brings it all together and hopefully will make it all clear.</p>
<p>Also note that each <code>Harmony</code> class stores both an array of notes and an array of note indices, which seems a tad odd. This is because in the above code the pitch adjustment portion needs access to the original index of the note in the array of possible notes, so it can find the next or previous index to adjust to. Thats why the <code>Harmony</code> class constructor accepts the <code>[[note, index], [note2, index2], ...]</code> style arguments, and the above accumulator returns <code>[note, noteIndex]</code>, instead of just doing arrays of notes.</p>
<h2>The Core</h2>
<p>Below is the core of the search algorithm, which actually executes the whole search process by generating new harmonies and moving them into the harmony memory if they are better than the current worst.</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
<span class='line-number'>9</span>
<span class='line-number'>10</span>
<span class='line-number'>11</span>
<span class='line-number'>12</span>
<span class='line-number'>13</span>
<span class='line-number'>14</span>
<span class='line-number'>15</span>
<span class='line-number'>16</span>
<span class='line-number'>17</span>
<span class='line-number'>18</span>
<span class='line-number'>19</span>
<span class='line-number'>20</span>
<span class='line-number'>21</span>
<span class='line-number'>22</span>
<span class='line-number'>23</span>
<span class='line-number'>24</span>
<span class='line-number'>25</span>
<span class='line-number'>26</span>
<span class='line-number'>27</span>
<span class='line-number'>28</span>
<span class='line-number'>29</span>
<span class='line-number'>30</span>
<span class='line-number'>31</span>
<span class='line-number'>32</span>
<span class='line-number'>33</span>
<span class='line-number'>34</span>
<span class='line-number'>35</span>
<span class='line-number'>36</span>
<span class='line-number'>37</span>
<span class='line-number'>38</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="k">class</span> <span class="nx">HarmonySearch</span>
</span><span class='line'>
</span><span class='line'> <span class="nv">search: </span><span class="nf">(callback) -></span>
</span><span class='line'> <span class="c1"># Initialize harmony memory</span>
</span><span class='line'> <span class="vi">@running = </span><span class="kc">true</span>
</span><span class='line'> <span class="vi">@harmonyMemory = </span><span class="k">for</span> <span class="nx">i</span> <span class="k">in</span> <span class="p">[</span><span class="mi">1</span><span class="p">..</span><span class="nx">@options</span><span class="p">.</span><span class="nx">harmonyMemorySize</span><span class="p">]</span>
</span><span class='line'> <span class="k">this</span><span class="p">.</span><span class="nx">getRandomHarmony</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Initialize tracker variables for the loop</span>
</span><span class='line'> <span class="vi">@tries = </span><span class="mi">0</span>
</span><span class='line'> <span class="p">[</span><span class="nx">worstQuality</span><span class="p">,</span> <span class="nx">worstIndex</span><span class="p">]</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">_getWorst</span><span class="p">()</span>
</span><span class='line'> <span class="p">[</span><span class="nx">bestQuality</span><span class="p">,</span> <span class="nx">bestIndex</span><span class="p">]</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">_getBest</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Iterate over the search until either the target quality is hit,</span>
</span><span class='line'> <span class="c1"># or the max iterations condition is passed.</span>
</span><span class='line'> <span class="k">while</span> <span class="kc">true</span>
</span><span class='line'> <span class="nx">@tries</span><span class="o">++</span> <span class="c1"># Increment tries counter each loop for end condition check</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Check end condition</span>
</span><span class='line'> <span class="k">if</span> <span class="nx">@tries</span> <span class="o">></span> <span class="nx">@options</span><span class="p">.</span><span class="nx">maxTries</span> <span class="o">||</span> <span class="nx">bestQuality</span> <span class="o">>=</span> <span class="nx">@options</span><span class="p">.</span><span class="nx">targetQuality</span>
</span><span class='line'> <span class="k">break</span> <span class="c1"># We either found it or exhausted the alloted time</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Otherwise, generate another harmony</span>
</span><span class='line'> <span class="nv">harmony = </span><span class="k">this</span><span class="p">.</span><span class="nx">getNextHarmony</span><span class="p">()</span>
</span><span class='line'> <span class="k">if</span> <span class="nx">harmony</span><span class="p">.</span><span class="nx">quality</span><span class="p">()</span> <span class="o">></span> <span class="nx">worstQuality</span>
</span><span class='line'> <span class="c1"># Better than worst harmony. Add this harmony to the memory and delete the worst.</span>
</span><span class='line'> <span class="nx">@harmonyMemory</span><span class="p">.</span><span class="nx">push</span><span class="p">(</span><span class="nx">harmony</span><span class="p">)</span>
</span><span class='line'> <span class="nx">@harmonyMemory</span><span class="p">.</span><span class="nx">splice</span><span class="p">(</span><span class="nx">worstIndex</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span> <span class="c1"># Javascript for Array#delete</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Update the worst quality for the next loop iteration</span>
</span><span class='line'> <span class="p">[</span><span class="nx">worstQuality</span><span class="p">,</span> <span class="nx">worstIndex</span><span class="p">]</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">_getWorst</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'> <span class="c1"># Track the best quality to see if we've met the target quality</span>
</span><span class='line'> <span class="k">if</span> <span class="nx">harmony</span><span class="p">.</span><span class="nx">quality</span><span class="p">()</span> <span class="o">></span> <span class="nx">bestQuality</span>
</span><span class='line'> <span class="nv">bestQuality = </span><span class="nx">harmony</span><span class="p">.</span><span class="nx">quality</span><span class="p">()</span>
</span><span class='line'>
</span><span class='line'> <span class="p">[</span><span class="nx">bestQuality</span><span class="p">,</span> <span class="nx">bestIndex</span><span class="p">]</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">_getBest</span><span class="p">()</span>
</span><span class='line'> <span class="k">return</span> <span class="nx">@harmonyMemory</span><span class="p">[</span><span class="nx">bestIndex</span><span class="p">]</span>
</span></code></pre></td></tr></table></div></figure>
<p>Thats about it! Feeling ok? Read on for a couple examples to gel all of this.</p>
<h1 id="exam_example">Exam Mark Example</h1>
<p>Consider the exam mark problem shown above. Suppose the mysterious exam mark equation has been implemented in a Javascript function called <code>Exam.mark(timeSleeping, timeStudying)</code>.</p>
<pre><code>:::coffeescript
$ Exam.mark
# => function(timeSleeping, timeStudying) { ... }
$ Exam.mark(0, 0)
# => 30
$ Exam.mark(10,10)
# => 50
</code></pre>
<p>We’re trying to find the global optimum to this equation. To model this in harmony search, we ask how many instruments there are, what notes each of them can play, and how to determine the quality of the harmony produced.</p>
<p>In this case, the <code>Exam.mark</code> equation is the one we are trying to optimize. We model its input arguments as notes, and use harmonies composed of different combinations of times. There are two instruments, one for each argument to the function, and each instrument can “play” any number between 0 and 10, which are the bounds as outlined in the problem. A harmony’s quality is the mark achieved when the time is spent in it’s particular allotment, which we model as the evaluation of the <code>Exam.mark</code> function for the two notes.</p>
<p>The harmony class we’d use for this problem would look like this:</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'> <span class="k">class</span> <span class="nx">ExamHarmony</span> <span class="k">extends</span> <span class="nx">Harmony</span>
</span><span class='line'> <span class="nv">quality: </span><span class="nf">-></span> <span class="nx">Exam</span><span class="p">.</span><span class="nx">mark</span><span class="p">(</span><span class="nx">@notes</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nx">@notes</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
</span></code></pre></td></tr></table></div></figure>
<p>That’s not so bad right? We’d then run the search for some sufficiently large number of iterations and look at the output.</p>
<figure class='code'><div class="highlight"><table><tr><td class="gutter"><pre class="line-numbers"><span class='line-number'>1</span>
<span class='line-number'>2</span>
<span class='line-number'>3</span>
<span class='line-number'>4</span>
<span class='line-number'>5</span>
<span class='line-number'>6</span>
<span class='line-number'>7</span>
<span class='line-number'>8</span>
</pre></td><td class='code'><pre><code class='coffeescript'><span class='line'><span class="nv">search = </span><span class="k">new</span> <span class="nx">HarmonySearch</span> <span class="p">{</span>
</span><span class='line'> <span class="nv">harmonyClass: </span><span class="nx">ExamHarmony</span>
</span><span class='line'> <span class="nv">notes: </span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="mi">3</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">6</span><span class="p">,</span><span class="mi">7</span><span class="p">,</span><span class="mi">8</span><span class="p">,</span><span class="mi">9</span><span class="p">,</span><span class="mi">10</span><span class="p">]</span>
</span><span class='line'> <span class="nv">instruments: </span><span class="mi">2</span>
</span><span class='line'> <span class="nv">targetQuality: </span><span class="mi">100</span>
</span><span class='line'> <span class="nv">maxIterations: </span><span class="mi">2000</span>
</span><span class='line'><span class="p">}</span>
</span><span class='line'><span class="nv">results = </span><span class="nx">search</span><span class="p">.</span><span class="nx">search</span><span class="p">()</span>
</span></code></pre></td></tr></table></div></figure>
<p>After this, results should hold the best quality <code>Harmony</code> found.</p>
<h2>Demo</h2>
<div id="examsearchVis"></div>
<p>Harmony Search is run live to power the visualization above, as well as the one at the top of the page. Each wedge in the purple circle represents a harmony in the memory, and so the circle is the whole HM. Each wedge is labeled by its quality, and as harmonies grow closer to the target quality, the color of their wedges grow more purple. The best and worst harmonies are are also highlighted using a green or red border. Feel free to click on a harmony to see it’s location on the heat map, and judge how close it is to the optimal point. Also, notice how new harmonies are added at the top of the circle and progress clockwise as new ones are added and poor ones are removed. The best harmonies will travel all the way around the circle but not get bumped out, and end up at the back for a long period of time, contributing good quality notes to the new harmonies being generated.</p>
<p>The grid of numbers to the right represents the selection of the notes in each new harmony. The harmony at the top of the list (with all the lines stemming from it) is the most recent addition to the harmony memory, and each line below is a progressively older harmony in the memory. Each note in the 1st row is generated either by picking a note from the set of those previously chosen in the memory, or by picking one randomly from the set of possible notes. Those chosen from the memory are signified by a grey line pointing towards the harmony from which the note was chosen. If pitch adjustment is applied, the line becomes blue. Otherwise, the note is chosen randomly, which is signified by a purple line pointing upwards into the set of notes possible for each note in the harmony. You’ll only notice this in the Sudoku demo above, because there isn’t enough room to show all the possibilities with the exam example.</p>
<h1 id="sudoku_example">Sudoku Example</h1>
<p>Harmony search can be applied to more complex problems than simple functions like the above. Sudoku is a specific case of the graph coloring problem, one of <a href="http://en.wikipedia.org/wiki/Karp%E2%80%99s_21_NP-complete_problems">Karp’s 21 NP-complete problems</a>. In other words, its very time consuming to brute force the solution to a sudoku by just trying random numbers and seeing if they work. There are excellent algorithms that often run faster than harmony search or any of its metaheuristic brethren which solve the sudoku using intelligent, problem aware methods and guess when needed.</p>
<p>These “smart” solvers are I’m sure the algorithms employed by true Sudoku software, but they rely on intimate knowledge of the Sudoku solving process and an understanding of the techniques used. We have to encode our knowledge of how to solve sudokus into a program, implementing the guessing feature, the backtracking, and all the methods for eliminating possibilities for a particular cell. Instead of developing an algorithm like this, we can use a search method to find us a solution as long as we have a heuristic to tell the quality of a given solution. By solving them in this way, we don’t need to concern ourselves with finding a general method or exploring edge cases or algorithmic nuances, and we let the search algorithm figure these things out on its own. We are able to lift the burden of understanding the relationship between the input variables from our own shoulders, and instead allow the algorithm to explore these relationships itself.</p>
<p>Hopefully you can see the advantage of using a search algorithm for problems where the smart, human written implementation is hard or impossible to create. If we have some measure of quality for a solution, and thus a way to tell when a solution is optimal, we can let the search algorithm, well, search.</p>
<h2>The Sudoku Model</h2>
<p>Let’s solve a particular Sudoku puzzle using harmony search. First, let us identify what the notes of a harmony are, and after, how to calculate the quality of one.</p>
<p>First off, notice that for any solution to be considered as such, each cell must have a value. Some of the values are given by the puzzle, and some must be decided by us. We seek a choice for each cell such that there are no conflicts, or in other words, the optimal solution to a sudoku is one which has all the cells filled in and breaks no rules.</p>
<p>We model the value of each one of the unknown cells as one note in a harmony, with the note’s value being an integer between 1 and 9. The harmony is the chord struck when we insert each of these choices into the puzzle, and the quality of the solution is how close to a valid filled-in puzzle this solution is. The order the array of notes is entered into the puzzle doesn’t really matter all that much, as long as it is consistent the algorithm will work just the same. The number of instruments is the count of unsolved cells.</p>