/
index.html
2048 lines (1911 loc) · 151 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype html>
<html>
<head>
<meta charset="utf-8"/>
<title>Scuttlebutt Protocol Guide</title>
<link rel="icon" href="img/favicon.png"/>
<link rel="stylesheet" href="css/fonts.css"/>
<link rel="stylesheet" href="css/guide.css"/>
</head>
<body>
<header>
<div class="title">
<a href="index.html">
<img src="img/logo.png" alt=""/>
<span>Scuttlebutt Protocol Guide</span>
</a>
</div>
<nav>
<a href="https://github.com/ssbc/scuttlebutt-protocol-guide"><div>Repo</div></a>
</nav>
</header>
<main>
<h1>Scuttlebutt Protocol Guide</h1>
<p class="subtitle">How Scuttlebutt peers find and talk to each other</p>
<hr/>
<aside class="kicker section-kicker">
<h2 id="toc">Contents</h2>
</aside>
<div class="toc">
<div class="col">
<div><a class="s1" href="#introduction">Introduction</a></div>
<div><a class="s1" href="#keys-and-identities">Keys and identities</a></div>
<div class="section">
<div><a class="s1" href="#discovery">Discovery</a></div>
<div class="s2">
<div><a href="#local-network">Local network</a></div>
<div><a href="#invite-code">Invite code</a></div>
<div><a href="#pub-message">Pub message</a></div>
</div>
</div>
<div class="section">
<div><a class="s1" href="#peer-connections">Peer connections</a></div>
<div class="s2">
<div><a href="#handshake">Handshake</a></div>
<div><a href="#box-stream">Box stream</a></div>
<div><a href="#rpc-protocol">RPC protocol</a></div>
</div>
</div>
</div>
<div class="col">
<div class="section">
<div><a class="s1" href="#feeds">Feeds</a></div>
<div class="s2">
<div><a href="#structure">Structure</a></div>
<div><a href="#message-format">Message format</a></div>
<div><a href="#createHistoryStream">createHistoryStream</a></div>
<div><a href="#ebt-replication">EBT replication</a></div>
</div>
</div>
<div class="section">
<div><a class="s1" href="#metafeeds">Metafeeds</a></div>
<div class="s2">
<div><a href="#bendy-butt">Structure</a></div>
<div><a href="#metafeed-contentSection">Content setion</a></div>
<div><a href="#metafeed-announcing">Announcing</a></div>
<div><a href="#metafeed-use-cases">Use Cases</a></div>
<div><a href="#metafeed-partial-replication">Partial Replication</a></div>
<div><a href="#metafeed-fusion-identity">Fusion Identity</a></div>
<div><a href="#metafeed-network-identity">Network Identity</a></div>
</div>
</div>
</div>
<div class="col">
<div class="section">
<div><a class="s1" href="#blobs">Blobs</a></div>
<div class="s2">
<div><a href="#fetching">Fetching</a></div>
<div><a href="#want-and-have">Want and have</a></div>
</div>
</div>
<div class="section">
<div><a class="s1" href="#following">Following</a></div>
<div class="s2">
<div><a href="#follow-graph">Follow graph</a></div>
</div>
</div>
<div class="section">
<div><a class="s1" href="#pubs">Pubs</a></div>
<div class="s2">
<div><a href="#invites">Invites</a></div>
</div>
</div>
</div>
<div class="col">
<div class="section">
<div><a class="s1" href="#rooms">Rooms</a></div>
<div class="s2">
<div><a href="#rooms-tunnel">Tunnel</a></div>
<div><a href="#rooms-joining">Joining</a></div>
</div>
</div>
<div class="section">
<div><a class="s1" href="#private-messages">Private messages</a></div>
<div class="s2">
<div><a href="#encrypting">Encrypting</a></div>
<div><a href="#decrypting">Decrypting</a></div>
</div>
</div>
</div>
</div>
<hr/>
<p id="introduction" class="first"><a href="https://www.scuttlebutt.nz/">Scuttlebutt</a> is a protocol for building decentralized applications that work well offline and that no one person can control. Because there is no central server, Scuttlebutt clients connect to their peers to exchange information. This guide describes the protocols used to communicate within the Scuttlebutt network.</p>
<p>Scuttlebutt is a flexible protocol, capable of supporting many different types of applications. One of its first applications was as a social network. This guide has a slight focus on how to use Scuttlebutt for social networking, but many of the explanations will still be useful if want to use it for something completely different, or are just curious how it works.</p>
<h2 id="cryptography">Cryptography</h2>
<h3 id="keys-and-identities">Keys and identities</h3>
<p>The first thing a user needs to participate in Scuttlebutt is an identity. An identity is an Ed25519 key pair and typically represents a person, a device, a server or a bot. It’s normal for a person to have several Scuttlebutt identities.</p>
<p>Upon starting for the first time, Scuttlebutt clients will automatically generate an Ed25519 key pair and save it in the user’s home folder under <code>.ssb/secret</code>.</p>
<img src="img/identity_keypair.png" style="height: 80px;" alt="The Scuttlebutt identity is a long-term Ed25519 key pair." />
<p>Because identities are long and random, no coordination or permission is required to create a new one, which is essential to the network’s design.</p>
<p>Later, a user can choose to give themselves a nickname or avatar to make themselves easier to refer to. Over time nicknames may change but identities stay the same. If a user loses their secret key or has it stolen they will need to generate a new identity and tell people to use their new one instead.</p>
<p>The public key of an identity is presented to users and transmitted in some parts of the network protocol using this format:</p>
<img src="img/format_public_key.png" style="height: 80px;" alt="@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519 where everything but the @ prefix and .ed25519 suffix is the public-key, base64-encoded." />
<aside style="position: relative; top: 6px;">
<p>Throughout the protocol all instances of base64 are the variant that uses <code>+</code> and <code>/</code>. The final padding <code>=</code> is also required.</p>
</aside>
<p>The beginning <code>@</code> sign signifies that this represents a public key rather than a message or blob, which start with <code>%</code> and <code>&</code>. Each identity has an associated <a href="#feeds">feed</a>, which is a list of all the messages posted by that identity. This is why the identity is also called a <em>feed ID</em>.</p>
<h3 id="cryptographic-primitives">Cryptographic primitives</h3>
<p>The Scuttlebutt protocol relies on NaCl/libsodium's cryptobox primitives. This guide uses the following:</p>
<dl>
<dt><code>nacl_scalarmult(n, p)</code></dt>
<dd>This is <a href="https://doc.libsodium.org/advanced/scalar_multiplication">Libsodium's scalar multiplication function</a>, which takes two scalars (usually public and/or secret keys). It has the useful property that, given two key pairs <code>(pk1, sk1)</code> and <code>(pk2, sk2)</code>, <code>nacl_scalarmult(sk1, pk2) == nacl_scalarmult(sk2, pk1)</code>, which allows shared secret derivation between peers who know each other's public key. More on this later.</dd>
<dt><code>nacl_auth(msg, key)</code> and <code>assert_nacl_auth_verify(authenticator, msg, key)</code></dt>
<dd>This functions are <a href="https://doc.libsodium.org/public-key_cryptography/authenticated_encryption">Libsodium's message authentication function</a>. The former takes a message and returns a 32-bytes authenticator, that acts as a detacted signature of the message. The latter verifies this authenticator is indeed valid for the given message and key; and errors if they don't.</dd>
<dt><code>nacl_secret_box(msg, nonce, key)</code> and <code>assert_nacl_secretbox_open(ciphertext, nonce, key)</code></dt>
<dd>These function are based on <a href="https://doc.libsodium.org/secret-key_cryptography/secretbox">Libsodium's crypto_secretbox_easy and crypto_secretbox_open_easy function</a>, which use symmetric cryptography to, respectively, encrypt+authenticate, and verify+decrypt a message using a nonce and a shared secret.</dd>
<dt><code>nacl_sign_detached(msg, key)</code> and <code>assert_nacl_sign_verify_detached(sig, msg, key)</code></dt>
<dd>The former is computed from <a href="https://doc.libsodium.org/public-key_cryptography/public-key_signatures">Libsodium's signature functions</a>. Unlike the usual Libsodium/NaCl functions, they work with signatures in independent buffers, rather than concatenated with the msg.</dd>
<dt><code>pk_to_curve25519(ed25519_pk)</code> and <code>sk_to_curve25519(ed25519_sk)</code></dt>
<dd>These functions convert Ed25519 keys (used for cryptobox) to Curve25519 (aka X25519) keys, used for signing. They are <a href="https://doc.libsodium.org/advanced/ed25519-curve25519">implemented by Libsodium as <code>crypto_sign_ed25519_pk_to_curve25519</code> and <code>crypto_sign_ed25519_sk_to_curve25519</code></a>, respectively.</dd>
</dl>
<h2 id="discovery">Discovery</h2>
<div>
<p>After a user has generated their identity they need to find some peers to connect to. To connect to a peer you need its public key and its address using any of the protocol it supports. Typically with TCP/IP, you would need its IP address and port, but the Scuttlebutt protocol is not restricted to TCP/IP as transport. The Scuttlebutt protocol currently has three methods for peers to discover each other.</p>
<h3 id="local-network">Local network</h3>
<p>Peers constantly broadcast UDP packets on their local network advertising their presence. The body of each packet is a string containing the peer’s IP address, port and base64-encoded public key (without <code>@</code> or <code>.ed25519</code>):</p>
</div>
<aside class="impl">
<img class="icon" src="img/impl.png" alt=""/>
<h5>Implementations</h5>
<div class="lang">JS</div>
<div class="vs"><a href="https://github.com/dominictarr/broadcast-stream/blob/master/index.js">broadcast-stream</a></div>
<div class="vs"><a href="https://github.com/ssbc/ssb-local/blob/master/index.js">ssb-local</a></div>
<div class="lang">Java</div>
<div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-discovery/src/main/java/org/apache/tuweni/scuttlebutt/discovery/ScuttlebuttLocalDiscoveryService.java">LocalDiscoveryService</a></div>
</aside>
<div>
<figure class="packets">
<div class="packet">
<div class="header">
<span class="key">Source IP</span><span class="value">192.168.1.123</span>
<span class="key">Source port</span><span class="value">8008</span>
<span class="key">Destination IP</span><span class="value">255.255.255.255</span>
<span class="key">Destination port</span><span class="value">8008</span>
</div>
<img src="img/format_udp_broadcast.png" style="height: 40px;" alt="net:192.168.1.123:8008:~shs:FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY="/>
</div>
</figure>
<p>This message format can be any valid <a href="https://github.com/ssb-js/multiserver#address-format">multiserver address</a>. In local networks, this is usually a <code>net</code> address with an IPv4 or IPv6 address.</p>
<p>Current implementations broadcast one of these packets every second. When a peer sees another peer’s broadcast packet they can connect to exchange messages. Some clients show nearby peers in the user interface so that the user can see other people on the same network as them.</p>
</div>
<aside>
<p>UDP source and destination ports are set to the same port number that the peer is listening on TCP for peer connections (normally 8008).</p>
</aside>
<h3 id="invite-code">Invite code</h3>
<p><a href="#invites">Invite codes</a> help new users get connected to their first <a href="#pubs">pub</a>, which is a Scuttlebutt peer that is publicly accessible over the internet. An invite code contains a pub’s domain name, port and public key.</p>
<p>They also contain a secret key that the user can <a href="#redeeming-invites">redeem</a> to make the pub <a href="#following">follow</a> them back. This lets the new user see messages posted by other members of the pub and share their own messages. Invite codes are the most common way for new users to get started on Scuttlebutt.</p>
<p>Pub operators can distribute invite codes any way they see fit, for example by posting them on existing social networks. Some pubs have a web page that anybody can visit to generate an invite code.</p>
<h3 id="pub-message">Pub message</h3>
<p>Users can post a message to their own <a href="#feeds">feed</a> advertising a pub:</p>
<aside class="kicker">
<p>Here the user <code>@FCX/ts…</code> is advertising that they know of pub <code>@VJM7w1…</code> along with the pub’s domain name and port.</p>
</aside>
<pre><code>{
"author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
"content": {
"type": "pub",
"address": {
"host": "one.butt.nz",
"port": 8008,
"key": "@VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=.ed25519"
}
},
…
}</code></pre>
<p>When others see this message they can make a note that this pub exists and connect to it in the future.</p>
<p>Pub messages are a useful way to find additional peers if you already know a few. Obviously this doesn’t work for new users who don’t know anyone else yet and therefore can’t see any pub messages.</p>
<h2 id="peer-connections">Peer connections</h2>
<p>Once a Scuttlebutt client has discovered the IP address, port number and public key of a peer they can connect via TCP to ask for updates and exchange messages.</p>
<h3 id="handshake">Handshake</h3>
<div>
<p>The connection begins with a 4-step handshake to authenticate each peer and set up an encrypted channel.</p>
<img src="img/message_flow.png" style="height: 350px;" alt="Message 1: Client hello (sent by the client). Message 2: Server hello. Message 3: Client authenticate. Message 4: Server accept" />
</div>
<aside class="impl">
<img class="icon" src="img/impl.png" alt=""/>
<h5>Implementations</h5>
<div class="lang">JS</div>
<div class="vs"><a href="https://github.com/auditdrivencrypto/secret-handshake/blob/master/protocol.js">protocol.js</a></div>
<div class="vs"><a href="https://github.com/auditdrivencrypto/secret-handshake/blob/master/crypto.js">crypto.js</a></div>
<div class="lang">Py</div>
<div class="vs"><a href="https://github.com/pferreir/PySecretHandshake/blob/master/secret_handshake/crypto.py">crypto.py</a></div>
<div class="lang">Go</div>
<div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/secrethandshake/state.go">state.go</a></div>
<div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/secrethandshake/conn.go">conn.go</a></div>
<div class="lang">C</div>
<div class="vs"><a href="https://github.com/AljoschaMeyer/shs1-c/blob/master/src/shs1.c">shs1.c</a></div>
<div class="vs"><a href="https://git.scuttlebot.io/%25133ulDgs%2FoC1DXjoK04vDFy6DgVBB%2FZok15YJmuhD5Q%3D.sha256/blob/fd953a1e72b4b16e6e5a74bcf2f893dbf1407ce4/sbotc.c">sbotc.c</a></div>
<div class="lang">Java</div>
<div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-handshake/src/main/java/org/apache/tuweni/scuttlebutt/handshake/SecureScuttlebuttHandshakeClient.java">HandshakeClient</a></div>
<div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-handshake/src/main/java/org/apache/tuweni/scuttlebutt/handshake/SecureScuttlebuttHandshakeServer.java">HandshakeServer</a></div>
</aside>
<p>The handshake uses the <a href="https://dominictarr.github.io/secret-handshake-paper/shs.pdf">Secret Handshake key exchange</a> which is designed to have these security properties:</p>
<ul>
<li>After a successful handshake the peers have verified each other’s public keys.</li>
<li>The handshake produces a shared secret that can be used with a bulk encryption cypher for exchanging further messages.</li>
<li>The client must know the server’s public key before connecting. The server learns the client’s public key during the handshake.</li>
<li>Once the client has proven their identity the server can decide they don’t want to talk to this client and disconnect without confirming their own identity.</li>
<li>A man-in-the-middle cannot learn the public key of either peer.</li>
<li>Both peers need to know a key that represents the particular Scuttlebutt network they wish to connect to, however a man-in-the-middle can’t learn this key from the handshake. If the handshake succeeds then both ends have confirmed that they wish to use the same network.</li>
<li>Past handshakes cannot be replayed. Attempting to replay a handshake will not allow an attacker to discover or confirm guesses about the participants’ public keys.</li>
<li>Handshakes provide forward secrecy. Recording a user’s network traffic and then later stealing their secret key will not allow an attacker to decrypt their past handshakes.</li>
</ul>
<aside>
<p><strong>Client</strong> is the computer initiating the TCP connection and <strong>server</strong> is the computer receiving it. Once the handshake is complete this distinction goes away.</p>
</aside>
<h4 id="starting-keys">Starting keys</h4>
<p>Upon starting the handshake, the client and server know these keys:</p>
<img src="img/starting_keys.png" style="height: 590px;" alt="Both the client and servers know: their own long term key pair, their own ephemeral key pair, and the network's (private) identifier. Additionally, the client knows the server's long term public key."/>
<h4 id="client-hello">1. Client hello</h4>
<img src="img/client_hello.png" style="height: 120px;" alt="The client sends their own ephemeral public key, hmac-authenticated using the network identifier"/>
<figure class="left-right code">
<h5 class="left">Client sends <span class="msgsize">(64 bytes)</span></h5>
<h5 class="right">Server verifies</h5>
<pre class="left"><code>concat(
nacl_auth(
msg: client_ephemeral_pk,
key: network_identifier
),
client_ephemeral_pk
)</code></pre>
<pre class="right"><code>assert(length(msg1) == 64)
client_hmac = first_32_bytes(msg1)
client_ephemeral_pk = last_32_bytes(msg1)
assert_nacl_auth_verify(
authenticator: client_hmac,
msg: client_ephemeral_pk,
key: network_identifier
)</code></pre>
</figure>
<div>
<p>First the client sends their <img class="inline-key" src="img/key_little_a_public.png" alt="public"/> generated ephemeral key. Also included is an hmac that indicates the client wishes to use their key with this specific instance of the Scuttlebutt network.</p>
<p>The <img class="inline-key" src="img/key_big_n.png" alt=""/> network identifier is a fixed key. On the main Scuttlebutt network it is the following 32-byte sequence:</p>
<div class="fixed-key binary">
<span>d4</span><span>a1</span><span>cb</span><span>88</span><span>a6</span><span>6f</span><span>02</span><span>f8</span><span>db</span><span>63</span><span>5c</span><span>e2</span><span>64</span><span>41</span><span>cc</span><span>5d</span>
<span>ac</span><span>1b</span><span>08</span><span>42</span><span>0c</span><span>ea</span><span>ac</span><span>23</span><span>08</span><span>39</span><span>b7</span><span>55</span><span>84</span><span>5a</span><span>9f</span><span>fb</span>
</div>
<p>Changing the key allows separate networks to be created, for example private networks or testnets. An eavesdropper cannot extract the network identifier directly from what is sent over the wire, although they could confirm a guess that it is the main Scuttlebutt network because that identifier is publicly known.</p>
<p>The server stores the client’s ephemeral public key and uses the hmac to verify that the client is using the same network identifier.</p>
</div>
<aside>
<p><strong>hmac</strong> is a function that allows verifying that a message came from someone who knows the same secret key as you. In this case the network identifier is used as the secret key.</p>
<p>Both the message creator and verifier have to know the same message and secret key for the verification to succeed, but the secret key is not revealed to an eavesdropper. </p>
<p>Throughout the protocol, all instances of hmac use HMAC-SHA-512-256 (which is the first 256 bits of HMAC-SHA-512).</p>
</aside>
<h4 id="server-hello">2. Server hello</h4>
<img src="img/server_hello.png" style="height: 120px;" alt="The server sends their own ephemeral public key, hmac-authenticated using the network identifier"/>
<figure class="left-right code">
<h5 class="left">Client verifies</h5>
<h5 class="right">Server sends <span class="msgsize">(64 bytes)</span></h5>
<pre class="left"><code>assert(length(msg2) == 64)
server_hmac = first_32_bytes(msg2)
server_ephemeral_pk = last_32_bytes(msg2)
assert_nacl_auth_verify(
authenticator: server_hmac,
msg: server_ephemeral_pk,
key: network_identifier
)</code></pre>
<pre class="right"><code>concat(
nacl_auth(
msg: server_ephemeral_pk,
key: network_identifier
),
server_ephemeral_pk
)</code></pre>
</figure>
<p>The server responds with their own <img class="inline-key" src="img/key_little_b_public.png" alt="public"/> ephemeral public key and hmac. The client stores the key and verifies that they are also using the same network identifier.</p>
<h4 id="shared-secret-derivation-1">Shared secret derivation</h4>
<img src="img/shared_secret_derivation_1.png" style="height: 220px;" alt="Each derivation uses one public key (their peer's) and one secret key (their own). The resultting shared secrets are identical between server and client."/>
<figure class="left-right code">
<h5 class="left">Client computes</h5>
<h5 class="right">Server computes</h5>
<pre class="left"><code>shared_secret_ab = nacl_scalarmult(
client_ephemeral_sk,
server_ephemeral_pk
)
shared_secret_aB = nacl_scalarmult(
client_ephemeral_sk,
pk_to_curve25519(server_longterm_pk)
)</code></pre>
<pre class="right"><code>shared_secret_ab = nacl_scalarmult(
server_ephemeral_sk,
client_ephemeral_pk
)
shared_secret_aB = nacl_scalarmult(
sk_to_curve25519(server_longterm_sk),
client_ephemeral_pk
)</code></pre>
</figure>
<div>
<p>Now that ephemeral keys have been exchanged, both ends use them to derive a shared secret <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/> using scalar multiplication.</p>
<p>The client and server each combine their own ephemeral secret key with the other’s ephemeral public key to produce the same shared secret on both ends. An eavesdropper doesn’t know either secret key so they can’t generate the shared secret. A man-in-the-middle could swap out the ephemeral keys in Messages 1 and 2 for their own keys, so the shared secret <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/> alone is not enough for the client and server to know that they are talking to each other and not a man-in-the-middle.</p>
<p>Because the client already knows the <img class="inline-key" src="img/key_big_b_public.png" alt=""/> server’s long term public key, both ends derive a second secret <img class="inline-key" src="img/key_little_a_big_b.png" alt="using the client's ephemeral key pair (either the public or the secret key) and the server's permanent key pair (respectively either the secret or private key)"/> that will allow the client to send a message that only the real server can read and not a man-in-the-middle.</p>
</div>
<aside>
<p><strong>Scalar multiplication</strong> is a function for deriving shared secrets from a pair of secret and public Curve25519 keys.</p>
<p>The order of arguments matters. In the NaCl API the secret key is provided first.</p>
<p>Note that long term keys are Ed25519 and must first be converted to Curve25519.</p>
</aside>
<h4 id="client-accept">3. Client authenticate</h4>
<img src="img/client_authenticate.png" style="height: 330px;" alt="The client computes a detached signature of the network identifier, the server's long-term public key, and a hash of the shared secret; signed with its permanent secret key. They add their permanent public key, and encrypt both so that they can only be opened by someone knowing the network identifier and both shared secrets; then send the cyphertext to the server." />
<figure class="left-right code">
<h5 class="left">Client computes</h5>
<h5 class="right">Server verifies</h5>
<pre class="left"><code>detached_signature_A = nacl_sign_detached(
msg: concat(
network_identifier,
server_longterm_pk,
sha256(shared_secret_ab)
),
key: client_longterm_sk
)</code></pre>
<pre class="right" style="grid-row: span 3;"><code>msg3_plaintext = assert_nacl_secretbox_open(
ciphertext: msg3,
nonce: 24_bytes_of_zeros,
key: sha256(
concat(
network_identifier,
shared_secret_ab,
shared_secret_aB
)
)
)
assert(length(msg3_plaintext) == 96)
detached_signature_A = first_64_bytes(msg3_plaintext)
client_longterm_pk = last_32_bytes(msg3_plaintext)
assert_nacl_sign_verify_detached(
sig: detached_signature_A,
msg: concat(
network_identifier,
server_longterm_pk,
sha256(shared_secret_ab)
),
key: client_longterm_pk
)</code></pre>
<h5 class="left">Client sends <span class="msgsize">(112 bytes)</span></h5>
<pre class="left"><code>nacl_secret_box(
msg: concat(
detached_signature_A,
client_longterm_pk
),
nonce: 24_bytes_of_zeros,
key: sha256(
concat(
network_identifier,
shared_secret_ab,
shared_secret_aB
)
)
)</code></pre>
</figure>
<div>
<p>The client reveals their identity to the server by sending their <img class="inline-key" src="img/key_big_a_public.png" alt=""/> long term public key. The client also makes a signature using their <img class="inline-key" src="img/key_big_a_secret.png" alt=""/> long term secret key. By signing the keys used earlier in the handshake the client proves their identity and confirms that they do indeed wish to be part of this handshake.</p>
<p>The client’s message is enclosed in a secret box to ensure that only the server can read it. Upon receiving it, the server opens the box, stores the client’s long term public key and verifies the signature.</p>
<p>An all-zero nonce is used for the secret box. The secret box construction requires that all secret boxes using a particular key must use different nonces. It’s important to get this detail right because reusing a nonce will allow an attacker to recover the key and encrypt or decrypt any secret boxes using that key. Using a zero nonce is allowed here because this is the only secret box that ever uses the key <span class="key-formula">sha256(concat( <img class="inline-key" src="img/key_big_n.png" alt=""/>, <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/>, <img class="inline-key" src="img/key_little_a_big_b.png" alt=""/>))</span>.</p>
</div>
<aside>
<p><strong>Detached signatures</strong> do not contain a copy of the message that was signed, only a tag that allows verifying the signature if you already know the message.</p>
<p>Here it is okay because the server knows all the information needed to reconstruct the message that the client signed.</p>
</aside>
<h4 id="shared-secret-derivation-2">Shared secret derivation</h4>
<img src="img/shared_secret_derivation_2.png" style="height: 220px;" alt="The client computes a new shared secret from their permanent secret key and the server's ephemeral public key. The server computes the same shared secret from the client's permanent public key and their own ephemeral secret key."/>
<figure class="left-right code">
<h5 class="left">Client computes</h5>
<h5 class="right">Server computes</h5>
<pre class="left"><code>shared_secret_Ab = nacl_scalarmult(
sk_to_curve25519(client_longterm_sk),
server_ephemeral_pk
)</code></pre>
<pre class="right"><code>shared_secret_Ab = nacl_scalarmult(
server_ephemeral_sk,
pk_to_curve25519(client_longterm_pk)
)</code></pre>
</figure>
<p>Now that the server knows the <img class="inline-key" src="img/key_big_a_public.png" alt=""/> client’s long term public key, another shared secret <img class="inline-key" src="img/key_big_a_little_b.png" alt=""/> is derived by both ends. The server uses this shared secret to send a message that only the real client can read and not a man-in-the-middle.</p>
<h4 id="server-accept">4. Server accept</h4>
<img src="img/server_accept.png" style="height: 400px;" alt="The server signs the network identifier, the previous detached signature, the client's permanent secret key, and the hash of the first shared secret, with their permanent secret key, as a new detached signature. They encrypt it so that they can only be opened by someone knowing the network identifier and all three shared secrets; then send the cyphertext to the client." />
<figure class="left-right code">
<h5 class="left">Client verifies</h5>
<h5 class="right">Server computes</h5>
<pre class="left" style="grid-row: span 3;"><code>detached_signature_B = assert_nacl_secretbox_open(
ciphertext: msg4,
nonce: 24_bytes_of_zeros,
key: sha256(
concat(
network_identifier,
shared_secret_ab,
shared_secret_aB,
shared_secret_Ab
)
)
)
assert_nacl_sign_verify_detached(
sig: detached_signature_B,
msg: concat(
network_identifier,
detached_signature_A,
client_longterm_pk,
sha256(shared_secret_ab)
),
key: server_longterm_pk
)</code></pre>
<pre class="right"><code>detached_signature_B = nacl_sign_detached(
msg: concat(
network_identifier,
detached_signature_A,
client_longterm_pk,
sha256(shared_secret_ab)
),
key: server_longterm_sk
)</code></pre>
<h5 class="right">Server sends <span class="msgsize">(80 bytes)</span></h5>
<pre class="right"><code>nacl_secret_box(
msg: detached_signature_B,
nonce: 24_bytes_of_zeros,
key: sha256(
concat(
network_identifier,
shared_secret_ab,
shared_secret_aB,
shared_secret_Ab
)
)
)</code></pre>
</figure>
<p>The server accepts the handshake by signing a message using their <img class="inline-key" src="img/key_big_b_secret.png" alt=""/> long term secret key. It includes a copy of the client’s previous signature. The server’s signature is enclosed in a secret box using all of the shared secrets.</p>
<p>Upon receiving it, the client opens the box and verifies the server’s signature.</p>
<p>Similarly to the previous message, this secret box also uses an all-zero nonce because it is the only secret box that ever uses the key <span class="key-formula">sha256(concat( <img class="inline-key" src="img/key_big_n.png"/>, <img class="inline-key" src="img/key_little_a_little_b.png" alt=""/>, <img class="inline-key" src="img/key_little_a_big_b.png" alt=""/>, <img class="inline-key" src="img/key_big_a_little_b.png" alt=""/>))</span>.</p>
<h4 id="handshake-complete">Handshake complete</h4>
<img src="img/final_shared_secret.png" style="height: 40px;" alt="" />
<p>At this point the handshake has succeeded. The client and server have proven their identities to each other.</p>
<p>The shared secrets established during the handshake are used to set up a pair of box streams for securely exchanging further messages.</p>
<h3 id="box-stream">Box stream</h3>
<div>
<p>Box stream is the bulk encryption protocol used to exchange messages following the handshake until the connection ends. It is designed to protect messages from being read or modified by a man-in-the-middle.</p>
<p>Each message in a box stream has a header and body. The header is always 34 bytes long and says how long the body will be.</p>
<img src="img/box_stream_overview.png" style="height: 140px;" alt="A stream is made of alternating headers (34 bytes) and bodies (1 to 4096 bytes); ending with a body followed by a 34-bytes 'goodbye' header"/>
<h4 id="sending">Sending</h4>
<p>Sending a message involves encrypting the body of the message and preparing a header for it. Two secret boxes are used; one to protect the header and another to protect the body.</p>
</div>
<aside class="impl">
<img class="icon" src="img/impl.png" alt=""/>
<h5>Implementations</h5>
<div class="lang">JS</div>
<div class="vs"><a href="https://github.com/dominictarr/pull-box-stream/blob/master/index.js">pull-box-stream</a></div>
<div class="lang">Py</div>
<div class="vs"><a href="https://github.com/pferreir/PySecretHandshake/blob/master/secret_handshake/boxstream.py">boxstream.py</a></div>
<div class="lang">Go</div>
<div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/boxstream/box.go">box.go</a></div>
<div class="vs"><a href="https://github.com/cryptoscope/secretstream/blob/ad7542b0cbda422a1ea3de7efa62a514672a2c88/boxstream/unbox.go">unbox.go</a></div>
<div class="lang">C</div>
<div class="vs"><a href="https://github.com/AljoschaMeyer/box-stream-c/blob/master/src/box-stream.c">box-stream.c</a></div>
<div class="vs"><a href="https://git.scuttlebot.io/%25133ulDgs%2FoC1DXjoK04vDFy6DgVBB%2FZok15YJmuhD5Q%3D.sha256/blob/fd953a1e72b4b16e6e5a74bcf2f893dbf1407ce4/sbotc.c">sbotc.c</a></div>
<div class="lang">Java</div>
<div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-handshake/src/main/java/org/apache/tuweni/scuttlebutt/handshake/SecureScuttlebuttStream.java">Stream</a></div>
</aside>
<img src="img/box_stream_send.png" class="pagewidth" style="height: 670px;" alt="The plaintext message body is enclosed in a secret box using the key and nonce shown below. Secret boxes put a 16-byte tag onto the front of messages so that tampering can be detected when the box is opened. This tag is sliced off the body and put inside the header. A temporary header is made of the body length (a two-bytes big-endian integer) and th previous tag. This temporary header is then encrypted too, including its own (16-bytes) authentication tag, producing a 16+2+16 bytes header."/>
<h4 id="receiving">Receiving</h4>
<p>Receiving a message involves reading the header to find out how long the body is then reassembling and opening the body secret box.</p>
<img src="img/box_stream_receive.png" class="pagewidth" style="height: 560px;" alt="Read the first 34 bytes. This is the secret box containing the header. Open this box, extract the body length and body authentication tag. Read the number of bytes specified in the header. Join the body authentication tag and encrypted body back together, open it, and read the secret text."/>
<h4 id="goodbye">Goodbye</h4>
<p>The stream ends with a special “goodbye” header. Because the goodbye header is authenticated it allows a receiver to tell the difference between the connection genuinely being finished and a man-in-the-middle forcibly resetting the underlying TCP connection.</p>
<img src="img/box_stream_goodbye.png" class="pagewidth" style="height: 230px;" alt="The 'goodbye' header is made of 18 bytes of zero, encrypted in a secret box (with a header authenticated tag like other headers)."/>
<p>When a receiver opens a header and finds that it contains all zeros then they will know that the connection is finished.</p>
<h4 id="keys-and-nonces">Keys and nonces</h4>
<p>Two box streams are used at the same time when Scuttlebutt peers communicate. One is for client-to-server messages and the other is for server-to-client messages. The two streams use different keys and starting nonces for their secret boxes.</p>
<img src="img/box_stream_params.png" class="pagewidth" style="height: 400px;" alt="The secret box key is made of a double-sha256 hash of the network identifier and three shared secrets, followed by either the server's permanent public key (for Client to Server) or the client's permanent public key (for Server to Client), both hashed again with sha256. The starting nonces are respectively the first 24 bytes of server's or the client's ephemeral public key, hmac-authenticated with the network identifier."/>
<p>The starting nonce is used for the first header in the stream (“secret box 1” in the above figures), then incremented for the first body (“secret box 2”), then incremented for the next header and so on.</p>
<h3 id="rpc-protocol">RPC protocol</h3>
<aside class="impl kicker">
<img class="icon" src="img/impl.png" alt=""/>
<h5>Implementations</h5>
<div class="lang">JS</div>
<div><a href="https://github.com/ssbc/packet-stream-codec/blob/master/index.js">packet-stream-codec</a></div>
<div class="lang">Py</div>
<div class="vs"><a href="https://github.com/pferreir/pyssb/blob/master/ssb/packet_stream.py">packet_stream.py</a></div>
<div class="vs"><a href="https://github.com/pferreir/pyssb/blob/master/ssb/muxrpc.py">muxrpc.py</a></div>
<div class="lang">Go</div>
<div class="vs"><a href="https://github.com/cryptoscope/go-muxrpc/tree/601b7be81ee6b2bd6f32b1247e4688537f696794/codec">codec</a></div>
<div class="vs"><a href="https://github.com/cryptoscope/go-muxrpc/blob/601b7be81ee6b2bd6f32b1247e4688537f696794/rpc.go">rpc.go</a></div>
<div class="lang">C</div>
<div class="vs"><a href="https://git.scuttlebot.io/%25133ulDgs%2FoC1DXjoK04vDFy6DgVBB%2FZok15YJmuhD5Q%3D.sha256/blob/fd953a1e72b4b16e6e5a74bcf2f893dbf1407ce4/sbotc.c">sbotc.c</a></div>
<div class="lang">Java</div>
<div class="vs"><a href="https://github.com/apache/incubator-tuweni/blob/master/scuttlebutt-rpc/src/main/java/org/apache/tuweni/scuttlebutt/rpc/RPCCodec.java">RPCCodec</a></div>
</aside>
<div>
<p>Scuttlebutt peers make requests to each other using an RPC protocol. Typical requests include asking for the latest messages in a particular feed or requesting a blob.</p>
<p>The RPC protocol can interleave multiple requests so that a slow request doesn’t block following ones. It also handles long-running asynchronous requests for notifying when an event occurs and streams that deliver multiple responses over time.</p>
<p>Similar to the box stream protocol, the RPC protocol consists of 9-bytes headers followed by variable-length bodies. There is also a 9-bytes goodbye message which is just a zeroed out header.</p>
<img src="img/rpc_overview.png" style="height: 155px;" alt=""/>
</div>
<aside>
<p><strong>Remote procedure calls</strong> are where a computer exposes a set of procedures that another computer can call over the network.</p>
<p>The requester tells the responder the name of the procedure they wish to call along with any arguments. The responder performs the action and returns a value back to the requester.</p>
</aside>
<p>Both peers make requests to each other at the same time using the pair of box streams that have been established. The box streams protect the RPC protocol from eavesdropping and tampering.</p>
<img src="img/rpc_alignment.png" style="height: 230px;" alt=""/>
<aside style="position: relative; top: 36px;">
<p>RPC messages are not necessarily aligned to box stream boxes.</p>
<p>Multiple RPC messages may be put inside one box or a single RPC message may be split over several boxes.</p>
</aside>
<h4 id="header-structure">Header structure</h4>
<p>RPC headers contain a set of flags to say what type of message it is, a field specifying its length and a request number which allows matching requests with their responses when there are several active at the same time.</p>
<img src="img/rpc_header.png" style="height: 750px;" alt="Headers are made of (in network order): 4 zero bits, a stream bit (1 = 'message is part of a stream'), a end/error bit (1 = 'message is the last in its stream or an error), and a 2-bits body type (00 = binary, 01 = UTF-8 string, 10 = JSON), the body length (4 bytes unsigned big-endian), and the request number (4 bytes signed big-endial)."/>
<h4 id="request-format">Request format</h4>
<p>To make an RPC request, send a JSON message containing the name of the procedure you wish to call, the type of procedure and any arguments.</p>
<p>The name is a list of strings. For a top-level procedure like <em>createHistoryStream</em> the list only has one element: <code>["createHistoryStream"]</code>. Procedures relating to blobs are grouped in the blobs namespace, for example to use <em>blobs.get</em> send the list: <code>["blobs", "get"]</code>.</p>
<div>
<p>There are three types of procedure used when Scuttlebutt peers talk to each other:</p>
<ul>
<li><em>Source</em> procedures return multiple responses over time and are used for streaming data or continually notifying when new events occur. When making one of these requests, the stream flag in the RPC header must be set.</li>
<li><em>Duplex</em> procedures are similar to <em>source</em> procedures but allow <em>multiple requests</em> as well as multiple responses over time. The many request events in a duplex utilize the same request number, and the stream flag must be set.</li>
<li><em>Async</em> procedures return a single response. Async responses can arrive quickly or arrive much later in response to a one-off event.</li>
</ul>
<p>For each procedure in the RPC protocol you must already know whether it is source or async and correctly specify this in the request body.</p>
</div>
<aside>
<p>The reference Scuttlebot implementation also has other internal procedures and procedure types which are used by graphical user interfaces like Patchwork.</p>
<p>This guide only covers the procedures that are publicly available to other Scuttlebutt peers.</p>
</aside>
<h4 id="source-example">Source example</h4>
<p>This RPC message shows an example of a <em>createHistoryStream</em> request:</p>
<aside class="kicker" style="align-self: start; position: relative; top: 58px;">
<p>JSON messages don’t have indentation or whitespace when sent over the wire.</p>
</aside>
<figure class="request-response">
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"name": ["createHistoryStream"],
"type": "source",
"args": [{"id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519"}]
}</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
</figure>
<div>
<p><em>createHistoryStream</em> is how Scuttlebutt peers ask each other for a list of messages posted by a particular feed. It has one argument that is a JSON dictionary specifying more options about the request. <em>id</em> is the only required option and says which feed you are interested in.</p>
<p>Because this is the first RPC request, the request number is 1. The next request made by this peer will be numbered 2. The other peer will also use request number 1 for their first request, but the peers can tell these apart because they know whether they sent or received each request.</p>
</div>
<p>Now the responder begins streaming back responses:</p>
<figure class="request-response">
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"key": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256",
"value": {
"previous": null,
"author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
"sequence": 1,
"timestamp": 1514517067954,
"hash": "sha256",
"content": {
"type": "post",
"text": "This is the first post!"
},
"signature": "QYOR/zU9dxE1aKBaxc3C0DJ4gRyZtlMfPLt+CGJcY73sv5abKK
Kxr1SqhOvnm8TY784VHE8kZHCD8RdzFl1tBA==.sig.ed25519"
},
"timestamp": 1514517067956
}</code></pre>
</div>
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"key": "%R7lJEkz27lNijPhYNDzYoPjM0Fp+bFWzwX0SmNJB/ZE=.sha256",
"value": {
"previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256",
"author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
"sequence": 2,
"timestamp": 1514517078157,
"hash": "sha256",
"content": {
"type": "post",
"text": "Second post!"
},
"signature": "z7W1ERg9UYZjNfE72ZwEuJF79khG+eOHWFp6iF+KLuSrw8Lqa6
IousK4cCn9T5qFa8E14GVek4cAMmMbjqDnAg==.sig.ed25519"
},
"timestamp": 1514517078160
}</code></pre>
</div>
</figure>
<aside style="align-self: start; position: relative; top: 29px;">
<p>Because the responses are part of a stream, their RPC headers have the stream flag set.</p>
<p>All responses use the same request number as the original request but negative.</p>
</aside>
<p>Each message posted by the feed is sent back in its own response. This feed only contains two messages.</p>
<p>To close the stream the responder sends an RPC message with both the stream and end/err flags set and a JSON body of <code>true</code>. When the requester sees that the stream is being closed they send a final message to close their own end of it (source type requests must always be closed by both ends).</p>
<figure class="request-response">
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">Yes</span>
</div>
<pre><code>true</code></pre>
</div>
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">Yes</span>
</div>
<pre><code>true</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
</figure>
<p>Alternatively, to abort a stream before it is finished the requester can send their closing message early, at which point the responder closes their own end.</p>
<figure class="request-response">
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">Yes</span>
</div>
<pre><code>true</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">Yes</span>
</div>
<pre><code>true</code></pre>
</div>
</figure>
<h4 id="async-example">Async example</h4>
<p>One of the few public async procedures is <em>blobs.has</em>, which peers use to ask each other whether they have a particular blob.</p>
<p>In this example the requester is asking the responder if they have blob <code>&WWw4tQJ6…</code>:</p>
<figure class="request-response">
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">2</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">No</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"name": ["blobs", "has"],
"type": "async",
"args": ["&WWw4tQJ6ZrM7o3gA8lOEAcO4zmyqXqb/3bmIKTLQepo=.sha256"]
}</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
</figure>
<p>The responder does in fact have this blob so they respond with <code>true</code>. Because this is an async procedure and not a stream, there is only one response and no need to close the stream afterwards:</p>
<figure class="request-response">
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-2</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">No</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>true</code></pre>
</div>
</figure>
<h4 id="error-example">Error example</h4>
<p>Let’s take the previous example and introduce a programming mistake to see how the RPC protocol handles errors:</p>
<figure class="request-response">
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">3</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">No</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"name": ["blobs", "has"],
"type": "async",
"args": ["<strong>this was a mistake</strong>"]
}</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
<img class="response-arrow" src="img/arrow.png" alt=""/>
<div class="response">
<div class="header">
<span class="key">Request number</span><span class="value">-3</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">No</span>
<span class="key">End/err</span><span class="value">Yes</span>
</div>
<pre><code>{
"name": "Error",
"message": "invalid hash:this was a mistake",
"stack": "…"
}</code></pre>
</div>
</figure>
<p>Most importantly, the response has the end/err flag set to indicate that an error occurred. The reference Scuttlebot implementation also includes an error message and a JavaScript stack trace.</p>
<p>For source type procedures an error will also end the stream because the end/err flag has the dual purpose of ending streams and indicating that an error occurred.</p>
<h2 id="feeds">Feeds</h2>
<p>A Scuttlebutt feed is a list of all the messages posted by a particular identity. When a user writes a message in a Scuttlebutt client and posts it, that message is put onto the end of their feed.</p>
<h3 id="structure">Structure</h3>
<p>The messages in a feed form an append-only log, meaning that once a message is posted it cannot be modified. Each message (except the first one) references the ID of the previous message, allowing a chain to be constructed back to the first message in the feed.</p>
<img src="img/sigchain.png" class="pagewidth" style="height: 855px;" alt=""/>
<h3 id="message-format">Message format</h3>
<p>To create a message to post in a feed, start by filling out these fields:</p>
<pre><code>{
"previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256",
"author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
"sequence": 2,
"timestamp": 1514517078157,
"hash": "sha256",
"content": {
"type": "post",
"text": "Second post!"
}
}
</code></pre>
<table class="defs">
<tr>
<td>previous</td>
<td>Message ID of the latest message posted in the feed. If this is the very first message then use <code>null</code>. See below for how to compute a message’s ID.</td>
</tr>
<tr>
<td>author</td>
<td>Public key of the feed that the message will be posted in.</td>
</tr>
<tr>
<td>sequence</td>
<td>1 for the first message in a feed, 2 for the second and so on.</td>
</tr>
<tr>
<td>timestamp</td>
<td>Time the message was created. Number of milliseconds since 1 January 1970 00:00 UTC.</td>
</tr>
<tr>
<td>hash</td>
<td>The fixed string <code>sha256</code>, which is the hash function used to compute the message ID.</td>
</tr>
<tr>
<td>content</td>
<td>If the message is not encrypted, This is a dictionary containing free-form data for applications to interpret, plus a mandatory <em>type</em> field. The <em>type</em> field allows applications to filter out message types they don’t understand and must be a Unicode string between 3 and 52 code units long (inclusive). If the message is encrypted, then this is a base64 encoded string, followed by a suffix of <code>.box</code>; we will describe private messages later in this document.</td>
</tr>
</table>
<aside style="align-self: start; position: relative; top: 19px;">
<p>Only these fields are allowed and they all must be present.</p>
<p>Fields must appear in this order (although <em>author</em> and <em>sequence</em> can be swapped for legacy reasons, but please don’t do this for new messages).</p>
<p>Fields within <em>content</em> can appear in any order but the order must be remembered for later.</p>
</aside>
<h4 id="signature">Signature</h4>
<div>
<p>All messages in a feed are signed by that feed’s long-term secret key. This enables recipients to verify that a message was really posted by a particular identity and not tampered with as it gets gossiped and replicated throughout the Scuttlebutt network.</p>
<p>Before signing a message it must be serialized according to a specific canonical JSON format. This means for any given message there is exactly one way to serialize it as a sequence of bytes, which is necessary for signature verification to work. The reference implementation verifies that all messages it receives are in the canonical format and rejects messages that aren’t.</p>
</div>
<aside class="impl">
<img class="icon" src="img/impl.png"/>
<h5>Implementations</h5>
<div class="lang">JS</div>
<div class="vs"><a href="https://github.com/ssbc/ssb-keys/blob/master/index.js">ssb-keys</a></div>
<div class="vs"><a href="https://github.com/ssbc/ssb-feed/blob/master/validator.js">ssb-feed/validator.js</a></div>
<div class="lang">Py</div>
<div class="vs"><a href="https://github.com/pferreir/pyssb/blob/master/ssb/feed/models.py">models.py</a></div>
</aside>
<p>The canonical format is defined by the <em>ECMA-262 6<sup>th</sup> Edition</em> section <em><a href="https://www.ecma-international.org/ecma-262/6.0/#sec-json.stringify">JSON.stringify</a></em>. For an example, see how the above message is formatted.</p>
<p>In brief, the rules are:</p>
<ul>
<li>Two spaces for indentation.</li>
<li>Dictionary entries and list elements each on their own line.</li>
<li>Newlines use the line feed character <code>\n</code>.</li>
<li>No trailing newline.</li>
<li>One space after the colon <code>:</code> for dictionary keys.</li>
<li>Empty dictionaries appear as <code>{}</code> and empty lists appear as <code>[]</code>.</li>
<li>Strings and numbers formatted according to the sections <em><a href="https://www.ecma-international.org/ecma-262/6.0/#sec-quotejsonstring">QuoteJSONString</a></em> and <em><a href="https://www.ecma-international.org/ecma-262/6.0/#sec-tostring-applied-to-the-number-type">ToString Applied to the Number Type</a></em>.
</ul>
<p>Then sign the message by computing:</p>
<pre><code>signature = nacl_sign_detached(
msg: formatted_json_message,
key: authors_longterm_sk
)</code></pre>
<p>Base64 encode the signature and put <code>.sig.ed25519</code> on the end. Finally, add the signature to the message itself. It must be the last entry in the dictionary:</p>
<pre><code>{
"previous": "%XphMUkWQtomKjXQvFGfsGYpt69sgEY7Y4Vou9cEuJho=.sha256",
"author": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
"sequence": 2,
"timestamp": 1514517078157,
"hash": "sha256",
"content": {
"type": "post",
"text": "Second post!"
},
"<strong>signature</strong>": "z7W1ERg9UYZjNfE72ZwEuJF79khG+eOHWFp6iF+KLuSrw8Lqa6
IousK4cCn9T5qFa8E14GVek4cAMmMbjqDnAg==.sig.ed25519"
}</code></pre>
<p>To verify the signature, first remove the signature field from the message. Make sure the remaining message is formatted as JSON using the same formatting rules. Here the order of dictionary entries within <em>content</em> matters; they must be in the same order that you received them.</p>
<p>Then remove the <code>.sig.ed25519</code> suffix from the signature, base64 decode it and verify with:</p>
<pre><code>nacl_sign_verify_detached(
sig: signature,
msg: formatted_json_message,
key: authors_longterm_pk
)</code></pre>
<h4 id="message-id">Message ID</h4>
<p>A message ID is a hash of the message including signature. Messages can refer to other messages by their ID.</p>
<p>To compute a message’s ID, first format it as JSON using the formatting rules. Like with signatures, dictionary keys must appear in the same order that you received them. The <em>signature</em> key must appear last as shown above (but without wrapping the line).</p>
<p>The ID of the above message is:</p>
<img src="img/format_message_id.png" style="height: 80px;" alt="%R7lJEkz27lNijPhYNDzYoPjM0Fp+bFWzwX0SmNJB/ZE=.sha256 where everything between % and .sha256 is the base64-encoding of the sha256 hash of the formatted message including signature" />
<p>Currently all IDs are computed using sha256. However the protocol includes the name of the hash function in message IDs and the messages themselves to allow migrating to a different one if needed in the future.</p>
<h3 id="createHistoryStream">createHistoryStream</h3>
<p>Scuttlebutt clients maintain a set of feeds that they are interested in. These could be feeds that the user has followed, bookmarked or subscribed to. When peers connect, one of the first things they do is ask each other whether the feeds they are interested in have any new messages.</p>
<aside class="impl">
<img class="icon" src="img/impl.png" alt=""/>
<h5>Implementations</h5>
<div class="lang">JS</div>
<div class="vs"><a href="https://github.com/ssbc/ssb-db/blob/master/indexes/clock.js">clock.js</a></div>
</aside>
<p>The RPC procedure <em>createHistoryStream</em> is how peers ask each other for a list of messages in a particular feed.</p>
<figure class="request-response">
<div class="request">
<div class="header">
<span class="key">Request number</span><span class="value">1</span>
<span class="key">Body type</span><span class="value">JSON</span>
<span class="key">Stream</span><span class="value">Yes</span>
<span class="key">End/err</span><span class="value">No</span>
</div>
<pre><code>{
"name": ["createHistoryStream"],
"type": "source",
"args": [
{
"id": "@FCX/tsDLpubCPKKfIrw4gc+SQkHcaD17s7GI6i/ziWY=.ed25519",
"sequence": 2,
"limit": 1,
"live": false,
"old": true,
"keys": true
}
]
}</code></pre>
</div>
<img class="request-arrow" src="img/arrow.png" alt=""/>
</figure>
<p><em>createHistoryStream</em> takes one argument which is a dictionary of additional options that controls how the feed is returned. Valid options are:</p>
<table class="defs">
<tr>
<td>id</td>
<td>Public key of the feed to return messages from. Required.</td>
</tr>
<tr>
<td>sequence</td>
<td>Only return messages later than this sequence number. If not specified then start from the very beginning of the feed.</td>
</tr>
<tr>
<td>limit</td>
<td>Maximum number of messages to return. If the limit is exceeded only the earliest messages are returned. Default: unlimited.</td>
</tr>
<tr>
<td>live</td>
<td>If true, keep the stream alive and send new messages as they are posted. If false, end the stream after messages are sent and don’t wait for new ones. Default: false.</td>
</tr>
<tr>
<td>old</td>
<td>Used together with <em>live</em>. If true, start by sending existing messages already posted by this feed. If false, don’t send existing messages posted by this feed, only send new messages as they arrive. Default: true.</td>
</tr>
<tr>
<td>keys</td>
<td>If true, also include message IDs and timestamps of when each message was received by this peer. If false, just send the messages themselves. Default: true.</td>
</tr>
</table>
<aside style="align-self: start; position: relative; top: 60px;">
<p>Clients must use the name <em>sequence</em> when producing requests. However for legacy reasons both <em>sequence</em> and <em>seq</em> should be accepted as the name of this argument when receiving requests. If that behaviour is supported then receiving both <em>sequence</em> and <em>seq</em> arguments with conflicting values must be considered an error.</p>
</aside>
<p>Here is a comparison of <em>createHistoryStream</em> responses with <em>keys</em> set to true and false:</p>