first-cut pub-sub proposal

docs/proposed/pub-sub.rst

@@ -0,0 +1,124 @@
+.. -*- coding: utf-8 -*-
+
+Pub/Sub for Tahoe-LAFS
+======================
+
+There are various use-cases in Tahoe-LAFS where it would be
+advantagous to send messages to particular clients (or for particular
+topics). A well-known pattern called "Publish/Subscribe" or "PubSub"
+would answer many of these use-cases, but Tahoe-LAFS does not include
+such messaging.
+
+
+Use Cases
+---------
+
+Many of these use-cases are special-cases of a general want:
+
+ - there exists a mutable capability;
+ - one client updates content in this capability;
+ - ..and one or more (different) clients wishes to know when it changes
+
+Thus, a "pub/sub" pattern could be implemented by adding a mechanism
+to notify clients that a particular mutable (to which they possess a
+valid read-capability) has changed. This would maintain many of the
+existing properties of capabilities.
+
+
+Magic Folders Updates
+`````````````````````
+
+Magic Folders in Tahoe currently poll, looking for updates. This
+involves periodically downloading some mutable directory-capabilities
+and determining if anything has changed. This is wasteful of bandwidth
+and server resources. It would nice if a client, Alice, could tell
+other participants in the magic-folder that she's added some
+content. More generally, it would be useful for a client to be able to
+ask a storage-server to tell it whenever a particular (mutable)
+capability has changed. (For the Magic Folders use-case, this would
+mean Alice simply makes changes as now, and relies on the server to
+update interested clients).
+
+
+Grid -> Client Communication
+````````````````````````````
+
+Allowing the operator(s) of a Grid to communicate with clients is
+often desired. For example, notifying users of upcoming maintenance,
+new terms or general news. There is currently no mechanism to do this.
+
+An obvious way to implement this right now would be with a mechanism
+like Magic Folders uses: a mutable capability containing news which
+clients poll (and which the operator(s) can update). This would suffer
+the same polling problems, however.
+
+
+Proposed Solution
+=================
+
+Anything wanting a "broadcast" or "pub-sub" like pattern can implement
+it using the following mechanism:
+
+ - the publisher creates a mutable directory, yielding a write-cap, "W"
+ - anyone wishing to subscribe is given a read-cap, "R"
+   - anyone possessing a valid read-cap "R" (including the publisher)
+     can obviously "invite" further clients
+   - nobody else may subscribe
+ - to add information, the publisher adds a new file to the mutable
+   directory
+ - old information can be removed (or not) as the publisher wishes
+
+Without further changes to Tahoe-LAFS, this can already be achieved:
+clients can "poll" the read-capability and determine if anything new
+exists.
+
+
+Improvements Over Polling
+-------------------------
+
+The above mechanism would scale somewhat poorly as the number of
+"broadcast" capabilities increased. It would be better if the
+storage-server could *tell* clients when a mutable capability has
+changed.
+
+Storage servers implement a new feature, based on WebSockets. See also
+:ref:`http-storage-node-protocol` with which this protocol aims to be
+compatible. A single WebSocket endpoint exists:
+
+ - <server>/v1/mutable_updates
+
+After connecting to this endpoint, a client may send any number of
+messages (encoded using JSON) asking for updates to mutable
+capabilities:
+
+    {
+        "version": 1,
+        "subscribe": [
+            "URI:DIR2:a...",
+            "URI:DIR2:b..."
+        }
+    }
+
+The "subscribe" key points to a list of 1 or more capabilities. The
+server replies with a message like:
+
+    {
+        "version": 1,
+        "status": {
+            "URI:DIR2-RO:a...": true,
+            "URI:DIR2-RO:b...": "some kind of error message"
+        }
+    }
+
+That is, a "status" dict containing each capability with either "true"
+if the subscribe was successful or a string with an error-message if
+it was not possible to subscribe (e.g. the capability wasn't a mutable
+one, or wasn't found on this server).
+
+After that, the server will send a single message for each update:
+
+    {
+        "version": 1,
+        "update": "URI:DIR2-RO:a..."
+    }
+