Widgets, the book

.buildkite/pipeline.yaml

@@ -56,6 +56,7 @@ steps:
       - wget https://github.com/rust-lang/mdBook/releases/download/v0.4.1/mdbook-v0.4.1-x86_64-unknown-linux-gnu.tar.gz
       - tar -xf mdbook-v0.4.1-x86_64-unknown-linux-gnu.tar.gz
       - ./mdbook build server -d /workdir/implementation-guides/implementation-guides/server
+      - ./mdbook build widgets -d /workdir/implementation-guides/implementation-guides/widgets
       - tar -czf implementation-guides.tar.gz implementation-guides
     artifact_paths:
       - implementation-guides/implementation-guides.tar.gz

implementation-guides/widgets/book.toml

@@ -0,0 +1,14 @@
+[book]
+authors = ["The Matrix.org Foundation C.I.C."]
+language = "en"
+multilingual = false
+src = "src"
+title = "Matrix Widget Implementors Guide"
+
+[output.html]
+theme = "../theme"
+git-repository-url = "https://github.com/matrix-org/matrix.org/tree/master/implementation-guides/widgets"
+
+[preprocessor.links]
+
+[preprocessor.index]

implementation-guides/widgets/src/SUMMARY.md

@@ -0,0 +1,14 @@
+# Summary
+
+[Introduction](./intro.md)
+
+- [Widget basics](./basics/readme.md)
+    - [URL templating](./basics/url-templating.md)
+- [Communicating with clients](./communication/readme.md)
+    - [Requests/Responses](./communication/requests-responses.md)
+    - [Error handling](./communication/errors.md)
+    - [Capabilities](./communication/capabilities.md)
+- [A simple stickerpicker](./example-stickerpicker/readme.md)
+    - [Defining our stickers](./example-stickerpicker/send-behaviour.md)
+    - [Communicating with the client](./example-stickerpicker/communication.md)
+    - [Using the stickerpicker in Element](./example-stickerpicker/usage-element-web.md)

implementation-guides/widgets/src/basics/readme.md

@@ -0,0 +1,77 @@
+# Widget basics
+
+Widgets exist in two places currently: rooms and on a user's account. Room widgets are accessible
+to anyone who can see the room while account widgets are only accessible to that user.
+
+Both forms of widget have the same general structure:
+
+```json
+{
+    "id": "20200827_WidgetExample",
+    "type": "m.custom",
+    "name": "My Cool Widget",
+    "url": "https://example.org/my/widget.html?roomId=$matrix_room_id",
+    "creatorUserId": "@alice:example.org",
+    "data": {
+        "custom-key": "This is a custom key",
+        "title": "This is a witty description for the widget"
+    }
+}
+```
+
+The `id` is the widget's ID, which must be unique to the room/account where the widget will be
+located. The `type` is almost always going to be `m.custom` to indicate it is a generic widget,
+though other types are available. The `name` is simply what the widget should be called, and the
+`url` is where the widget is located.
+
+The `creatorUserId` is the user ID of who added the widget. For account widgets this should be
+the user's own ID, though for rooms it should be whoever originally added the widget. Room widgets
+can be edited over time by other members of the room, so this indicates who was responsible for
+the widget's construction rather than who edited it last.
+
+`data` has a special meaning for the `url` in that the keys of the the object can be used as variables
+to the `url`. This is most useful when using a custom `type` so variables can be provided to the
+client when they are using purpose-built UI. An optional `title` can be specified in the `data`
+to give a short summary of what the widget is representing alongside the `name`.
+
+## Room widgets
+
+Widgets at the room level are stored as state events in the room with an event type of `m.widget`
+and a state key matching the widget's `id`. The state event's content is the same as the object
+described above.
+
+State events that are missing a `url` or `type` in the event content will not be rendered by clients.
+
+## Account widgets
+
+Widgets at the user/account level are stored in that user's account data under a single `m.widgets`
+type. This type has keys which are each widget's `id` and a value consisting of a minimal room widget.
+
+For example:
+
+```json
+{
+    "20200827_WidgetExample": {
+        "content": {
+            "id": "20200827_WidgetExample",
+            "type": "m.custom",
+            "name": "My Cool Widget",
+            "url": "https://example.org/my/widget.html?roomId=$matrix_room_id",
+            "creatorUserId": "@alice:example.org",
+            "data": {
+                "custom-key": "This is a custom key",
+                "title": "This is a witty description for the widget"
+            }
+        },
+        "sender": "@alice:example.org",
+        "state_key": "20200827_WidgetExample",
+        "type": "m.widget"
+    }
+}
+```
+
+This looks a bit confusing, though the idea is that clients can use this similarity to render
+widgets mroe easily.
+
+To remove a widget from the user's account, simply remove all references to it from the `m.widgets`
+object.

implementation-guides/widgets/src/basics/url-templating.md

@@ -0,0 +1,14 @@
+# URL templating
+
+To render a widget by URL, clients will first replace any keys from `data` with their associated
+values in the `url`. For example, if the `data` object was
+`{"custom-key": 1234, "another-key": "hello!"}` then the client would replace `$custom-key` with
+`1234` and `$another-key` with `hello!`, wherever those appear in the URL.
+
+Some additional variables are defined by the specification for use in the widget URL:
+
+* `$matrix_user_id` - The current user's ID.
+* `$matrix_room_id` - The room ID the user is currently viewing, or an empty string if none
+  applicable.
+* `$matrix_display_name` - The current user's display name, or user ID if not set.
+* `$matrix_avatar_url` - An HTTP URL to the current user's avatar, or an empty string if not set.

implementation-guides/widgets/src/communication/capabilities.md

@@ -0,0 +1,8 @@
+# Capabilities
+
+As part of establishing a session for widgets and clients to communicate with each other, a
+capabilities negotation happens where the client asks the widget what permissions it wants and
+the widget replies with its ideal set. The client is then supposed to ask the user what permissions
+to grant, or if it's obvious for certain widgets, approve them implicitly.
+
+The widget specification has a list of available capabilities.

implementation-guides/widgets/src/communication/errors.md

@@ -0,0 +1,23 @@
+# Error handling
+
+All requests made by clients/widgets over the Widget API should have a timeout of about 10 seconds.
+If after 10 seconds no response was received, the application should either try again or give up.
+
+Errors related to executing a request should be sent as a `response` looking like the following:
+
+```json
+{
+    "api": "fromWidget",
+    "requestId": "generated-id-1234",
+    "widgetId": "20200827_WidgetExample",
+    "action": "com.example.say_hello",
+    "data": {
+        "request-param": "value"
+    },
+    "response": {
+        "error": {
+            "message": "Failed to process request: Server returned 500 error"
+        }
+    }
+}
+```

implementation-guides/widgets/src/communication/readme.md

@@ -0,0 +1,9 @@
+# Communicating with clients
+
+Widgets can communicate with their host client over a `postMessage` API known as the Widget API.
+This API provides the widget with some abilities, such as being able to send stickers or ask
+to be left on screen, and allows the client to more directly integrate with the widget.
+
+The API is split into two parts: the `toWidget` API and `fromWidget` API, where the difference
+is where a request over the API comes from. The communication channel in which widgets and clients
+speak to each other is known as a "session".

implementation-guides/widgets/src/communication/requests-responses.md

@@ -0,0 +1,30 @@
+# Requests/Responses
+
+Requests are structured objects sent over `postMessage` to the other side, where they then
+respond with responses. All requests must have a response, or time out.
+
+An example request is:
+
+```json
+{
+    "api": "fromWidget",
+    "widgetId": "20200827_WidgetExample",
+    "requestId": "generated-id-1234",
+    "action": "com.example.say_hello",
+    "data": {
+        "request-param": "value"
+    }
+}
+```
+
+The `api` is either `fromWidget` or `toWidget` depending on where the request is originating from.
+If it's from the widget, `fromWidget`. If it's from the client, `toWidget`.
+
+The `widgetId` is the widget's ID and should be used to ensure requests are coming from a valid
+source. The `requestId` is generated and should be unique within the session.
+
+The `action` is what describes the request, and what `data` attributes to include. A full description
+of all actions available can be found in the widget specification.
+
+Responses are simply a copy of the request object with an added `response` field, where the attributes
+are defined by the `action` being executed.

implementation-guides/widgets/src/example-stickerpicker/communication.md

@@ -0,0 +1,89 @@
+# Communicating with the client
+
+To recap, at this point we have a stickerpicker which tries to send stickers to the client
+but instead just encounters errors. Most importantly, we need to know our `widgetId` so we can
+send a valid request and we need to ask for permission to send stickers.
+
+Let's set up some basic request handling and validation at the top of our `stickerpicker.js`
+file:
+
+```javascript
+let widgetId = null; // to be populated on the first `toWidget` request.
+
+// First we need to set up a listener to ensure we're able to hear the client's requests
+window.onmessage = function(event) {
+    // First make sure we are roughly in shape to be a widget: we need a parent window to
+    // make sure it's not another tab trying to contact us.
+    if (!window.parent) return;
+
+    // Next we validate to make sure the request is a valid shape
+    const request = event.data;
+    if (!request) return;
+    if (!request['requestId'] || !request['widgetId'] || !request['action']) return;
+    if (request['api'] !== "toWidget") return;
+
+    // Now we see if it is for us, if we know what our widget ID is
+    if (widgetId) {
+        if (widgetId !== request['widgetId']) return;
+    } else {
+        widgetId = request['widgetId'];
+    }
+
+    // We'll finish this function in a moment.
+};
+```
+
+We define a variable outside our function so we can use it wherever we want, which in our case will
+be the `sendSticker` function where we currently have `widgetId: null`. Change that to
+`widgetId: widgetId` so we can make sure we have a valid request.
+
+A lot of the `onmessage` event handler function is just making sure that the client is sending a
+valid request. We don't want the user's browser extensions to interfere with our stickerpicker, so
+we try and filter them out. We also filter out any requests that are not sent over the `toWidget`
+API because we're not supposed to receive any other kind of request as a widget.
+
+Once we know we have a valid request, we'll capture the `widgetId` sent by the client or, if we
+already know the widget ID, we'll make sure the client is talking to us.
+
+What we're looking to handle with this function is a capabilities request from the client so we can
+claim our required permissions. If we set up Element Web correctly later in this guide, it will
+auto-accept the permissions we need as long as we're not trying to ask for too much. This means
+we need to watch out for an `action` of `capabilities` and reply to it, like the following. We'll
+also respond with an error to all other actions because we aren't worried about handling them in
+this proof of concept.
+
+```javascript
+// Finally, we can get on to the action handling.
+if (request['action'] === 'capabilities') {
+    // We're going to respond with the capabilities we want: m.sticker
+    window.parent.postMessage({
+        ...request, // include the original request
+        response: {
+            capabilities: ['m.sticker'],
+        },
+    }, event.origin);
+} else {
+    // We'll send an error response for this. Ideally we'd do a full implementation
+    // of the widget API, but that is out of scope for this tutorial.
+    window.parent.postMessage({
+        ...request, // include the original request
+        response: {
+            error: {
+                message: "Action not supported",
+            },
+        },
+    }, event.origin);
+}
+```
+
+Both for the `capabilities` response and the error response we have `...request` which just includes
+the original request object in the response for us. This ensures that we don't miss any detail when
+replying, and the client will know what we're replying to.
+
+The `response` object for the `capabilities` request is simply the capabilities we want. We only
+want to send stickers, so that's all we'll ask for.
+
+**Note**: This is where we diverge from the spec: we're supposed to handle a lot more request actions,
+but that would make this guide complicated so we've excluded them here.
+
+We should now have enough to try out our widget, but we need to add it to our account first.

implementation-guides/widgets/src/example-stickerpicker/readme.md

@@ -0,0 +1,81 @@
+# A simple stickerpicker
+
+This guide covers writing your very own simple stickerpicker to explore some of the widget API
+and gain a better understanding of the protocol. The end result for this is available on
+[GitHub](https://github.com/matrix-org/simple-stickerpicker-widget) for ease of reference.
+
+If all goes according to plan, your widget will look like this in Element Web/Desktop:
+
+![stickerpicker](https://raw.githubusercontent.com/matrix-org/simple-stickerpicker-widget/master/stickerpicker.png)
+
+**Note**: The widget that is created this way is not fully compliant with the spec. It is for
+demonstration and educational purposes only.
+
+## The HTML
+
+Luckily for this kind of widget the amount of HTML is relatively small. We're using hardcoded
+stickers here, however this is just meant to teach the basics.
+
+To get started, create a `stickerpicker.html` with the following HTML in it:
+
+```html
+<!DOCTYPE html>
+<html lang="en">
+
+<head>
+    <meta charset="utf-8">
+    <title>Example Stickerpicker</title>
+    <!-- Make the sticker picker look roughly like the matrix.org site -->
+    <link rel="stylesheet" href="https://rsms.me/inter/inter.css" />
+    <link rel="stylesheet" href="stickerpicker.css" />
+</head>
+
+<body>
+    <h1>Sample Stickers</h1>
+    <p>Click a sticker to send it.</p>
+    <div class="sticker" onclick="sendSticker('normal')">
+        <img src="https://matrix-client.matrix.org/_matrix/media/r0/thumbnail/matrix.org/AFdISYOGCRXUIJejgxeRxaEg?width=256&height=256&method=crop" />
+    </div>
+    <div class="sticker" onclick="sendSticker('inverted')">
+        <img src="https://matrix-client.matrix.org/_matrix/media/r0/thumbnail/matrix.org/wKGtfcEVxrUFXbbipBzfvfpD?width=256&height=256&method=crop" />
+    </div>
+    <script src="stickerpicker.js"></script>
+</body>
+
+</html>
+```
+
+*Full source: https://github.com/matrix-org/simple-stickerpicker-widget/blob/master/stickerpicker.html*
+
+The CSS is also minimal and is just there to make it look relatively okay compared to the default
+browser styling. Put this in a file named `stickerpicker.css` next to your `stickerpicker.html`:
+
+```css
+* {
+    font-family: Inter, Arial, Helvetica, sans-serif;
+    color: #333333;
+    background-color: #ffffff;
+}
+
+.sticker {
+    display: inline-block;
+    margin-right: 8px;
+    padding: 8px;
+    border-radius: 4px;
+    background-color: #f4f4f4;
+    cursor: pointer;
+    position: relative;
+}
+
+.sticker:hover {
+    border-bottom: 5px solid #0098d4;
+}
+
+h1 {
+    font-size: 1.5em;
+}
+```
+
+*Full source: https://github.com/matrix-org/simple-stickerpicker-widget/blob/master/stickerpicker.css*
+
+All we have to do now is create and populate `stickerpicker.js` so the buttons work!