Browse files

First stab at metadata implementation

Hooks up meta operations throughout the code base. The only thing
implemented is "shout" which multicasts an object to all clients.
Some changes were made versus the original metadat proposal, current
proposal is documented in

- Does not coalesce ops and meta ops before sending.
- No tests implemented.
1 parent 121d908 commit 59e438139cd3e0417dc9c9577f671f91575ca45b @wmertens wmertens committed Jan 24, 2012
Showing with 139 additions and 12 deletions.
  1. +88 −0
  2. +1 −1 src/client/
  3. +15 −4 src/client/
  4. +8 −2 src/server/
  5. +23 −3 src/server/
  6. +4 −2 src/server/
@@ -0,0 +1,88 @@
+# Document Metadata Design Proposal
+> This feature isn't fully implemented yet
+I'm planning on adding metadata support to documents. The idea is to add a sidechannel for data like creation time, cursor positions, connected users, etc.
+## New document interface
+A document will consist of:
+- **Version**: A version number counting from 0
+- **Snapshot**: The current document contents
+- **Meta**: *(NEW, not yet implemented)* An object containing misc data about the document:
+ - Arbitrary user data, set when an object is created. Editable by the server only.
+ - **Creation time**
+ - **Last modified time**: This is updated automatically on each client each time it sees a document operation
+ - **Session data**: An object with an entry for every client that is currently connected. Map clientIds to:
+ - **Username** (optional)
+ - **Cursor position** (type dependant)
+ - **Connection time** maybe?
+ - Any other user data. This can be filled in by the auth function when a client connects. (And maybe clients should be able to edit this as well?)
+Unlike the document data, metadata changes will not be persisted. Metadata changes will not bump the document's version number. (-wm: I assume that some things like the time stamps will be persisted? And what about when client IDs are stored with ops, shouldn't the user names be persisted?)
+Initial document metadata can be set at document creation time.
+Some metadata fields like last modified time and cursor positions will be updated automatically on all clients whenever an operation is submitted. (-wm: this will probably mean storing the time delta between the server and client)
+## Metadata operations
+> This is implemented, but the only path handled now is 'shout'
+We also add a new kind of operation, a **meta operation**. I've thought about using the JSON OT type for this, but it means that if someone wants to implement the sharejs wire protocol, they have to implement JSON OT (which is really complicated). So I'm going to keep it simple.
+Metadata operations express changes in the metadata object. They look like this:
+- NOT **Version**: Metadata changes should be independent of current document version number
+- **Path**: List of object keys to reach the target of the change. All path elements except the last must already exist.
+- **New value**: *(optional)* JSON object which replaces whatever was at the metadata object before. If this is missing, the object is removed.
+Some paths are special:
+- ``['shout', ...]``: Broadcasts the value to all clients, doesn't keep it in memory. Full path is ignored.
+- ``['ctime']``, ``['mtime']``, ``['sessions']``: Read only, see above
+Metadata gets consistency guarantees by restricting who is allowed to submit metadata changes.
+The server can send metadata operations to clients:
+- [``shout``]: ``value`` is the value being broadcast, ``by`` is the clientid that shouted (not implemented). This results in an event emitted by Doc: ``('shout', value)``.
+ > (Right now the full path used is given to the client, perhaps this could be used to send specific Doc events instead of just "shout", like ``('shout_foo', value)`` for ``path: ['shout', 'foo']``?)
+The model emits ``('applyMetaOp', path, value)`` for all successful meta operations
+## Transforming cursor positions
+> not implemented
+Types can also specify cursor transforms. This is important to make cursors move as you edit content surrounding them.
+TYPE.transformCursor = (position, operation) -> newPosition
+Clients are responsible for updating cursor positions in two scenarios:
+- When they generate operations locally they transform everyone's cursor position by the operation
+- When they receive updated cursor positions from the server against an old version
+The server will pre-transform cursor positions before rebroadcasting them.
+## Expected usage
+- **A new client connects** - the server will add an entry corresponding to the client in the session data
+- **A user moves their cursor** - they send a metadata op which is broadcast via the server to indicate their new cursor position. (Note that cursor positions may be more complicated than just a number. Imagine a user exploring a spreadsheet...)
+- **The client sees a new document operation** It updates the last modified time of the session data using its local clock. It doesn't tell anyone else - they'll each have each made the same change locally as well.
+- - -
+#### Still to figure out
+- How does the client learn its own ID?
+ - It could get another special metadata field when it opens a document
+ - It could be told its ID when it gets its first message from the server, or when it opens a document
+- Are clients allowed to make arbitrary changes to the document's metadata?
+- What is the client API for querying cursor positions and getting notified of metadata changes?
@@ -112,7 +112,7 @@ class Connection
else if == false then 'close'
else if msg.snapshot != undefined then 'snapshot'
else if msg.create then 'create'
- else if msg.op then 'op'
+ else if msg.op or msg.meta then 'op'
else if msg.v != undefined then 'op response'
callbacks = @handlers[docName]?[type]
@@ -105,18 +105,25 @@ Doc = (connection, @name, @version, @type, @snapshot) ->
# Internal API
# The connection uses this method to notify a document that an op is received from the server.
@_onOpReceived = (msg) ->
- # msg is {doc:, op:, v:}
+ # msg is {doc:, op:, v:, meta:}
# There is a bug in (produced on firefox 3.6) which causes messages
# to be duplicated sometimes.
- # We'll just silently drop subsequent messages.
- return if msg.v < @version
+ # We'll just silently drop subsequent messages. This is not possible for metaOps
+ return if msg.op and msg.v < @version
throw new Error("Expected docName '#{@name}' but got #{msg.doc}") unless msg.doc == @name
- throw new Error("Expected version #{@version} but got #{msg.v}") unless msg.v == @version
+ throw new Error("Expected version #{@version} but got #{msg.v}") unless msg.v == @version or msg.meta
# p "if: #{i @inflightOp} pending: #{i @pendingOp} doc '#{@snapshot}' op: #{i msg.op}"
+ if msg.meta
+ {path, value} = msg.meta
+ if path?[0] == 'shout'
+ @emit 'shout', value
+ return
op = msg.op
serverOps[@version] = op
@@ -151,6 +158,10 @@ Doc = (connection, @name, @version, @type, @snapshot) ->
# together and sent together.
setTimeout @flush, 0
+ @shout = (msg) =>
+ # Meta ops don't have to queue, they can go direct. Good/bad idea?
+ connection.send {'doc':@name, 'meta': { path: ['shout'], value: msg } }
# Close a document.
# No unit tests for this so far.
@close = (callback) ->
@@ -44,6 +44,7 @@ module.exports = (model, options) ->
when 'create' then 'create'
when 'get snapshot', 'get ops', 'open' then 'read'
when 'submit op' then 'update'
+ when 'submit meta' then 'update'
when 'delete' then 'delete'
else throw new Error "Invalid action name #{name}"
@@ -82,8 +83,13 @@ module.exports = (model, options) ->
opData.meta ||= {}
opData.meta.source = @id
- @doAuth {docName, op:opData.op, v:opData.v, meta:opData.meta}, 'submit op', callback, =>
- model.applyOp docName, opData, callback
+ # If ops and meta get coalesced, they should be separated here.
+ if opData.op
+ @doAuth {docName, op:opData.op, v:opData.v, meta:opData.meta}, 'submit op', callback, =>
+ model.applyOp docName, opData, callback
+ else
+ @doAuth {docName, meta:opData.meta}, 'submit meta', callback, =>
+ model.applyMetaOp docName, opData, callback
# Delete the named operation.
# Callback is passed (deleted?, error message)
@@ -8,6 +8,8 @@
queue = require './syncqueue'
types = require '../types'
+isArray = (o) -> == '[object Array]'
# This constructor creates a new Model object. There will be one model object
# per server context.
@@ -487,10 +489,28 @@ module.exports = Model = (db, options) ->
refreshReapingTimeout docName
callback? error, newVersion
- # Not yet implemented.
+ # TODO: store (some) metadata in DB
+ # TODO: op and meta should be combineable in the op that gets sent
@applyMetaOp = (docName, metaOpData, callback) ->
- {v, op} = metaOpData
- throw new Error 'Not implemented'
+ {path, value} = metaOpData.meta
+ if isArray path
+ load docName, (error, doc) ->
+ if error?
+ callback? error
+ else
+ applied = false
+ switch path[0]
+ when 'shout'
+ doc.eventEmitter.emit 'op', metaOpData
+ applied = true
+ model.emit 'applyMetaOp', docName, path, value if applied
+ callback error, doc.v
+ else
+ callback? new Error "path should be an array"
# Listen to all ops from the specified version. If version is in the past, all
# ops since that version are sent immediately to the listener.
@@ -72,6 +72,7 @@ exports.attach = (server, createClient, options) ->
callback 'Doc already opened' if docState[docName].listener?
p "Registering listener on #{docName} by #{} at #{version}"
+ # This passes op events to the client
docState[docName].listener = listener = (opData) ->
throw new Error 'Consistency violation - doc listener invalid' unless docState[docName].listener == listener
@@ -244,7 +245,7 @@ exports.attach = (server, createClient, options) ->
# We received an op from the socket
handleOp = (query, callback) ->
throw new Error 'No docName specified' unless query.doc?
- throw new Error 'No version specified' unless query.v?
+ throw new Error 'No version specified' unless query.v? or (query.meta?.path? and query.meta?.value?)
op_data = {v:query.v, op:query.op}
op_data.meta = query.meta || {}
@@ -257,6 +258,7 @@ exports.attach = (server, createClient, options) ->
{doc:query.doc, v:appliedVersion}
+ p "sending #{i msg}"
send msg
@@ -280,7 +282,7 @@ exports.attach = (server, createClient, options) ->
# request. They're all handled together.
handleOpenCreateSnapshot query, callback
- else if query.op? # The socket is applying an op.
+ else if query.op? or query.meta? # The socket is applying an op.
handleOp query, callback

0 comments on commit 59e4381

Please sign in to comment.