Skip to content
This repository

Adding Etherpad/Etherpad-lite document type to ShareJS #112

Closed
wants to merge 6 commits into from

4 participants

Constantin Jucovschi wmertens Joseph Gentle Jeremy Apthorp
Constantin Jucovschi

Adding Etherpad-lite document type which allows text+attribute OT. The added example does not yet make use of attributes though...

wmertens

There's a lof of JS-isms in this code. For example, this could be written instead as:

  @connection.send
    doc: @name
    meta: state

Could you fix these to adhere to the style of the project?

wmertens

Likewise:

etherpad.serialize = (snapshot) ->
  text: snapshot.text
  attribs: snapshot.attribs
  pool: snapshot.pool.toJsonable()
wmertens

Not sure about this. Obviously it's a lot of work to convert this to coffeescript. Maybe this should be in a lib dir somewhere? It's not really code you'll change right?

No I will not change any of these libraries and so I'd rather put them in some lib directory. In the end there should be a nice way of using legacy JS code. BTW: I really do not like how I solved the problem of getting the Changeset library available in the browser (the window.ShareJS.Changeset). Are you ok with it?

Collaborator

@josephg your call :-)

Owner

... I don't mind, so long as someone's happy to maintain this code. My biggest concern is the lack of tests for any of this code.

Collaborator

Ok so how about putting the etherpad js under src/lib-etherpad and having just the coffeescript in src/types?

wmertens
Collaborator

@josephg how do you feel about the javascript ported from the etherpad project? Should it be translated to coffee? Should it be moved elsewhere?

wmertens wmertens referenced this pull request August 07, 2012
Open

Rich text support #1

src/types/etherpad-api.coffee
... ...
@@ -0,0 +1,117 @@
  1
+# Text document API for text
  2
+# :tabSize=4:indentSize=4:
1
Joseph Gentle Owner
josephg added a note August 14, 2012

ShareJS uses 2 spaces, not tabs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/types/etherpad.coffee
((106 lines not shown))
  106
+ *      http://www.apache.org/licenses/LICENSE-2.0
  107
+ *
  108
+ * Unless required by applicable law or agreed to in writing, software
  109
+ * distributed under the License is distributed on an "AS-IS" BASIS,
  110
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  111
+ * See the License for the specific language governing permissions and
  112
+ * limitations under the License.
  113
+ */
  114
+
  115
+var _opt = null;
  116
+
  117
+/**
  118
+ * ==================== General Util Functions =======================
  119
+ */
  120
+
  121
+Changeset = {};
1
Joseph Gentle Owner
josephg added a note August 14, 2012
var Changeset = {};
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Joseph Gentle
Owner

Great work on getting this working, @jucovschi.

Nitpicks aside:

  • It doesn't have any tests. Even just hooking up the random op generator would make me feel a lot better about this. Take a look at test/types/text.coffee.
  • This is 2.5k lines, which increases the number of lines of code in sharejs by about 50%. The JSON OT code (which is probably the most complicated thing in sharejs) is only 441 lines. The web client (compiled to unminified JS) fits in about 1000 lines.

My instinct is that it shouldn't live in sharejs, but I have no idea where it should live. I don't want to maintain 2.5k lines of someone else's javascript. But also, I want rich text. @wmertens @nornagon - thoughts? What do?

wmertens
Collaborator

I think it's not too bad to have this added code because it's separated from the other types and the added code is really only touched by the etherpad type.

Perhaps it would be good to have a mechanism in place to only require types that the server is going to allow. That way, the server only pays the large JS codebase price when it's being used. The client has to include it separately anyway so that's fine. As an added bonus, it will make the memory footprint of text- or json-only server a little smaller.

Jeremy Apthorp
Owner

I don't think the problem is pure runtime weight of JS (especially not on the server side), it's maintenance cost.

That said, I haven't touched the JSON OT code since I wrote it, which hasn't really turned out to be a problem.

wmertens
Collaborator

Ok so it comes down to code support. How about we put in the docs that this code uses rather a lot of third-party javascript and it may break in the future?
The coffee code still contains a lot of JS-isms though - is that a dealbreaker?

We could pull it, mark it experimental and ship 0.6?

Constantin Jucovschi

Sorry for not being very active lately on this issue. So I've added the tests from the old etherpad repo. So that's out of the way. I also cleaned up the code so that it does not have JS-isms. Please recheck.

About code support: I don't think anyone wants to change Changeset and AttributePool libraries. So I guess maintenance cost is minimal. Now it also has test cases...

Cakefile
((5 lines not shown))
86 89
 	buildtype 'text-tp2'
87 90
 
88 91
 	# TODO: This should also be closure compiled.
89 92
 	extrafiles = expandNames extras
90 93
 	e "coffee --compile --output webclient/ #{extrafiles}", ->
91  
-		# For backwards compatibility. (The ace.js file used to be called share-ace.js)
92  
-		e "cp webclient/ace.js webclient/share-ace.js"
  94
+	# For backwards compatibility. (The ace.js file used to be called share-ace.js)
1
wmertens Collaborator

Why do you undent here? You can only copy ace when it's compiled...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Constantin Jucovschi

sorry :) fixed.

wmertens wmertens commented on the diff September 07, 2012
src/types/etherpad-api.coffee
... ...
@@ -0,0 +1,93 @@
  1
+# Text document API for text
  2
+# :tabSize=2:indentSize=2:
  3
+
  4
+if WEB? 
  5
+  if window.ShareJS? && window.ShareJS.Changeset?
  6
+    Changeset = window.ShareJS.Changeset
  7
+    AttributePool = window.ShareJS.AttributePool
  8
+  else
  9
+    console.log("Etherpad library not found. Make sure to include Attributepool.js and Changeset.js in your javascript sourcecode");
  10
+else 
  11
+  etherpad = require './../lib-etherpad/etherpad'
1
wmertens Collaborator

Use '../lib-etherpad/etherpad'. Also, that doesn't exist :-). The path to Changeset below is also wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
wmertens wmertens commented on the diff September 07, 2012
src/types/etherpad-api.coffee
((12 lines not shown))
  12
+  AttributePool = require './../lib-etherpad/AttributePool'  
  13
+  Changeset = require './Changeset'
  14
+  
  15
+etherpad.api =
  16
+  provides:  { text:true }
  17
+
  18
+  # The number of characters in the string
  19
+  getLength: -> @snapshot.text.length
  20
+
  21
+  # Get the text contents of a document
  22
+  getText: -> @snapshot.text
  23
+
  24
+  # Get metadata starting from offset startOffset and having length length
  25
+  getMeta: (startOffset, length) ->
  26
+    if typeof @snapshot.pool.getAttrib == "undefined"
  27
+      @snapshot = etherpad.tryDeserializeSnapshot(@snapshot);
1
wmertens Collaborator

@snapshot = etherpad.tryDeserializeSnapshot @snapshot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
wmertens wmertens commented on the diff September 07, 2012
src/types/etherpad-api.coffee
((18 lines not shown))
  18
+  # The number of characters in the string
  19
+  getLength: -> @snapshot.text.length
  20
+
  21
+  # Get the text contents of a document
  22
+  getText: -> @snapshot.text
  23
+
  24
+  # Get metadata starting from offset startOffset and having length length
  25
+  getMeta: (startOffset, length) ->
  26
+    if typeof @snapshot.pool.getAttrib == "undefined"
  27
+      @snapshot = etherpad.tryDeserializeSnapshot(@snapshot);
  28
+    snapshot = @snapshot;
  29
+    iter = Changeset.opIterator(snapshot.attribs)
  30
+    offset = 0;
  31
+    result = [];
  32
+    rangeStart = Changeset.numToString(@snapshot.pool.putAttrib(["range.start",1], true));
  33
+    rangeEnd = Changeset.numToString(@snapshot.pool.putAttrib(["range.end",1], true));
1
wmertens Collaborator

rangeEnd = Changeset.numToString @snapshot.pool.putAttrib ["range.end",1], true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
wmertens wmertens commented on the diff September 07, 2012
src/types/etherpad.coffee
((7 lines not shown))
  7
+#  }
  8
+
  9
+# The Changesets have the structure
  10
+#  {
  11
+#    "changeset" - serialized version of the changeset
  12
+#    "pool"  - the pool
  13
+#  }
  14
+
  15
+if WEB?
  16
+  if window.ShareJS? && window.ShareJS.Changeset?
  17
+    Changeset = window.ShareJS.Changeset
  18
+    AttributePool = window.ShareJS.AttributePool
  19
+  else
  20
+    console.log("Etherpad library not found. Make sure to include Attributepool.js and Changeset.js in your javascript sourcecode");
  21
+else
  22
+  Changeset = require("./../lib-etherpad/Changeset");
1
wmertens Collaborator

Changeset = require "../lib-etherpad/Changeset"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
wmertens
Collaborator

Starting to look good!

Jeremy Apthorp
Owner

Do we have a fuzzer for it yet?

wmertens
Collaborator

Errm, what is a fuzzer?

wmertens
Collaborator

Hi @jucovschi, there have been some changes since your pull request (e.g. ace), would you mind freshening up the code? Also, could you address the comments I made above?

Thanks!

Constantin Jucovschi

Created a remake (#172). Closing this pull.

Constantin Jucovschi jucovschi closed this February 25, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
6  Cakefile
@@ -19,6 +19,8 @@ client = [
19 19
 	'types/helpers'
20 20
 	'types/text'
21 21
 	'types/text-api'
  22
+	'types/etherpad'
  23
+	'types/etherpad-api'
22 24
 	'client/doc'
23 25
 	'client/connection'
24 26
 	'client/index'
@@ -83,13 +85,15 @@ buildtype = (name) ->
83 85
 task 'webclient', 'Build the web client into one file', ->
84 86
 	compile client, 'webclient/share'
85 87
 	buildtype 'json'
  88
+	buildtype 'etherpad'
86 89
 	buildtype 'text-tp2'
87 90
 
88 91
 	# TODO: This should also be closure compiled.
89 92
 	extrafiles = expandNames extras
90 93
 	e "coffee --compile --output webclient/ #{extrafiles}", ->
91  
-		# For backwards compatibility. (The ace.js file used to be called share-ace.js)
  94
+	# For backwards compatibility. (The ace.js file used to be called share-ace.js)
92 95
 		e "cp webclient/ace.js webclient/share-ace.js"
  96
+	e "cp src/lib-etherpad/* webclient/"
93 97
 
94 98
 #task 'lightwave', ->
95 99
 #	buildclosure ['client/web-prelude', 'client/microevent', 'types/text-tp2'], 'lightwave'
259  src/client/ace.coffee
... ...
@@ -1,128 +1,131 @@
1  
-# This is some utility code to connect an ace editor to a sharejs document.
2  
-
3  
-Range = require("ace/range").Range
4  
-
5  
-# Convert an ace delta into an op understood by share.js
6  
-applyToShareJS = (editorDoc, delta, doc) ->
7  
-  # Get the start position of the range, in no. of characters
8  
-  getStartOffsetPosition = (range) ->
9  
-    # This is quite inefficient - getLines makes a copy of the entire
10  
-    # lines array in the document. It would be nice if we could just
11  
-    # access them directly.
12  
-    lines = editorDoc.getLines 0, range.start.row
13  
-      
14  
-    offset = 0
15  
-
16  
-    for line, i in lines
17  
-      offset += if i < range.start.row
18  
-        line.length
19  
-      else
20  
-        range.start.column
21  
-
22  
-    # Add the row number to include newlines.
23  
-    offset + range.start.row
24  
-
25  
-  pos = getStartOffsetPosition(delta.range)
26  
-
27  
-  switch delta.action
28  
-    when 'insertText' then doc.insert pos, delta.text
29  
-    when 'removeText' then doc.del pos, delta.text.length
30  
-    
31  
-    when 'insertLines'
32  
-      text = delta.lines.join('\n') + '\n'
33  
-      doc.insert pos, text
34  
-      
35  
-    when 'removeLines'
36  
-      text = delta.lines.join('\n') + '\n'
37  
-      doc.del pos, text.length
38  
-
39  
-    else throw new Error "unknown action: #{delta.action}"
40  
-  
41  
-  return
42  
-
43  
-# Attach an ace editor to the document. The editor's contents are replaced
44  
-# with the document's contents unless keepEditorContents is true. (In which case the document's
45  
-# contents are nuked and replaced with the editor's).
46  
-window.sharejs.extendDoc 'attach_ace', (editor, keepEditorContents) ->
47  
-  throw new Error 'Only text documents can be attached to ace' unless @provides['text']
48  
-
49  
-  doc = this
50  
-  editorDoc = editor.getSession().getDocument()
51  
-  editorDoc.setNewLineMode 'unix'
52  
-
53  
-  check = ->
54  
-    window.setTimeout ->
55  
-        editorText = editorDoc.getValue()
56  
-        otText = doc.getText()
57  
-
58  
-        if editorText != otText
59  
-          console.error "Text does not match!"
60  
-          console.error "editor: #{editorText}"
61  
-          console.error "ot:     #{otText}"
62  
-          # Should probably also replace the editor text with the doc snapshot.
63  
-      , 0
64  
-
65  
-  if keepEditorContents
66  
-    doc.del 0, doc.getText().length
67  
-    doc.insert 0, editorDoc.getValue()
68  
-  else
69  
-    editorDoc.setValue doc.getText()
70  
-
71  
-  check()
72  
-
73  
-  # When we apply ops from sharejs, ace emits edit events. We need to ignore those
74  
-  # to prevent an infinite typing loop.
75  
-  suppress = false
76  
-  
77  
-  # Listen for edits in ace
78  
-  editorListener = (change) ->
79  
-    return if suppress
80  
-    applyToShareJS editorDoc, change.data, doc
81  
-
82  
-    check()
83  
-
84  
-  editorDoc.on 'change', editorListener
85  
-
86  
-  # Listen for remote ops on the sharejs document
87  
-  docListener = (op) ->
88  
-    suppress = true
89  
-    applyToDoc editorDoc, op
90  
-    suppress = false
91  
-
92  
-    check()
93  
-
94  
-
95  
-  # Horribly inefficient.
96  
-  offsetToPos = (offset) ->
97  
-    # Again, very inefficient.
98  
-    lines = editorDoc.getAllLines()
99  
-
100  
-    row = 0
101  
-    for line, row in lines
102  
-      break if offset <= line.length
103  
-
104  
-      # +1 for the newline.
105  
-      offset -= lines[row].length + 1
106  
-
107  
-    row:row, column:offset
108  
-
109  
-  doc.on 'insert', (pos, text) ->
110  
-    suppress = true
111  
-    editorDoc.insert offsetToPos(pos), text
112  
-    suppress = false
113  
-    check()
114  
-
115  
-  doc.on 'delete', (pos, text) ->
116  
-    suppress = true
117  
-    range = Range.fromPoints offsetToPos(pos), offsetToPos(pos + text.length)
118  
-    editorDoc.remove range
119  
-    suppress = false
120  
-    check()
121  
-
122  
-  doc.detach_ace = ->
123  
-    doc.removeListener 'remoteop', docListener
124  
-    editorDoc.removeListener 'change', editorListener
125  
-    delete doc.detach_ace
126  
-
127  
-  return
128  
-
  1
+# This is some utility code to connect an ace editor to a sharejs document.
  2
+
  3
+if (!(require?) && ace.require?)
  4
+  require = ace.require
  5
+
  6
+Range = require("ace/range").Range
  7
+
  8
+# Convert an ace delta into an op understood by share.js
  9
+applyToShareJS = (editorDoc, delta, doc) ->
  10
+  # Get the start position of the range, in no. of characters
  11
+  getStartOffsetPosition = (range) ->
  12
+    # This is quite inefficient - getLines makes a copy of the entire
  13
+    # lines array in the document. It would be nice if we could just
  14
+    # access them directly.
  15
+    lines = editorDoc.getLines 0, range.start.row
  16
+      
  17
+    offset = 0
  18
+
  19
+    for line, i in lines
  20
+      offset += if i < range.start.row
  21
+        line.length
  22
+      else
  23
+        range.start.column
  24
+
  25
+    # Add the row number to include newlines.
  26
+    offset + range.start.row
  27
+
  28
+  pos = getStartOffsetPosition(delta.range)
  29
+
  30
+  switch delta.action
  31
+    when 'insertText' then doc.insert pos, delta.text
  32
+    when 'removeText' then doc.del pos, delta.text.length
  33
+    
  34
+    when 'insertLines'
  35
+      text = delta.lines.join('\n') + '\n'
  36
+      doc.insert pos, text
  37
+      
  38
+    when 'removeLines'
  39
+      text = delta.lines.join('\n') + '\n'
  40
+      doc.del pos, text.length
  41
+
  42
+    else throw new Error "unknown action: #{delta.action}"
  43
+  
  44
+  return
  45
+
  46
+# Attach an ace editor to the document. The editor's contents are replaced
  47
+# with the document's contents unless keepEditorContents is true. (In which case the document's
  48
+# contents are nuked and replaced with the editor's).
  49
+window.sharejs.extendDoc 'attach_ace', (editor, keepEditorContents) ->
  50
+  throw new Error 'Only text documents can be attached to ace' unless @provides['text']
  51
+
  52
+  doc = this
  53
+  editorDoc = editor.getSession().getDocument()
  54
+  editorDoc.setNewLineMode 'unix'
  55
+
  56
+  check = ->
  57
+    window.setTimeout ->
  58
+        editorText = editorDoc.getValue()
  59
+        otText = doc.getText()
  60
+
  61
+        if editorText != otText
  62
+          console.error "Text does not match!"
  63
+          console.error "editor: #{editorText}"
  64
+          console.error "ot:     #{otText}"
  65
+          # Should probably also replace the editor text with the doc snapshot.
  66
+      , 0
  67
+
  68
+  if keepEditorContents
  69
+    doc.del 0, doc.getText().length
  70
+    doc.insert 0, editorDoc.getValue()
  71
+  else
  72
+    editorDoc.setValue doc.getText()
  73
+
  74
+  check()
  75
+
  76
+  # When we apply ops from sharejs, ace emits edit events. We need to ignore those
  77
+  # to prevent an infinite typing loop.
  78
+  suppress = false
  79
+  
  80
+  # Listen for edits in ace
  81
+  editorListener = (change) ->
  82
+    return if suppress
  83
+    applyToShareJS editorDoc, change.data, doc
  84
+
  85
+    check()
  86
+
  87
+  editorDoc.on 'change', editorListener
  88
+
  89
+  # Listen for remote ops on the sharejs document
  90
+  docListener = (op) ->
  91
+    suppress = true
  92
+    applyToDoc editorDoc, op
  93
+    suppress = false
  94
+
  95
+    check()
  96
+
  97
+
  98
+  # Horribly inefficient.
  99
+  offsetToPos = (offset) ->
  100
+    # Again, very inefficient.
  101
+    lines = editorDoc.getAllLines()
  102
+
  103
+    row = 0
  104
+    for line, row in lines
  105
+      break if offset <= line.length
  106
+
  107
+      # +1 for the newline.
  108
+      offset -= lines[row].length + 1
  109
+
  110
+    row:row, column:offset
  111
+
  112
+  doc.on 'insert', (pos, text) ->
  113
+    suppress = true
  114
+    editorDoc.insert offsetToPos(pos), text
  115
+    suppress = false
  116
+    check()
  117
+
  118
+  doc.on 'delete', (pos, text) ->
  119
+    suppress = true
  120
+    range = Range.fromPoints offsetToPos(pos), offsetToPos(pos + text.length)
  121
+    editorDoc.remove range
  122
+    suppress = false
  123
+    check()
  124
+
  125
+  doc.detach_ace = ->
  126
+    doc.removeListener 'remoteop', docListener
  127
+    editorDoc.removeListener 'change', editorListener
  128
+    delete doc.detach_ace
  129
+
  130
+  return
  131
+
112  src/lib-etherpad/AttributePool.js
... ...
@@ -0,0 +1,112 @@
  1
+/**
  2
+ * This code represents the Attribute Pool Object of the original Etherpad.
  3
+ * 90% of the code is still like in the original Etherpad
  4
+ * Look at https://github.com/ether/pad/blob/master/infrastructure/ace/www/easysync2.js
  5
+ * You can find a explanation what a attribute pool is here:
  6
+ * https://github.com/Pita/etherpad-lite/blob/master/doc/easysync/easysync-notes.txt
  7
+ */
  8
+
  9
+/*
  10
+ * Copyright 2009 Google Inc., 2011 Peter 'Pita' Martischka (Primary Technology Ltd)
  11
+ *
  12
+ * Licensed under the Apache License, Version 2.0 (the "License");
  13
+ * you may not use this file except in compliance with the License.
  14
+ * You may obtain a copy of the License at
  15
+ *
  16
+ *      http://www.apache.org/licenses/LICENSE-2.0
  17
+ *
  18
+ * Unless required by applicable law or agreed to in writing, software
  19
+ * distributed under the License is distributed on an "AS-IS" BASIS,
  20
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  21
+ * See the License for the specific language governing permissions and
  22
+ * limitations under the License.
  23
+ */
  24
+
  25
+/*
  26
+  An AttributePool maintains a mapping from [key,value] Pairs called
  27
+  Attributes to Numbers (unsigened integers) and vice versa. These numbers are
  28
+  used to reference Attributes in Changesets.
  29
+*/
  30
+
  31
+var AttributePool = function () {
  32
+  this.numToAttrib = {}; // e.g. {0: ['foo','bar']}
  33
+  this.attribToNum = {}; // e.g. {'foo,bar': 0}
  34
+  this.nextNum = 0;
  35
+};
  36
+
  37
+AttributePool.prototype.putAttrib = function (attrib, dontAddIfAbsent) {
  38
+  var str = String(attrib);
  39
+  if (str in this.attribToNum) {
  40
+    return this.attribToNum[str];
  41
+  }
  42
+  if (dontAddIfAbsent) {
  43
+    return -1;
  44
+  }
  45
+  var num = this.nextNum++;
  46
+  this.attribToNum[str] = num;
  47
+  this.numToAttrib[num] = [String(attrib[0] || ''), String(attrib[1] || '')];
  48
+  return num;
  49
+};
  50
+
  51
+AttributePool.prototype.getAttrib = function (num) {
  52
+  var pair = this.numToAttrib[num];
  53
+  if (!pair) {
  54
+    return pair;
  55
+  }
  56
+  return [pair[0], pair[1]]; // return a mutable copy
  57
+};
  58
+
  59
+AttributePool.prototype.getAttribKey = function (num) {
  60
+  var pair = this.numToAttrib[num];
  61
+  if (!pair) return '';
  62
+  return pair[0];
  63
+};
  64
+
  65
+AttributePool.prototype.clone = function() {
  66
+  var result = new AttributePool();
  67
+  result.numToAttrib = this.numToAttrib;
  68
+  result.attribToNum = this.attribToNum;
  69
+  result.nextNum = this.nextNum;
  70
+  return result;
  71
+}
  72
+
  73
+AttributePool.prototype.getAttribValue = function (num) {
  74
+  var pair = this.numToAttrib[num];
  75
+  if (!pair) return '';
  76
+  return pair[1];
  77
+};
  78
+
  79
+AttributePool.prototype.eachAttrib = function (func) {
  80
+  for (var n in this.numToAttrib) {
  81
+    var pair = this.numToAttrib[n];
  82
+    func(pair[0], pair[1]);
  83
+  }
  84
+};
  85
+
  86
+AttributePool.prototype.toJsonable = function () {
  87
+  return {
  88
+    numToAttrib: this.numToAttrib,
  89
+    nextNum: this.nextNum
  90
+  };
  91
+};
  92
+
  93
+AttributePool.prototype.fromJsonable = function (obj) {
  94
+  this.numToAttrib = obj.numToAttrib;
  95
+  this.nextNum = obj.nextNum;
  96
+  this.attribToNum = {};
  97
+  for (var n in this.numToAttrib) {
  98
+    this.attribToNum[String(this.numToAttrib[n])] = Number(n);
  99
+  }
  100
+  return this;
  101
+};
  102
+
  103
+if (typeof window !== "undefined") {
  104
+	if (typeof window.ShareJS === "undefined") {
  105
+		window.ShareJS = {};
  106
+	}
  107
+	window.ShareJS.AttributePool = AttributePool
  108
+} else {
  109
+	exports.AttributePool = AttributePool;
  110
+	module.exports = AttributePool;
  111
+}
  112
+
2,261  src/lib-etherpad/Changeset.js
... ...
@@ -0,0 +1,2261 @@
  1
+/*
  2
+ * This is the Changeset library copied from the old Etherpad with some modifications to use it in node.js
  3
+ * Can be found in https://github.com/ether/pad/blob/master/infrastructure/ace/www/easysync2.js
  4
+ */
  5
+
  6
+/**
  7
+ * This code is mostly from the old Etherpad. Please help us to comment this code.
  8
+ * This helps other people to understand this code better and helps them to improve it.
  9
+ * TL;DR COMMENTS ON THIS FILE ARE HIGHLY APPRECIATED
  10
+ */
  11
+
  12
+/*
  13
+ * Copyright 2009 Google Inc., 
  14
+ *
  15
+ * Licensed under the Apache License, Version 2.0 (the "License");
  16
+ * you may not use this file except in compliance with the License.
  17
+ * You may obtain a copy of the License at
  18
+ *
  19
+ *      http://www.apache.org/licenses/LICENSE-2.0
  20
+ *
  21
+ * Unless required by applicable law or agreed to in writing, software
  22
+ * distributed under the License is distributed on an "AS-IS" BASIS,
  23
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  24
+ * See the License for the specific language governing permissions and
  25
+ * limitations under the License.
  26
+ */
  27
+
  28
+if (typeof window === "undefined") {
  29
+	AttributePool = require("./AttributePool");
  30
+} else {
  31
+	exports = {}
  32
+	AttributePool = window.ShareJS.AttributePool
  33
+}
  34
+
  35
+var _opt = null;
  36
+
  37
+/**
  38
+ * ==================== General Util Functions =======================
  39
+ */
  40
+
  41
+/**
  42
+ * This method is called whenever there is an error in the sync process
  43
+ * @param msg {string} Just some message
  44
+ */
  45
+exports.error = function error(msg) {
  46
+  var e = new Error(msg);
  47
+  e.easysync = true;
  48
+  throw e;
  49
+};
  50
+
  51
+/**
  52
+ * This method is user for assertions with Messages 
  53
+ * if assert fails, the error function called.
  54
+ * @param b {boolean} assertion condition
  55
+ * @param msgParts {string} error to be passed if it fails
  56
+ */
  57
+exports.assert = function assert(b, msgParts) {
  58
+  if (!b) {
  59
+    var msg = Array.prototype.slice.call(arguments, 1).join('');
  60
+    exports.error("exports: " + msg);
  61
+  }
  62
+};
  63
+
  64
+/**
  65
+ * Parses a number from string base 36
  66
+ * @param str {string} string of the number in base 36
  67
+ * @returns {int} number
  68
+ */
  69
+exports.parseNum = function (str) {
  70
+  return parseInt(str, 36);
  71
+};
  72
+
  73
+/**
  74
+ * Writes a number in base 36 and puts it in a string
  75
+ * @param num {int} number
  76
+ * @returns {string} string
  77
+ */
  78
+exports.numToString = function (num) {
  79
+  return num.toString(36).toLowerCase();
  80
+};
  81
+
  82
+/**
  83
+ * Converts stuff before $ to base 10
  84
+ * @obsolete not really used anywhere??
  85
+ * @param cs {string} the string
  86
+ * @return integer 
  87
+ */
  88
+exports.toBaseTen = function (cs) {
  89
+  var dollarIndex = cs.indexOf('$');
  90
+  var beforeDollar = cs.substring(0, dollarIndex);
  91
+  var fromDollar = cs.substring(dollarIndex);
  92
+  return beforeDollar.replace(/[0-9a-z]+/g, function (s) {
  93
+    return String(exports.parseNum(s));
  94
+  }) + fromDollar;
  95
+};
  96
+
  97
+
  98
+/**
  99
+ * ==================== Changeset Functions =======================
  100
+ */
  101
+
  102
+/**
  103
+ * returns the required length of the text before changeset 
  104
+ * can be applied
  105
+ * @param cs {string} String representation of the Changeset
  106
+ */ 
  107
+exports.oldLen = function (cs) {
  108
+  return exports.unpack(cs).oldLen;
  109
+};
  110
+
  111
+/**
  112
+ * returns the length of the text after changeset is applied
  113
+ * @param cs {string} String representation of the Changeset
  114
+ */ 
  115
+exports.newLen = function (cs) {
  116
+  return exports.unpack(cs).newLen;
  117
+};
  118
+
  119
+/**
  120
+ * this function creates an iterator which decodes string changeset operations
  121
+ * @param opsStr {string} String encoding of the change operations to be performed 
  122
+ * @param optStartIndex {int} from where in the string should the iterator start 
  123
+ * @return {Op} type object iterator 
  124
+ */
  125
+exports.opIterator = function (opsStr, optStartIndex) {
  126
+  //print(opsStr);
  127
+  var regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|\?|/g;
  128
+  var startIndex = (optStartIndex || 0);
  129
+  var curIndex = startIndex;
  130
+  var prevIndex = curIndex;
  131
+
  132
+  function nextRegexMatch() {
  133
+    prevIndex = curIndex;
  134
+    var result;
  135
+    if (_opt) {
  136
+      result = _opt.nextOpInString(opsStr, curIndex);
  137
+      if (result) {
  138
+        if (result.opcode() == '?') {
  139
+          exports.error("Hit error opcode in op stream");
  140
+        }
  141
+        curIndex = result.lastIndex();
  142
+      }
  143
+    } else {
  144
+      regex.lastIndex = curIndex;
  145
+      result = regex.exec(opsStr);
  146
+      curIndex = regex.lastIndex;
  147
+      if (result[0] == '?') {
  148
+        exports.error("Hit error opcode in op stream");
  149
+      }
  150
+    }
  151
+    return result;
  152
+  }
  153
+  var regexResult = nextRegexMatch();
  154
+  var obj = exports.newOp();
  155
+
  156
+  function next(optObj) {
  157
+    var op = (optObj || obj);
  158
+    if (_opt && regexResult) {
  159
+      op.attribs = regexResult.attribs();
  160
+      op.lines = regexResult.lines();
  161
+      op.chars = regexResult.chars();
  162
+      op.opcode = regexResult.opcode();
  163
+      regexResult = nextRegexMatch();
  164
+    } else if ((!_opt) && regexResult[0]) {
  165
+      op.attribs = regexResult[1];
  166
+      op.lines = exports.parseNum(regexResult[2] || 0);
  167
+      op.opcode = regexResult[3];
  168
+      op.chars = exports.parseNum(regexResult[4]);
  169
+      regexResult = nextRegexMatch();
  170
+    } else {
  171
+      exports.clearOp(op);
  172
+    }
  173
+    return op;
  174
+  }
  175
+
  176
+  function hasNext() {
  177
+    return !!(_opt ? regexResult : regexResult[0]);
  178
+  }
  179
+
  180
+  function lastIndex() {
  181
+    return prevIndex;
  182
+  }
  183
+  return {
  184
+    next: next,
  185
+    hasNext: hasNext,
  186
+    lastIndex: lastIndex
  187
+  };
  188
+};
  189
+
  190
+/**
  191
+ * Cleans an Op object
  192
+ * @param {Op} object to be cleared
  193
+ */
  194
+exports.clearOp = function (op) {
  195
+  op.opcode = '';
  196
+  op.chars = 0;
  197
+  op.lines = 0;
  198
+  op.attribs = '';
  199
+};
  200
+
  201
+/**
  202
+ * Creates a new Op object
  203
+ * @param optOpcode the type operation of the Op object
  204
+ */
  205
+exports.newOp = function (optOpcode) {
  206
+  return {
  207
+    opcode: (optOpcode || ''),
  208
+    chars: 0,
  209
+    lines: 0,
  210
+    attribs: ''
  211
+  };
  212
+};
  213
+
  214
+/**
  215
+ * Clones an Op
  216
+ * @param op Op to be cloned
  217
+ */
  218
+exports.cloneOp = function (op) {
  219
+  return {
  220
+    opcode: op.opcode,
  221
+    chars: op.chars,
  222
+    lines: op.lines,
  223
+    attribs: op.attribs
  224
+  };
  225
+};
  226
+
  227
+/**
  228
+ * Copies op1 to op2
  229
+ * @param op1 src Op
  230
+ * @param op2 dest Op
  231
+ */
  232
+exports.copyOp = function (op1, op2) {
  233
+  op2.opcode = op1.opcode;
  234
+  op2.chars = op1.chars;
  235
+  op2.lines = op1.lines;
  236
+  op2.attribs = op1.attribs;
  237
+};
  238
+
  239
+/**
  240
+ * Writes the Op in a string the way that changesets need it
  241
+ */
  242
+exports.opString = function (op) {
  243
+  // just for debugging
  244
+  if (!op.opcode) return 'null';
  245
+  var assem = exports.opAssembler();
  246
+  assem.append(op);
  247
+  return assem.toString();
  248
+};
  249
+
  250
+/**
  251
+ * Used just for debugging
  252
+ */
  253
+exports.stringOp = function (str) {
  254
+  // just for debugging
  255
+  return exports.opIterator(str).next();
  256
+};
  257
+
  258
+/**
  259
+ * Used to check if a Changeset if valid
  260
+ * @param cs {Changeset} Changeset to be checked
  261
+ */
  262
+exports.checkRep = function (cs) {
  263
+  // doesn't check things that require access to attrib pool (e.g. attribute order)
  264
+  // or original string (e.g. newline positions)
  265
+  var unpacked = exports.unpack(cs);
  266
+  var oldLen = unpacked.oldLen;
  267
+  var newLen = unpacked.newLen;
  268
+  var ops = unpacked.ops;
  269
+  var charBank = unpacked.charBank;
  270
+
  271
+  var assem = exports.smartOpAssembler();
  272
+  var oldPos = 0;
  273
+  var calcNewLen = 0;
  274
+  var numInserted = 0;
  275
+  var iter = exports.opIterator(ops);
  276
+  while (iter.hasNext()) {
  277
+    var o = iter.next();
  278
+    switch (o.opcode) {
  279
+    case '=':
  280
+      oldPos += o.chars;
  281
+      calcNewLen += o.chars;
  282
+      break;
  283
+    case '-':
  284
+      oldPos += o.chars;
  285
+      exports.assert(oldPos < oldLen, oldPos, " >= ", oldLen, " in ", cs);
  286
+      break;
  287
+    case '+':
  288
+      {
  289
+        calcNewLen += o.chars;
  290
+        numInserted += o.chars;
  291
+        exports.assert(calcNewLen < newLen, calcNewLen, " >= ", newLen, " in ", cs);
  292
+        break;
  293
+      }
  294
+    }
  295
+    assem.append(o);
  296
+  }
  297
+
  298
+  calcNewLen += oldLen - oldPos;
  299
+  charBank = charBank.substring(0, numInserted);
  300
+  while (charBank.length < numInserted) {
  301
+    charBank += "?";
  302
+  }
  303
+
  304
+  assem.endDocument();
  305
+  var normalized = exports.pack(oldLen, calcNewLen, assem.toString(), charBank);
  306
+  exports.assert(normalized == cs, normalized, ' != ', cs);
  307
+
  308
+  return cs;
  309
+}
  310
+
  311
+
  312
+/**
  313
+ * ==================== Util Functions =======================
  314
+ */
  315
+
  316
+/**
  317
+ * creates an object that allows you to append operations (type Op) and also
  318
+ * compresses them if possible
  319
+ */
  320
+exports.smartOpAssembler = function () {
  321
+  // Like opAssembler but able to produce conforming exportss
  322
+  // from slightly looser input, at the cost of speed.
  323
+  // Specifically:
  324
+  // - merges consecutive operations that can be merged
  325
+  // - strips final "="
  326
+  // - ignores 0-length changes
  327
+  // - reorders consecutive + and - (which margingOpAssembler doesn't do)
  328
+  var minusAssem = exports.mergingOpAssembler();
  329
+  var plusAssem = exports.mergingOpAssembler();
  330
+  var keepAssem = exports.mergingOpAssembler();
  331
+  var assem = exports.stringAssembler();
  332
+  var lastOpcode = '';
  333
+  var lengthChange = 0;
  334
+
  335
+  function flushKeeps() {
  336
+    assem.append(keepAssem.toString());
  337
+    keepAssem.clear();
  338
+  }
  339
+
  340
+  function flushPlusMinus() {
  341
+    assem.append(minusAssem.toString());
  342
+    minusAssem.clear();
  343
+    assem.append(plusAssem.toString());
  344
+    plusAssem.clear();
  345
+  }
  346
+
  347
+  function append(op) {
  348
+    if (!op.opcode) return;
  349
+    if (!op.chars) return;
  350
+
  351
+    if (op.opcode == '-') {
  352
+      if (lastOpcode == '=') {
  353
+        flushKeeps();
  354
+      }
  355
+      minusAssem.append(op);
  356
+      lengthChange -= op.chars;
  357
+    } else if (op.opcode == '+') {
  358
+      if (lastOpcode == '=') {
  359
+        flushKeeps();
  360
+      }
  361
+      plusAssem.append(op);
  362
+      lengthChange += op.chars;
  363
+    } else if (op.opcode == '=') {
  364
+      if (lastOpcode != '=') {
  365
+        flushPlusMinus();
  366
+      }
  367
+      keepAssem.append(op);
  368
+    }
  369
+    lastOpcode = op.opcode;
  370
+  }
  371
+
  372
+  function appendOpWithText(opcode, text, attribs, pool) {
  373
+    var op = exports.newOp(opcode);
  374
+    op.attribs = exports.makeAttribsString(opcode, attribs, pool);
  375
+    var lastNewlinePos = text.lastIndexOf('\n');
  376
+    if (lastNewlinePos < 0) {
  377
+      op.chars = text.length;
  378
+      op.lines = 0;
  379
+      append(op);
  380
+    } else {
  381
+      op.chars = lastNewlinePos + 1;
  382
+      op.lines = text.match(/\n/g).length;
  383
+      append(op);
  384
+      op.chars = text.length - (lastNewlinePos + 1);
  385
+      op.lines = 0;
  386
+      append(op);
  387
+    }
  388
+  }
  389
+
  390
+  function toString() {
  391
+    flushPlusMinus();
  392
+    flushKeeps();
  393
+    return assem.toString();
  394
+  }
  395
+
  396
+  function clear() {
  397
+    minusAssem.clear();
  398
+    plusAssem.clear();
  399
+    keepAssem.clear();
  400
+    assem.clear();
  401
+    lengthChange = 0;
  402
+  }
  403
+
  404
+  function endDocument() {
  405
+    keepAssem.endDocument();
  406
+  }
  407
+
  408
+  function getLengthChange() {
  409
+    return lengthChange;
  410
+  }
  411
+
  412
+  return {
  413
+    append: append,
  414
+    toString: toString,
  415
+    clear: clear,
  416
+    endDocument: endDocument,
  417
+    appendOpWithText: appendOpWithText,
  418
+    getLengthChange: getLengthChange
  419
+  };
  420
+};
  421
+
  422
+if (_opt) {
  423
+  exports.mergingOpAssembler = function () {
  424
+    var assem = _opt.mergingOpAssembler();
  425
+
  426
+    function append(op) {
  427
+      assem.append(op.opcode, op.chars, op.lines, op.attribs);
  428
+    }
  429
+
  430
+    function toString() {
  431
+      return assem.toString();
  432
+    }
  433
+
  434
+    function clear() {
  435
+      assem.clear();
  436
+    }
  437
+
  438
+    function endDocument() {
  439
+      assem.endDocument();
  440
+    }
  441
+
  442
+    return {
  443
+      append: append,
  444
+      toString: toString,
  445
+      clear: clear,
  446
+      endDocument: endDocument
  447
+    };
  448
+  };
  449
+} else {
  450
+  exports.mergingOpAssembler = function () {
  451
+    // This assembler can be used in production; it efficiently
  452
+    // merges consecutive operations that are mergeable, ignores
  453
+    // no-ops, and drops final pure "keeps".  It does not re-order
  454
+    // operations.
  455
+    var assem = exports.opAssembler();
  456
+    var bufOp = exports.newOp();
  457
+
  458
+    // If we get, for example, insertions [xxx\n,yyy], those don't merge,
  459
+    // but if we get [xxx\n,yyy,zzz\n], that merges to [xxx\nyyyzzz\n].
  460
+    // This variable stores the length of yyy and any other newline-less
  461
+    // ops immediately after it.
  462
+    var bufOpAdditionalCharsAfterNewline = 0;
  463
+
  464
+    function flush(isEndDocument) {
  465
+      if (bufOp.opcode) {
  466
+        if (isEndDocument && bufOp.opcode == '=' && !bufOp.attribs) {
  467
+          // final merged keep, leave it implicit
  468
+        } else {
  469
+          assem.append(bufOp);
  470
+          if (bufOpAdditionalCharsAfterNewline) {
  471
+            bufOp.chars = bufOpAdditionalCharsAfterNewline;
  472
+            bufOp.lines = 0;
  473
+            assem.append(bufOp);
  474
+            bufOpAdditionalCharsAfterNewline = 0;
  475
+          }
  476
+        }
  477
+        bufOp.opcode = '';
  478
+      }
  479
+    }
  480
+
  481
+    function append(op) {
  482
+      if (op.chars > 0) {
  483
+        if (bufOp.opcode == op.opcode && bufOp.attribs == op.attribs) {
  484
+          if (op.lines > 0) {
  485
+            // bufOp and additional chars are all mergeable into a multi-line op
  486
+            bufOp.chars += bufOpAdditionalCharsAfterNewline + op.chars;
  487
+            bufOp.lines += op.lines;
  488
+            bufOpAdditionalCharsAfterNewline = 0;
  489
+          } else if (bufOp.lines == 0) {
  490
+            // both bufOp and op are in-line
  491
+            bufOp.chars += op.chars;
  492
+          } else {
  493
+            // append in-line text to multi-line bufOp
  494
+            bufOpAdditionalCharsAfterNewline += op.chars;
  495
+          }
  496
+        } else {
  497
+          flush();
  498
+          exports.copyOp(op, bufOp);
  499
+        }
  500
+      }
  501
+    }
  502
+
  503
+    function endDocument() {
  504
+      flush(true);
  505
+    }
  506
+
  507
+    function toString() {
  508
+      flush();
  509
+      return assem.toString();
  510
+    }
  511
+
  512
+    function clear() {
  513
+      assem.clear();
  514
+      exports.clearOp(bufOp);
  515
+    }
  516
+    return {
  517
+      append: append,
  518
+      toString: toString,
  519
+      clear: clear,
  520
+      endDocument: endDocument
  521
+    };
  522
+  };
  523
+}
  524
+
  525
+if (_opt) {
  526
+  exports.opAssembler = function () {
  527
+    var assem = _opt.opAssembler();
  528
+    // this function allows op to be mutated later (doesn't keep a ref)
  529
+
  530
+    function append(op) {
  531
+      assem.append(op.opcode, op.chars, op.lines, op.attribs);
  532
+    }
  533
+
  534
+    function toString() {
  535
+      return assem.toString();
  536
+    }
  537
+
  538
+    function clear() {
  539
+      assem.clear();
  540
+    }
  541
+    return {
  542
+      append: append,
  543
+      toString: toString,
  544
+      clear: clear
  545
+    };
  546
+  };
  547
+} else {
  548
+  exports.opAssembler = function () {
  549
+    var pieces = [];
  550
+    // this function allows op to be mutated later (doesn't keep a ref)
  551
+
  552
+    function append(op) {
  553
+      pieces.push(op.attribs);
  554
+      if (op.lines) {
  555
+        pieces.push('|', exports.numToString(op.lines));
  556
+      }
  557
+      pieces.push(op.opcode);
  558
+      pieces.push(exports.numToString(op.chars));
  559
+    }
  560
+
  561
+    function toString() {
  562
+      return pieces.join('');
  563
+    }
  564
+
  565
+    function clear() {
  566
+      pieces.length = 0;
  567
+    }
  568
+    return {
  569
+      append: append,
  570
+      toString: toString,
  571
+      clear: clear
  572
+    };
  573
+  };
  574
+}
  575
+
  576
+/**
  577
+ * A custom made String Iterator
  578
+ * @param str {string} String to be iterated over
  579
+ */ 
  580
+exports.stringIterator = function (str) {
  581
+  var curIndex = 0;
  582
+
  583
+  function assertRemaining(n) {
  584
+    exports.assert(n <= remaining(), "!(", n, " <= ", remaining(), ")");
  585
+  }
  586
+
  587
+  function take(n) {
  588
+    assertRemaining(n);
  589
+    var s = str.substr(curIndex, n);
  590
+    curIndex += n;
  591
+    return s;
  592
+  }
  593
+
  594
+  function peek(n) {
  595
+    assertRemaining(n);
  596
+    var s = str.substr(curIndex, n);