Skip to content
This repository

API & backend to list pads on epl instance #1342

Merged
merged 7 commits into from over 1 year ago

5 participants

Swen Jordan Hollinger Marcel Klehr John McLear Wikinaut
Swen

Introduces the ability to list all pads currently on this instance.
The pads can be retrieved via api calling /api/1.2.1/listAllPads

How it works:

  • Startup:
    • retrieves dataset containing all known padIDs from db key pads into js object storage
  • Runtime:
    • opening pad:
      • adds padID to storage if not existing
      • saves storage to db
    • creating pad
      • adds padID to storage
      • saves storage to db
    • remove pad
      • removes padID from storage
      • saves storage to db

Limitations:

  • Pads created before this feature are not listed
Jordan Hollinger

There have been several attempts at something like this, but they all have the "Pads created before this feature are not listed" limitation. Code looks really clean, but IMHO that limitation is a deal-breaker; users will never understand. To them, any software that uses this API will seem broken.

Marcel Klehr
Owner

That's quite true, sadly.. In fact, it's this software -- Etherpad lite -- that's broken. It's a shame, actually

Swen

Alright, thanks for info.
From my understanding, to provide "legacy" support, I would have to introduce some sort of find functionality to ueberDB.
Do you know if there are any limitations on extending this to provide a find feature?

Jordan Hollinger

That's my understanding, too. Since it hasn't been done yet, I bet it's not easy :). I think @Pita is the author/maintainer? I would start with him.

Also, you may want to consider making pagination part of this API method. Instances could have 100's or 1000's of pads. Unless your ueberDB find method is ridiculously fast, listing them all at once won't be quick.

Marcel Klehr
Owner

Indeed, that's one of the most broken parts of epl and it's a serious issue. You can't query the database.

But, you want to query the database. You need to query the database. And most reasonable dbs even provide this functionality natively.

Imagine for a moment, you observed that all verhicles have wheels, would you thereafter ignore all other properties of any vehicle you stumple upon, attach a rope to it and just pull it by hand? In my humble opinion, this is what ueberDB does. It limits us. So, when the user sets up some any high-end, super fast database, ueberDB prevents us from using this awesomeness and instead limits us to using stuff that only the most feature-lacking piece of software ever to be called "database" can do (so you do end up pulling your car with a rope, eventually).

Instead of using a tool that provides us with the features of the lowest common denominator, we should be using a tool that allows us to use any piece of software called "database" as if it had all the shiny, useful features, something that monkeypatches all those halfbreeds to be actually usable.

Jordan Hollinger

+1 @marcelklehr. So epl should be using a more traditional db abstraction layer. Is it practical to switch while providing a painless migration path for existing installations? I have less free time these days, but I would be willing to contribute to such an effort.

John McLear
Owner

fyi I think I have admin on UeberDB too so that kinda makes me a maintainer but im not so much interested in writing query functionality

Swen

Ok, implemented the basic functionality for some dbs (dirty, mysql and mongo).
https://github.com/spcsser/ueberDB/tree/feature/findKeys
How it works is in the wiki: https://github.com/spcsser/ueberDB/wiki/findKeys-function

For my desired feature - to list all pads - this would suffice. (Having only to call it once on startup and work later with live data.) For other use, especially heavy searching this might not perfom, since it has to rely on regex to retrieve data - at least for some dbs.

+1 @marcelklehr For the future - especially to provide better search functionality - the use of some dbal to gain native db functionality back would be a good thing.

Swen spcsser referenced this pull request in Pita/ueberDB January 09, 2013
Merged

findKeys functionality #18

John McLear
Owner
Wikinaut

@spcsser you might be interested in fast fuzzy text searching. Then search for agrep.

Marcel Klehr
Owner

You might also be interested in the thoughts I wrote down over here: #697 (comment)

John McLear
Owner

UeberDB stuff merged, waiting on Peter to give me ability to npm publish ;\

Swen

While we're waiting: I created a lil' epl addon that lists the pads - https://github.com/spcsser/ep_adminpads.

John McLear
Owner

Just spotted, getPads is listed in the APIHandler file for 1.2.1 when it wont be available then?

Swen

I'm not sure what you mean, so I'm trying to guess: If I'm not mistaken, getPads is to retrieve all pads for a certain groupID. The api function to list all pads I called getAllPads.
Is that what you meant?

John McLear
Owner

My understanding is that

+#### listAllPads()
     426    
+ * API >= 1.2.1

needs to be changed to
+#### listAllPads()
426

  • * API >= 1.2.5

as it wont be in until 1.2.5

@marcelklehr knows more about this convention
Swen spcsser referenced this pull request in spcsser/ep_adminpads January 12, 2013
Closed

Please publish to npm #1

Marcel Klehr
Owner
Swen

So, API version == EPL version? I always thought they'd be kept separated.

Edit: @marcelklehr Thanks for clarifying :)

Marcel Klehr
Owner
John McLear
Owner

Then once Pita ads me as on Ueber and I push we can merge this :)

John McLear
Owner
John McLear JohnMcLear merged commit 654654b into from January 14, 2013
John McLear JohnMcLear closed this January 14, 2013
Deleted user

The content you are editing has changed. Reload the page and try again.

For me, using mysql the padList is not initialized (or remains empty) on epl start. Once opend pads are listed though.

Log says:
[2013-01-22 22:05:46.009] [DEBUG] ueberDB - GET - pad:-:: - [] - from database

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Swen

This pull request should fix that: Pita/ueberDB#26.

Deleted user

The content you are editing has changed. Reload the page and try again.

With an updates ueberDB it works. thx

Sending Request…

Attach images by dragging & dropping or selecting them. Octocat-spinner-32 Uploading your images… Unfortunately, we don't support that file type. Try again with a PNG, GIF, or JPG. Yowza, that's a big file. Try again with an image file smaller than 10MB. This browser doesn't support image attachments. We recommend updating to the latest Internet Explorer, Google Chrome, or Firefox. Something went really wrong, and we can't process that image. Try again.

Marcel Klehr marcelklehr referenced this pull request June 30, 2013
Closed

very slow startup #1821

John McLear
Owner

Ta @marcelklehr @spcsser can you investigate please?

John McLear
Owner

@spcsser why does this need to run on startup?

FYI this is really hurting startup performance on large deployments

No other way to ensure to have all pads and enable search & stuff. Querying db was not possible afaik.
Could be run async or delayed somehow.

seems ok to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
10  doc/api/http_api.md
Source Rendered
@@ -419,3 +419,13 @@ returns ok when api token is valid
419 419
 *Example returns:*
420 420
   * `{"code":0,"message":"ok","data":null}`
421 421
   * `{"code":4,"message":"no or wrong API Key","data":null}`
  422
+
  423
+### Pads
  424
+
  425
+#### listAllPads()
  426
+ * API >= 1.2.1
  427
+ 
  428
+lists all pads on this epl instance
  429
+
  430
+*Example returns:*
  431
+ * `{code: 0, message:"ok", data: ["testPad", "thePadsOfTheOthers"]}`
6  src/node/db/API.js
@@ -43,6 +43,12 @@ exports.listPads = groupManager.listPads;
43 43
 exports.createGroupPad = groupManager.createGroupPad;
44 44
 
45 45
 /**********************/
  46
+/**PADLIST FUNCTION****/
  47
+/**********************/
  48
+
  49
+exports.listAllPads = padManager.getPads;
  50
+
  51
+/**********************/
46 52
 /**AUTHOR FUNCTIONS****/
47 53
 /**********************/
48 54
 
3  src/node/db/Pad.js
@@ -473,8 +473,7 @@ Pad.prototype.remove = function remove(callback) {
473 473
     //delete the pad entry and delete pad from padManager
474 474
     function(callback)
475 475
     {
476  
-      db.remove("pad:"+padID);
477  
-      padManager.unloadPad(padID);
  476
+      padManager.removePad(padID);
478 477
       hooks.callAll("padRemove", {'padID':padID});
479 478
       callback();
480 479
     }
69  src/node/db/PadManager.js
@@ -34,10 +34,59 @@ var db = require("./DB").db;
34 34
  */
35 35
 var globalPads = {
36 36
     get: function (name) { return this[':'+name]; },
37  
-    set: function (name, value) { this[':'+name] = value; },
  37
+    set: function (name, value) 
  38
+    {
  39
+      this[':'+name] = value;
  40
+      padList.addPad(name);
  41
+    },
38 42
     remove: function (name) { delete this[':'+name]; }
39 43
 };
40 44
 
  45
+var padList = {
  46
+  list: [],
  47
+  sorted : false,
  48
+  init: function()
  49
+  {
  50
+    db.findKeys("pad:*", "*:*:*", function(err, dbData)
  51
+    {
  52
+      if(ERR(err)) return;
  53
+      if(dbData != null){
  54
+        dbData.forEach(function(val){
  55
+          padList.addPad(val.replace(/pad:/,""),false);
  56
+        });
  57
+      }
  58
+    });
  59
+    return this;
  60
+  },
  61
+  /**
  62
+   * Returns all pads in alphabetical order as array.
  63
+   */
  64
+  getPads: function(){
  65
+    if(!this.sorted){
  66
+      this.list=this.list.sort();
  67
+      this.sorted=true;
  68
+    }
  69
+    return this.list;
  70
+  },
  71
+  addPad: function(name)
  72
+  {
  73
+    if(this.list.indexOf(name) == -1){
  74
+      this.list.push(name);
  75
+      this.sorted=false;
  76
+    }
  77
+  },
  78
+  removePad: function(name)
  79
+  {
  80
+    var index=this.list.indexOf(name);
  81
+    if(index>-1){
  82
+      this.list.splice(index,1);
  83
+      this.sorted=false;
  84
+    }
  85
+  }
  86
+};
  87
+//initialises the allknowing data structure
  88
+padList.init();
  89
+
41 90
 /**
42 91
  * An array of padId transformations. These represent changes in pad name policy over
43 92
  * time, and allow us to "play back" these changes so legacy padIds can be found.
@@ -109,6 +158,15 @@ exports.getPad = function(id, text, callback)
109 158
   }
110 159
 }
111 160
 
  161
+exports.getPads = function(callback)
  162
+{
  163
+  if(callback != null){
  164
+    callback(null,padList.getPads());
  165
+  }else{
  166
+    return padList.getPads();
  167
+  }
  168
+}
  169
+
112 170
 //checks if a pad exists
113 171
 exports.doesPadExists = function(padId, callback)
114 172
 {
@@ -163,6 +221,15 @@ exports.isValidPadId = function(padId)
163 221
   return /^(g.[a-zA-Z0-9]{16}\$)?[^$]{1,50}$/.test(padId);
164 222
 }
165 223
 
  224
+/**
  225
+ * Removes the pad from database and unloads it.
  226
+ */
  227
+exports.removePad = function(padId){
  228
+  db.remove("pad:"+padId);
  229
+  exports.unloadPad(padId);
  230
+  padList.removePad(padId);
  231
+}
  232
+
166 233
 //removes a pad from the array
167 234
 exports.unloadPad = function(padId)
168 235
 {
36  src/node/handler/APIHandler.js
@@ -138,6 +138,42 @@ var version =
138 138
   , "listAllGroups"             : []
139 139
   , "checkToken"                : []
140 140
   }
  141
+, "1.2.1":
  142
+  { "createGroup"               : []
  143
+  , "createGroupIfNotExistsFor" : ["groupMapper"]
  144
+  , "deleteGroup"               : ["groupID"]
  145
+  , "listPads"                  : ["groupID"]
  146
+  , "listAllPads"               : []
  147
+  , "createPad"                 : ["padID", "text"]
  148
+  , "createGroupPad"            : ["groupID", "padName", "text"]
  149
+  , "createAuthor"              : ["name"]
  150
+  , "createAuthorIfNotExistsFor": ["authorMapper" , "name"]
  151
+  , "listPadsOfAuthor"          : ["authorID"]
  152
+  , "createSession"             : ["groupID", "authorID", "validUntil"]
  153
+  , "deleteSession"             : ["sessionID"]
  154
+  , "getSessionInfo"            : ["sessionID"]
  155
+  , "listSessionsOfGroup"       : ["groupID"]
  156
+  , "listSessionsOfAuthor"      : ["authorID"]
  157
+  , "getText"                   : ["padID", "rev"]
  158
+  , "setText"                   : ["padID", "text"]
  159
+  , "getHTML"                   : ["padID", "rev"]
  160
+  , "setHTML"                   : ["padID", "html"]
  161
+  , "getRevisionsCount"         : ["padID"]
  162
+  , "getLastEdited"             : ["padID"]
  163
+  , "deletePad"                 : ["padID"]
  164
+  , "getReadOnlyID"             : ["padID"]
  165
+  , "setPublicStatus"           : ["padID", "publicStatus"]
  166
+  , "getPublicStatus"           : ["padID"]
  167
+  , "setPassword"               : ["padID", "password"]
  168
+  , "isPasswordProtected"       : ["padID"]
  169
+  , "listAuthorsOfPad"          : ["padID"]
  170
+  , "padUsersCount"             : ["padID"]
  171
+  , "getAuthorName"             : ["authorID"]
  172
+  , "padUsers"                  : ["padID"]
  173
+  , "sendClientsMessage"        : ["padID", "msg"]
  174
+  , "listAllGroups"             : []
  175
+  , "checkToken"                : []
  176
+  }
141 177
 };
142 178
 
143 179
 /**
2  src/package.json
@@ -46,5 +46,5 @@
46 46
   "engines"        : { "node" : ">=0.6.0",
47 47
                        "npm"  : ">=1.0"
48 48
                      }, 
49  
-  "version"        : "1.2.3"
  49
+  "version"        : "1.2.4"
50 50
 }
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.