Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

API & backend to list pads on epl instance #1342

Merged
merged 7 commits into from Jan 14, 2013

Conversation

Projects
None yet
5 participants
Contributor

spcsser commented Jan 8, 2013

Introduces the ability to list all pads currently on this instance.
The pads can be retrieved via api calling /api/1.2.1/listAllPads

How it works:

  • Startup:
    • retrieves dataset containing all known padIDs from db key pads into js object storage
  • Runtime:
    • opening pad:
      • adds padID to storage if not existing
      • saves storage to db
    • creating pad
      • adds padID to storage
      • saves storage to db
    • remove pad
      • removes padID from storage
      • saves storage to db

Limitations:

  • Pads created before this feature are not listed
Contributor

jhollinger commented Jan 8, 2013

There have been several attempts at something like this, but they all have the "Pads created before this feature are not listed" limitation. Code looks really clean, but IMHO that limitation is a deal-breaker; users will never understand. To them, any software that uses this API will seem broken.

Owner

marcelklehr commented Jan 8, 2013

That's quite true, sadly.. In fact, it's this software -- Etherpad lite -- that's broken. It's a shame, actually

Contributor

spcsser commented Jan 8, 2013

Alright, thanks for info.
From my understanding, to provide "legacy" support, I would have to introduce some sort of find functionality to ueberDB.
Do you know if there are any limitations on extending this to provide a find feature?

Contributor

jhollinger commented Jan 8, 2013

That's my understanding, too. Since it hasn't been done yet, I bet it's not easy :). I think @Pita is the author/maintainer? I would start with him.

Also, you may want to consider making pagination part of this API method. Instances could have 100's or 1000's of pads. Unless your ueberDB find method is ridiculously fast, listing them all at once won't be quick.

Owner

marcelklehr commented Jan 8, 2013

Indeed, that's one of the most broken parts of epl and it's a serious issue. You can't query the database.

But, you want to query the database. You need to query the database. And most reasonable dbs even provide this functionality natively.

Imagine for a moment, you observed that all verhicles have wheels, would you thereafter ignore all other properties of any vehicle you stumple upon, attach a rope to it and just pull it by hand? In my humble opinion, this is what ueberDB does. It limits us. So, when the user sets up some any high-end, super fast database, ueberDB prevents us from using this awesomeness and instead limits us to using stuff that only the most feature-lacking piece of software ever to be called "database" can do (so you do end up pulling your car with a rope, eventually).

Instead of using a tool that provides us with the features of the lowest common denominator, we should be using a tool that allows us to use any piece of software called "database" as if it had all the shiny, useful features, something that monkeypatches all those halfbreeds to be actually usable.

Contributor

jhollinger commented Jan 8, 2013

+1 @marcelklehr. So epl should be using a more traditional db abstraction layer. Is it practical to switch while providing a painless migration path for existing installations? I have less free time these days, but I would be willing to contribute to such an effort.

Owner

JohnMcLear commented Jan 8, 2013

fyi I think I have admin on UeberDB too so that kinda makes me a maintainer but im not so much interested in writing query functionality

Contributor

spcsser commented Jan 9, 2013

Ok, implemented the basic functionality for some dbs (dirty, mysql and mongo).
https://github.com/spcsser/ueberDB/tree/feature/findKeys
How it works is in the wiki: https://github.com/spcsser/ueberDB/wiki/findKeys-function

For my desired feature - to list all pads - this would suffice. (Having only to call it once on startup and work later with live data.) For other use, especially heavy searching this might not perfom, since it has to rely on regex to retrieve data - at least for some dbs.

+1 @marcelklehr For the future - especially to provide better search functionality - the use of some dbal to gain native db functionality back would be a good thing.

@spcsser spcsser referenced this pull request in Pita/ueberDB Jan 9, 2013

Merged

findKeys functionality #18

Owner

JohnMcLear commented Jan 9, 2013

Any tests included?

-----Original Message-----

From: Swen
Sent: 9 Jan 2013 19:08:08 GMT
To: ether/etherpad-lite
Cc: John McLear
Subject: Re: [etherpad-lite] API & backend to list pads on epl instance (#1342)

Ok, implemented the basic functionality for some dbs (dirty, mysql and mongo).
https://github.com/spcsser/ueberDB/tree/feature/findKeys
How it works is in the wiki: https://github.com/spcsser/ueberDB/wiki/findKeys-function

For my desired feature - to list all pads - this would suffice. (Having only to call it once on startup and work later with live data.) For other use, especially heavy searching this might not perfom, since it has to rely on regex to retrieve data - at least for some dbs.


Reply to this email directly or view it on GitHub:
#1342 (comment)

Contributor

spcsser commented Jan 9, 2013

Yepp, the test file is called findKeysTest.js. https://github.com/spcsser/ueberDB/blob/feature/findKeys/findKeysTest.js

Contributor

Wikinaut commented Jan 10, 2013

@spcsser you might be interested in fast fuzzy text searching. Then search for agrep.

Owner

marcelklehr commented Jan 10, 2013

You might also be interested in the thoughts I wrote down over here: #697 (comment)

Owner

JohnMcLear commented Jan 10, 2013

UeberDB stuff merged, waiting on Peter to give me ability to npm publish ;\

Contributor

spcsser commented Jan 11, 2013

While we're waiting: I created a lil' epl addon that lists the pads - https://github.com/spcsser/ep_adminpads.

Owner

JohnMcLear commented Jan 12, 2013

Just spotted, getPads is listed in the APIHandler file for 1.2.1 when it wont be available then?

Contributor

spcsser commented Jan 12, 2013

I'm not sure what you mean, so I'm trying to guess: If I'm not mistaken, getPads is to retrieve all pads for a certain groupID. The api function to list all pads I called getAllPads.
Is that what you meant?

Owner

JohnMcLear commented Jan 12, 2013

My understanding is that

+#### listAllPads()
     426    
+ * API >= 1.2.1

needs to be changed to
+#### listAllPads()
426

  • * API >= 1.2.5

as it wont be in until 1.2.5

@marcelklehr knows more about this convention

@spcsser spcsser referenced this pull request in spcsser/ep_adminpads Jan 12, 2013

Closed

Please publish to npm #1

Owner

marcelklehr commented Jan 12, 2013

yea.. about that: Those annotations about which version an endpoint was
introduced with, they refer to the API version, not epl's release version...

Contributor

spcsser commented Jan 12, 2013

So, API version == EPL version? I always thought they'd be kept separated.

Edit: @marcelklehr Thanks for clarifying :)

Owner

marcelklehr commented Jan 12, 2013

no, API version != epl version.

Owner

JohnMcLear commented Jan 12, 2013

Then once Pita ads me as on Ueber and I push we can merge this :)

Owner

JohnMcLear commented Jan 13, 2013

https://npmjs.org/package/ueberDB
I published 0.1.9 just waiting on npmjs.org web ui to catch up.
In the mean time let's have a smoke and a pancake.

JohnMcLear added a commit that referenced this pull request Jan 14, 2013

Merge pull request #1342 from spcsser/feature/padlisting
API & backend to list pads on epl instance

@JohnMcLear JohnMcLear merged commit 654654b into ether:develop Jan 14, 2013

1 check failed

default The Travis build failed
Details
@ghost

ghost commented Jan 22, 2013

For me, using mysql the padList is not initialized (or remains empty) on epl start. Once opend pads are listed though.

Log says:
[2013-01-22 22:05:46.009] [DEBUG] ueberDB - GET - pad:-:: - [] - from database

Contributor

spcsser commented Jan 22, 2013

This pull request should fix that: Pita/ueberDB#26.

@ghost

ghost commented Jan 23, 2013

With an updates ueberDB it works. thx

This was referenced Mar 11, 2013

Owner

JohnMcLear commented Jun 30, 2013

Ta @marcelklehr @spcsser can you investigate please?

Owner

JohnMcLear commented on 1987542 Oct 9, 2013

@spcsser why does this need to run on startup?

Owner

JohnMcLear replied Oct 9, 2013

FYI this is really hurting startup performance on large deployments

Contributor

spcsser replied Oct 9, 2013

No other way to ensure to have all pads and enable search & stuff. Querying db was not possible afaik.
Could be run async or delayed somehow.

Contributor

spcsser replied Oct 9, 2013

seems ok to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment