Dstore and positioning: tutorial outline

[mercmobily]

Hi people,

I am writing a tutorial to explain "placement" in dstore. This is aimed at people writing the server side of the story.
This is the "factual" outline of what I will explain. Can you please OK all this? I am 99.9% sure I didn't write anything stupid, but I would love to be double-checked!

I am also writing another tutorial where I explain pretty much every possible use-case scenario when using stores (cached, un-cached, with filtering on, etc.) explaining the (rare) cases where a refresh is needed. But that's a different story.

STORES AND POSITIONING

• Natural ordering is where things are placed when querying without specifying sorting

• Natural ordering should always exist for stores (although it could be switched off on server to optimise)

• An item's placement is set by

  • the beforeId parameter in put() (existing and new elements)
  • or at the end unless the store's defaultNewToStart is set to true (new elements)

• In Memory.js, the JS library has access to the store; so, when adding to Memory without beforeId, Memory checks for the store's defaultNewToStart attribute to see where to place a new element. When beforeId is set to null, then defaultNewToStart is ignored and element is added at the end.

• In Rest.js, two headers are set depending on put's options.beforeId and the store's defaultNewToStart.

  • If beforeId is set:
    • If it's !== null, X-Put-Before will be set to beforeId;
    • If it's == null, X-Put-Default-Position will be set to end.
  • If beforeId is not set:
    • If it's a new element, then X-Put-Default-Position is set to start or end (depending on the store's defaultNewToStart).

  Note that X-Put-Default-Position is used BOTH if beforeId is null (to make up for the inability to pass anything but a string via headers), AND for new elements where beforeId is not set (to tell the server where to place the element in such a case, to make up for the fact that the server doesn't know what the client store's default is).

• The server doesn't know anything about the store; for positioning, two headers are used" X-Put-Before and X-Put-Default-Position. If the server tracks natural ordering for the collection, it will

  1. Check if X-Put-Before is set. If so, will need to place the item there
  2. OR check X-Put-Default-Position -- if it's end, it will place things at the end. If it's start, it will place the item as first. If it's a new element, and neither one of the headers is set, then will place at the end (the default)

[neonstalwart]

on a side note... X- headers are deprecated (http://tools.ietf.org/html/rfc6648) this reminded me to bring this up, i think dstore should stop using them.

[kriszyp]

This looks good to me. I agree with Ben, that we should probably eliminate the X- prefix. I tend to think of "natural ordering" as being an order that is deterministic, controlled, and meaningful (when no sort order is defined), but I think that is more conceptual, and I think you basically mean the same thing since you said the server might opt out of that and use an optimization driven order.

[mercmobily]

There is the possibility that I have been misusing the term "natural ordering".

To me, "natural ordering" is when you add an item, placing it in a specific "spot" (could be at the end, at the beginning, or in the middle of a collection) and then expect it to find it exactly there when you query the collection without specifying a sorting option.

With "server opting out" I mean this: with "natural ordering", the only meaningful way to do it is by adding a hidden field to the DB table (call it position) and then, every time you add something in a specific spot (or at the end by default), you make sure that sorting by position will result in the order you requested. Again, position is completely hidden to the user who must never worry about its value. (In contrast, the Memory store doesn't need a position field because it will just place the element in the right spot in the array).

There are several ways of keeping natural ordering database-wide, but they are all quite expensive. With MySql, you end up with at least one mass-update per insert/update, to "make space" for example.

So, the server might end up ignoring completely "natural ordering": if so, you can use beforeId, defaultNewToStart, and whatever else you want: if you then ask for a query without specifying any sorting, the collection's order will depend on the database server's mood (basically: very unpredictable).

Makes sense? Should I use a different term rather than "natural ordering"? (Which also means "ordering thing alphabetically so that it makes sense to humans")

[kriszyp]

That is exactly what I had in mind with the term, sounds good.

[mercmobily]

For the tutorial, does it make sense for me to post it here once it's written so that you guys can comment on it etc.? Or shall we close this issue and shall I email it to somebody when I am done? If so, who should I email it to?

[kriszyp]

For now, I think you could submit this tutorial in a branch/PR under docs. Thank you for putting this together!

[mercmobily]

Hi,

OK will do!
The tutorial will be a "beefed up" version of this document, with a nice
introduction etc.

Merc.

On 29 August 2014 04:31, Kris Zyp notifications@github.com wrote:

For now, I think you could submit this tutorial in a branch/PR under docs.
Thank you for putting this together!

—
Reply to this email directly or view it on GitHub
#42 (comment).