Ability to get many items (aka. getItems())

[peterbe]

After doing some performance measurements on my app I notice I suffer a lot from doing this:

localForage.getItem('all_ids', function(ids) {
    _.each(ids, function(id) {
        localForage.get(id, function(value) {
            ...
        });
    })
})

This is causing a lot of excess work for the browser if the list is long (e.g. 1,000+).

I know the localStorage spec doesn't have support getItems() but in most cases IndexDB is supported and it appears to have low level support for getting multiple keys which could hopefully reduce the overhead in total.

For the localStorage fallback, this would be trivial to wrap up with a for loop of kinds.

[peterbe]

For what it's worth, rather cryptically, memcache's get() takes multiple keys.
http://code.google.com/p/memcached/wiki/NewCommands#get

[kyoshino]

With IndexedDB, the undocumented IDBObjectStore.mozGetAll method (+ filtering) is much faster than iterating each items. I have used it in my app. This method is currently Gecko-specific but will be standardized and unprefixed.

[fwenzel]

fwiw, websql obviously also supports bulk get.

[peterbe]

@kyoshino thanks a bunch! I didn't know. I don't like the "moz" part of that but I'm guessing that can be solved with just being careful and falling back on something else if it's not available.

[designbyadrian]

Maybe it's a bit unrelated, or overkill for localForage, but it would be nice if LF supported queries like WebSQL and IndexedDB already do.

Or would you guys recommend another aggregator if I want queries?

[tofumatt]

Translating actual SQL queries to something that would work in IndexedDB or localStorage sounds like it would be a pain, and not really the point of this library. I like the simple localStorage API, but I'm happy to extend it with things like getItems().

I think the queries would be overkill.

[peterbe]

"Translating actual SQL queries"?? No, all I want is something like localForage.getItems(some array of keys, function(array of responses).
It would be moderately trivial to do my own fake wrapper but ultimately what I want is just one query to the database instead of one per every key I want to look up.

[bbinckes]

How would getitems work? Currently using LF and I have a key that is USERID:ORDER and storing a ORDER object. I want to pull all order objects for a certain USERID so something like getitems( USERID+'%', function()... would work good for my use case.

[magalhas]

It would be even more awesome to support MongoDB like queries.

[dwhogg]

Is it possible to get a list of all keys currently stored? Something like keys(callback) implemented in lawnchair. Would be useful as a way of doing some housekeeping if the max size of the store was being reached.

[tofumatt]

There's nothing like that yet, but I suppose it's also not a bad API to include. Did you want to take a crack at it? If you'd like to see it please file a separate issue (or send a pull request if you want to code it).

[dwhogg]

For now I've created Issue 136. Thanks all for localForage!

[tofumatt]

@thgreasi wrote in #291 an API I like for these multi-item calls:

How about:

Promise.all([
    localforage.getItem('item1'),
    localforage.getItem('item2')
]).then(function(results) {
    console.log(results);
});

// OR

var items = ['item1', 'item2', 'item3'];
var promises  = items.map(function(item) { return localforage.getItem(item); });
Promise.all(promises).then(function(results) {
    console.log(results);
});

If getItems() / setItems() are going to be implemented by localforage at some point, then something like the 2nd example might actually have to be used by some drivers.

[thgreasi]

If something like this was to be implemented, how the return object should look like?

• Array of KeyValues

[{
  key:'test1',
  value: 'value1'
}, {
  key:'test2',
  value: 'value2'
}]

• or just an object like

{
  'test1': 'value1',
  'test2': 'value2'
}

???
What's most preferable of the two in your opinion?
What are the pros and cons in each case in your opinion?

For me:
+1 for obj result

obj pros:

• It is more straightforward/efficient in retrieving a specific value.

array pros:

• Someone might argue that the array is more convenient for iterating over the results.

[tofumatt]

To iterate over the results they can always use Object.keys, I think the second (object) one is nicer as it’s a basic return object that can be extended but is easy to grab items out of. Using the array the syntax is more annoying if you know exactly what you want. It would also mean we’d either need to guarantee order or you’d have to iterate through to get a key.

Object for sure!

-tofumatt

On 10 May 2015 at 10:24:45, Thodoris Greasidis (notifications@github.com) wrote:

If something like this was to be implemented, how the return object should look like?

Array of KeyValues
[{
key:'test1',
value: 'value1'
}, {
key:'test2',
value: 'value2'
}]
or just an object like
{
'test1': 'value1',
'test2': 'value2'
}
???
What's most preferable of the two in your opinion?
What are the pros and cons in each case in your opinion?

For me:
+1 for obj result
obj pros:

It is more straightforward/efficient in retrieving a specific value. array pros
Someone might argue that the array is more convenient for iterating over the results.
—
Reply to this email directly or view it on GitHub.

[thgreasi]

@tofumatt I was just wandering if anyone could report any edge case or strange behavior when setting any special/reserved key as a property to an object.

PS: I also like for..in with Object.hasOwnProperty to support pre-es5 browsers. Hope it has the same results.

[tofumatt]

I can’t think of too many and I’d think it would crop up before saving to localForage though… it would be a problem in the data BEFORE we did a get/set. I think ^_^

-tofumatt

On 10 May 2015 at 11:27:24, Thodoris Greasidis (notifications@github.com) wrote:

@tofumatt I was just wandering if anyone could report any edge case or strange behavior when setting any special/reserved key as a property to an object.

PS: I also like for..in with Object.hasOwnProperty to support pre-es5 browsers. Hope it has the same results.

—
Reply to this email directly or view it on GitHub.

[kyoshino]

FYI:

Chrome is about to implement IDBObjectStore.getAll.
https://code.google.com/p/chromium/issues/detail?id=457450

Firefox has already implemented the unprefixed getAll method, but it's still behind a pref. mozGetAll is not documented on MDN but available by default.
https://bugzilla.mozilla.org/show_bug.cgi?id=920633

[tofumatt]

Oh, that is awesome!

[thgreasi]

Can you give a try to thgreasi/localForage-getItems ?
I thought it would be better for developing and reviewing if it was written (initially, until merged) as an extension library for localforage.

The jsperf testcases (found in README) gave me mixed results.
For big enough number of requested keys the gains are pretty good.
For small number of requested keys, chrome does a pretty god job parallelizing the independent read-only requests. As a result the generic method is sometimes faster.

The generic method is based on the code described above and it should be the default implementation for any driver that can't offer a better (driver specific) implementation (eg localstorage).

[thgreasi]

This is probably not the best place to discus it, but I have also been experimenting with a startsWith implementation.

[siddo420]

whats the status of this feature?

[tofumatt]

It's not available yet, but we'll update the issue when it is. I'm not sure anyone is working on it right now.

[thgreasi]

You can use the aforementioned plugin if you need it now.

[thgreasi]

@tofumatt The plugin itself was structured in a way that would ease the creation of a PR. I could open one after the holidays if we want something like this as part of the main library. In order to reduce duplication, we could wire up the existing getItem method to use the composite version with a single item as a parameter.

On the other hand setItems gave me much better performance gains than getItems.
It seems that doing concurrent getItem requests and composing them with Promise.all() works faster than fetching the items in a single transaction when there is no one else writing on the db at the same time. See next comment.

[tofumatt]

I do like the plugin way, I’m starting to feel like we can strike a balance on library size and features. For now I’d recommend using the plugin and I'll think about whether we want to merge this in directly.

[thgreasi]

👍 on the plugins... that way we can push new experimental features
As a side note & correction on my earlier performance related comments,
I just had a rerun of the jsperf benchmarks and it seems that the current versions of Firefox (v44) and Chrome (v47) give better result on getItems() than multiple generic concurent getItem() calls.

[tamer1an]

Yes, it would be convenient.
Please consider adding that feature.