Perf tests

[sindresorhus]

We need some perf tests.
It's hard to accurately measure the performance, so we'll need to discuss how to best approach this.

I've compiled a list with some existing localStorage jsPerf tests we can use as inspiration:

• localStorage key size diff
  http://jsperf.com/localstorage-read-write-test/3
• localStorage vs direct object access
  http://jsperf.com/localstorage-vs-objects/16
• localStorage vs IndexedDB
  http://jsperf.com/indexeddb-vs-localstorage/2
• localStorage getItem / setItem vs getter / setter
  http://jsperf.com/localstorage-getitem-setitem-vs-getter-setter

I've noticed localStorage in Safari 5 is about 240x faster than in Chrome 17. It seems like they're doing something right. Anyone why that is?

Not surprising, it's exponentially slower to access big data.

Also, it looks like IndexedDB is way slower than localStorage. Though the jsPerf might be biased. We should create our own to confirm.

Related

https://plus.google.com/117281036949330216542/posts/PM51anxWXSA

[peol]

Nice work!

This is tricky business and we'll probably bump into implementations where we should use one or the other, and that can't really be feature tested, which means we'll have to rely on browser sniffing in those cases. :(

[addyosmani]

Thanks for putting this together!.

I agree that this project really needs some performance testing to validate whether there are benefits to using localStorage for caching. I think there are a few different facets to benchmarking we need to consider though: there's localStorage vs alterative 'modern' storage formats and localStorage vs browser caching (which is the major thing we need to cover).

More random thoughts..

From what research I've done IDB is slower than localStorage at the moment (as partially confirmed by the above), but I wonder if some of the problem is that there's much confusion about which storage options should be used for particular use cases.

For offline, there are three problems, and three solutions :

• Application => cache manifest
• Configuration => localStorage
• Data => IndexedDB, file system (File API)

For our particular needs, maybe we should be using IDB, but the problem there is it's direly slow. Does that mean we should be looking at the File API compared to localStorage instead? Libraries like https://github.com/ebidel/filer.js could help with that, but the issue is that even if we did use the most optimal solution we'd have to implement it in a tiered way.

ie.

If ( fileAPI ) => use
else ( IndexedDB )
else ( localStorage )
etc.

That could get complicated fairly quickly. Part of me really wants to stick with localStorage for now and I think Jen's comments (lower down) here sum up why fairly well: http://hacks.mozilla.org/2012/03/there-is-no-simple-solution-for-local-storage/comment-page-1/

That said..lets continue with the perf testing front. More data to confirm a best path forward either way cant be a bad thing :)

[sindresorhus]

The problem with IndexedDB is that it requires user approval, which for me, is a no go. Who would want to annoy their users with a prompt to only save a few milliseconds?

A JS mutable cache manifest is what we really need. Where we have some APIs to add/remove files from the manifest. Maybe we should suggest something like that to the right people? Since there is clearly a need for better control of caching.

Using other storage methods to cache script is a temp hack, which does not scale in the long run. At the moment localStorage is our best bet, even though I have my doubts.

[addyosmani]

I agree with that ^. We could either bring this up with dev-rel on Moz/Opera/Chrome fronts, speak to implementers (via mailing lists) or speak to the jQuery standards team http://blog.jquery.com/2011/10/24/announcing-the-jquery-standards-team/

In terms of what we do with this project, I see us sticking with LS until something better comes around.

[sindresorhus]

I commented on the jQuery Standards issue tracker, but there seems to be little to no activity there the last few months.

[addyosmani]

You did get one or two responses :) That said, its looking like there might be better luck on the official mailing lists.

[sindresorhus]

Yes, I thought so, though Louis-Rémi has already tried, and there's an open issue on the W3 issue tracker.

Btw, discovered today that Chrome only gives you 2.5 MB localStorage, since it's saves everything in UTF-16...

Have you explored the possibility of using the FileSystem API? If you use the temporary storage, you don't have to ask the user for permission.

[peol]

I think that if we just abstract the storage implementation we have, we can easily add others by just plugging them in. 2,5mb should be sufficient for almost all JavaScript application, or you might have to reconsider your codebase. :)

[addyosmani]

I think 2.5mb is absolutely fine for most applications. I have yet to see a (properly built) web app that requires more than that for their JS payload and I think it should be okay to use.

With respect to the File/FileSystem API I mentioned this further up in the thread. The problem is that support for this is currently limited to Chrome http://caniuse.com/filesystem. We could say.. use it if supported otherwise fallback to localStorage but is that too much effort? What do you guys think?

[sindresorhus]

I don't disagree, just an observation. Might be an edge case, but Twitter is over 2 MB :P

I think we should at least include the Filsystem API in our perf tests first. It's worth the effort if there's any gain from using that API. Remember that having only Chrome support that API, still means over 30% of users would benefit from it.

[addyosmani]

I totally agree ^ :)

[sindresorhus]

From the Sencha Touch 2 announcement today:

Second, our groundbreaking new app loader caches your entire application within localStorage.

[addyosmani]

haha. groundbreaking my foot :) At least we know that others are looking at this space seriously too!

[sindresorhus]

Another article on localStorage.

His perf test also show that localStorage in Safari is way faster than any other browser, and that Chrome is way slower. Wonder why that is... If every browsers was as fast as Safari, no one would actually complain about localStorage. It looks like a lot of the implementations could be drastically improved without changing the spec.

[addyosmani]

It might be worth bringing @beverloo from Chrome into this discussion as he might be able to shed some light on this. I also have to say: I wonder just how many (or how large) the scripts people are trying to load in must be for the performance to be visibly impacted. In my own experiments I've never seen read/writes taking so long that a user would actually notice it, but I'm generally dealing with payloads of under 500KB. Do we think others are assuming the maximum that can be stored for their benchmarks?