Handle Binary Data

[janl]

hoodie.js should provide a nice API to handle binary data.

One possible implementation route is using CouchDB/PouchDB attachments.

Another alternative is storing binary data base64 encoded, but that is not very useful beyond a few single digit megabytes of binary data.

The hoodie.js API could implement clever handling of selective storage of binary data. E.g. in a photo management app, RAW photo data should probably always be referenced from the server and then just not be available offline, while photo previews could be made available using PouchDB attachments or some such.

[matthew-andrews]

Sorry for jumping in on this one but we have solved* a very similar problem.

We do this by base64 encoding the binary data and then storing in WebSQL / IndexedDB. As those client side data layers support upwards of 50mb we can store quite a lot of images (although @lucas42 has done it with audio too).

To squeeze the most out of the browser we 'compress' the base 64 encoded data using a technique my colleague describes here:- http://labs.ft.com/2012/06/text-re-encoding-for-optimising-storage-capacity-in-the-browser/ (basically as we know the data is base 64 encoded storing it as plain text in UTF16 - which supports many more than 64 character is a bit of a waste - so we squeeze every 2-and-a-bit characters into a single UTF16 character).**

Here's a helper module (just made it public): https://github.com/ftlabs/ftdatasquasher. (We do the 'compression' on the server in PHP 😢 so the client side compression code - whilst functional - isn't optimised)

To be able to store bigger files in the browser (e.g. for this app: https://chrome.google.com/webstore/detail/the-economist/gebjgjhbjedcomcajgpodjgfjgkepgpl they have spoken versions of every single article) we need even more space and we're a bit stuck as to how we can get it...

* Well, found a working hack.
** Isn't this bad for performance? Yes. No. Kinda. To make this less performance degrading I've wanted to make some time to turn the job of decompressing into a WebWorker. Need to find some time to do this...

[gr2m]

wow @matthew-andrews that sounds pretty sophisticated! I'm very open to hack together on this! I'll be traveling until Nov 20th, but would love to hack on this afterwards. You interested to pair on this?

[matthew-andrews]

In principle yes - but would love to learn more about the direction of the project first if that's ok? Can I email? I'm matt@mattandre.ws.

[gr2m]

sure, you can email me at gregor@thehoodiefirm.com.

I'd aim to make a Hoodie plugin out of your work and see how it goes. Then eventually we can incorporate it into Hoodie core if it makes sense

[janl]

Matt, in addition, feel free to open issues with questions, as general as they may be, so we can log them all for the public’s benefit :)

On 07 Nov 2013, at 00:10 , Matt Andrews notifications@github.com wrote:

In principle yes - but would love to learn more about the direction of the project first if that's ok? Can I email? I'm matt@mattandre.ws.

—
Reply to this email directly or view it on GitHub.

[janl]

PouchDB migration is well in progress, closing.