env-web: LZString-like compression

[Ivshti]

see:

http://jsfiddle.net/yshadmehr/gzd52hh1/
https://github.com/adumbidiot/lz-string-rs
http://pieroxy.net/blog/pages/lz-string/index.html
https://github.com/pieroxy/lz-string

The point is, by using the full UTF16 character space that JavaScript supports, plus LZMA compression, we can end up with using only 10-20% of the original storage space required

Conclusion: after some research (see below), this will be put on the back burner for now and revisited before the 5.x release

[Ivshti]

this is not as easy as expected, cause we need access to UTF16 strings, but strings in Rust, including what's returned from get_item and given to set_item, is in UTF8

we could compromise and use utf8 strings, but that may create further issues

[Ivshti]

using TextEncoder to encode everything as UTF16 bytes is not possible, cause TextEncoder dropped support for utf16

The only option seems to be getting JSString and using .iter(), which may be show cause it calls char_code_at for each character

[Ivshti]

from_char_code is also out of the question cause RangeError: too many function arguments and it's 2x slower than using TextEncoder/TextDecoder; so this is kinda blocked unless we just stick to utf8

[Ivshti]

Results with UTF8:

https://github.com/Stremio/stremio-core-example/commit/3e99f89af07378dce7c0a916ad69175c757acdd0

compression 212ms 282261 104904 62.83% savings
compression 372ms 410320 141538 65.51% savings
compression 319ms 410723 141682 65.50% savings
compression 351ms 569712 178961 68.59% savings

Takes a while to compress this half a megabyte..I'm pretty sure it can be optimized ; the lz-string crate does seem pretty immature

---

in comparison, the JS version here, did:

The byte size checks out cause it's 2x cause of utf8

Time: Size:1139260 bytes
Time:290 ms Size:179006 bytes (16%)

[Ivshti]

much better results with flate2:

compression 48ms 569566 115723 79.68% savings
compression 40ms 568002 115549 79.66% savings
compression 42ms 567974 115547 79.66% savings

Although we're producing invalid utf8 so that's not a real solution

[Ivshti]

Two realistic solutions:

• Use UTF8, and find a more maintained crate than lz-string to compress/decompress
• Implement the storage in JS, use the object in Rust (env-web)

A key insight here is that we can't use invalid UTF8, cause of the transcoding process involved (TextEncoder/TextDecoder); producing actual valid UTF8 might be expensive(ish)

[Ivshti]

Another option would be to use IndexedDB, which supports binary data

[Ivshti]

actually, using IndexedDB with binary data directly from Rust might improve performance, and remove the need for compression altogether! So this issue could be replaced by implementing IndexedDB

[Ivshti]

Closing in favor of https://github.com/Stremio/stremio-core/issues/83