Error: Memory allocated was shorter than necessary for multi-byte UTF-8 chars

[UnsignedInt8]

I was trying to compress a very long string, but failed. It prints Error: Memory allocated was shorter than necessary. How can I fix it?

[lovell]

Hello, are you able to share a string that fails in this manner?

[UnsignedInt8]

I was trying to compress this article (Chinese)

import shorter from 'shorter';

shorter.compress(full_article)

// Error: Memory allocated was shorter than necessary

OS: macOS 10.13, 8G Memory

[lovell]

Thank you for the example, this falls under the problem discussed in Ed-von-Schleck/shoco#11

In this case, the Chinese characters are 3 byte UTF-8 chars, which each become 6 bytes when encoded, so double their original size.

The proposal discussed in Ed-von-Schleck/shoco#11 to use the 0x02, 0x03 and 0x04 prefixes for the char length would definitely help so if you're able to submit a PR upstream with that change that would be great.

For now I'll add a comment to the README about the lack of usefulness for strings consisting heavily of multi-byte UTF-8 chars.

(This module is designed for short text strings so if you're regularly compressing >1MB then perhaps consider an alternative such as gzip.)

[UnsignedInt8]

Thanks a lot.

[lovell]

v0.3.0 now available with a fix for the problem you experienced, thanks for reporting.