Shrink generated packages

[lightmare]

There is a glaring inefficiency in serialization of gzipped data, which is addressed by the first commit in this PR. Constructing gunzip input buffer from base64-encoded string instead of an array of numbers shrinks generated unicode-13 package from ~150MB down to ~60MB.

Subsequent two commits are a bit more complicated.

The second commit compresses binary properties. It comes from the observation that many generated code-points.js and symbols.js files contain long sequences of adjacent code points. These can be run-length encoded, and the lengths (which are often short) then encoded in variable-length base64 prefix code. This shrinks generated unicode-13 package from ~60MB down to ~15.5MB.

The third commit applies run-length encoding to named properties, shrinking the generated package down to ~13.5MB.

[mathiasbynens]

Nice work!

Have you done any testing to confirm that the new generated output matches the old generated output from a user perspective? I.e. all exported arrays are still deeply equal to each other, etc.

[lightmare]

I've run the existing test, it passes. Then I took the list of filenames from that test, plus a few other than code-points.js, and instead of comparing against ./expected/*, I did compare them against @unicode/unicode-13.0.0/*. Didn't check all files, yet. So far I've found 2 directories that don't match exactly:

/Binary_Property/Composition_Exclusion/code-points.js
/Line_Break/Unknown/code-points.js

The arrays are not deepEqual, but contain the same code points. This is because the code points are not sorted in the released package, whereas here I needed to sort them for run-length encoding.

[mathiasbynens]

I've run the existing test, it passes. Then I took the list of filenames from that test, plus a few other than code-points.js, and instead of comparing against ./expected/*, I did compare them against @unicode/unicode-13.0.0/*. Didn't check all files, yet. So far I've found 2 directories that don't match exactly:

/Binary_Property/Composition_Exclusion/code-points.js
/Line_Break/Unknown/code-points.js

The arrays are not deepEqual, but contain the same code points. This is because the code points are not sorted in the released package, whereas here I needed to sort them for run-length encoding.

I'm surprised the arrays aren't already sorted. Let me take a look.

[mathiasbynens]

Independent of this patch, I think we should enforce + test sorting of code points. I was expecting them to already be sorted in the published packages.

[mathiasbynens]

Suggested change

	let codePointsExports = `require('./ranges').flatMap(r=>Array.from(r.keys()))`;
	let codePointsExports = `require('./ranges.js').flatMap(r=>Array.from(r.keys()))`;

[mathiasbynens]

Thanks!

[mathiasbynens]

Thanks so much for your work on this!

[lightmare]

Happy to have smaller dependencies ;)