Strings in MoarVM
MoarVM implements strings using NFG (Normalization Form Grapheme).
This is an extension on Unicode NFC (Normalization Form Canonical).
Strings have either a 32-bit signed or 8-bit signed fixed width representation, with negative numbers used to represent graphemes which contain more than one codepoint per grapheme (Synthetic graphemes)
||Remember, all input text is normalized by default.|
Strings are represented by the
(source). A string’s length is stored as
a 32-bit unsigned integer, so the maximum number of graphemes allowed in a
string is 32³² - 1 (4,294,967,295).
For a given string
string→body.storage_type can be one of the following types:
Can contain any Synthetic
Can contain the CRLF Synthetic
References other strings
Created by: concatenation, substring ops & string repeat op
Strands are a type of
MVMString which instead of being a flat string with
contiguous data, actually contains references to other strings. Strands are
created during concatenation or substring operations. When two flat strings are
concatenated together, a Strand with references to both string a and string b is
created. If string a and string b were strands themselves, the references of
string a and references of string b are copied one after another into the
Graphemes are segmented (which codepoints are apart of which graphemes) follow Unicode’s Text Segmentation rules for Grapheme Clusters Technical Report 29 [TR29].
Synthetics are graphemes which contain multiple codepoints. In MoarVM these are stored and accessed using a trie, while the actual data itself stores the base character seprately and then the combiners are stored in an array. We also store whether or not it is a UTF8-C8 synthetic. The struct’s source is in src/strings/nfg.h.
||Currently the maximum number of combiners in a synthetic is 1024. MoarVM will throw an exception if you attempt to create a grapheme with more than 1024 codepoints in it. (source)|
Synthetic’s codepoints are stored in a single array, with the base character pointed to by storing the location of its index in the array. The reason for this is for compatibility with Prepend characters.
Before Unicode 9.0, base characters were always the first codepoint in the grapheme.
Prepend property was added in Unicode 9.0, which does the opposite of the
Extend property. Codepoints with the
Prepend property combine with the
codepoint which comes immediately afterward. MoarVM supports both segmentation,
as well as getting the base codepoint out of a synthetic that starts with one or
more Prepend codepoint(s).
MoarVM normalizes into NFG form all input text. This can cause the data to change as normalization takes place. Developers and users may be used to systems which treat strings as "bags of bytes" and do not ensure they are valid Unicode (or any other encoding for that matter). MoarVM goes beyond ensuring correct Unicode and also ensures correct normalization in NFC form.
The C type used to represent strings
Normalization Form Grapheme. Similar to NFC except graphemes which contain multiple codepoints are stored in Synthetic graphemes.
Normalization Form Canonical
Short for Grapheme Cluster. See [TR29] for more information.
In MoarVM, a special representative to store a grapheme containing more than one codepoint using the same space as a standard codepoint. Internally stored using negative numbers in the C string data array.
[TR29] Unicode Technical Report 29. Unicode Text Segmentation. http://unicode.org/reports/tr29/