Skip to content
Switch branches/tags
Go to file
Cannot retrieve contributors at this time
115 lines (90 sloc) 4.55 KB

Strings in MoarVM


MoarVM implements strings using NFG (Normalization Form Grapheme).

This is an extension on Unicode NFC (Normalization Form Canonical).

Strings have either a 32-bit signed or 8-bit signed fixed width representation, with negative numbers used to represent graphemes which contain more than one codepoint per grapheme (Synthetic graphemes)

Remember, all input text is normalized by default.


Strings are represented by the MVMString struct (source). A string’s length is stored as a 32-bit unsigned integer, so the maximum number of graphemes allowed in a string is 2³² - 1 (4,294,967,295).

For a given string MVMString *string, string→body.storage_type can be one of the following types:

MVMString types:




Type Storage Notes


32-bit signed

Can contain any Synthetic


8-bit signed

Can contain the CRLF Synthetic


8-bit signed


References other strings

Created by: concatenation, substring ops & string repeat op


Strands are a type of MVMString which instead of being a flat string with contiguous data, actually contains references to other strings. Strands are created during concatenation or substring operations. When two flat strings are concatenated together, a Strand with references to both string a and string b is created. If string a and string b were strands themselves, the references of string a and references of string b are copied one after another into the Strand.

Grapheme Segmentation

Graphemes are segmented (which codepoints are apart of which graphemes) follow Unicode’s Text Segmentation rules for Grapheme Clusters Technical Report 29 [TR29].


Synthetics are graphemes which contain multiple codepoints. In MoarVM these are stored and accessed using a trie, while the actual data itself stores the base character seprately and then the combiners are stored in an array. We also store whether or not it is a UTF8-C8 synthetic. The struct’s source is in src/strings/nfg.h.

Currently the maximum number of combiners in a synthetic is 1024. MoarVM will throw an exception if you attempt to create a grapheme with more than 1024 codepoints in it. (source)

Synthetic’s codepoints are stored in a single array, with the base character pointed to by storing the location of its index in the array. The reason for this is for compatibility with Prepend characters.


Before Unicode 9.0, base characters were always the first codepoint in the grapheme. The Prepend property was added in Unicode 9.0, which does the opposite of the Extend property. Codepoints with the Prepend property combine with the codepoint which comes immediately afterward. MoarVM supports both segmentation, as well as getting the base codepoint out of a synthetic that starts with one or more Prepend codepoint(s).


MoarVM normalizes into NFG form all input text. This can cause the data to change as normalization takes place. Developers and users may be used to systems which treat strings as "bags of bytes" and do not ensure they are valid Unicode (or any other encoding for that matter). MoarVM goes beyond ensuring correct Unicode and also ensures correct normalization in NFC form.



The C type used to represent strings


Normalization Form Grapheme. Similar to NFC except graphemes which contain multiple codepoints are stored in Synthetic graphemes.


Normalization Form Canonical


Short for Grapheme Cluster. See [TR29] for more information.


In MoarVM, a special representative to store a grapheme containing more than one codepoint using the same space as a standard codepoint. Internally stored using negative numbers in the C string data array.