Character count component counts code points, not characters

[36degrees]

The character count currently uses string.length to establish the length of the user input. string.length counts code units, not characters, and this can lead to some confusing results when using certain strings.

You can see this by trying the following strings into the character component:

String	Result
cat😹	The emoji is counted as 2 code units, and the length is reported as 5 characters.
cafȩ́	Each combining mark is counted separately, and the reported length is 6 characters.
👩🏻‍🚀	Because this emoji includes both gender and skin modifiers and a zero-width joiner, this single character is counted as 7 characters.

We should probably find a less naive way to count characters in strings, but we also need to work out how this will work with any backend validation or data storage on a service, which may already be using a different definition of a 'character' (for example, where the backend or storage treats one character as one byte).

Further reading:

• https://en.wikipedia.org/wiki/Character_encoding
• http://developer.teradata.com/blog/jasonstrimpel/2011/11/javascript-string-length-and-internationalizing-web-applications
• https://dmitripavlutin.com/what-every-javascript-developer-should-know-about-unicode/#33stringlength

[selfthinker]

Another good article on the subject: https://blog.jonnew.com/posts/poo-dot-length-equals-two

[dashouse]

@36degrees this seems like it could end up being quite a serious bug if used in a service with multiple language support.

If we can't fix it should it be documented?

[NickColley]

Seems like a robust solution is very code heavy which would not be suitable for clientside.

I think Dave's suggestion of leaving it as is but documenting how it works would be the best way forwards...

• https://github.com/orling/grapheme-splitter
• https://bundlephobia.com/result?p=grapheme-splitter@1.0.4

[simonneb]

We noticed a similar issue in the character counts on the GOV.UK Notify service when sending non-English characters to the service. It turns out that Notify was counting bytes and not characters - this was fixed by by the team.