Low-level Unicode code-point-based functions

[lionel-rowe]

Is your feature request related to a problem? Please describe.

For reasons of interoperability etc. it's often useful to know how long a string is, compare string length, collate strings, or index into a string based on its code points.

The built-in JS functionality for this is instead based on UTF-16 code units, which give the same results for many but not all strings.

Meanwhile, it's trivial to split a string into its constituent code points using Array.from or similar, but doing so generally isn't very efficient compared to more optimized alternatives.

Describe the solution you'd like

Low-level, well optimized, but fully Unicode code point aware functions such as:

• String length
• String length comparison¹
• Indexing by code point²
• Two-way conversion between code-point index and code-unit index²
• String comparison by code point (stage-1 proposal)
• Maybe others?

Describe alternatives you've considered

N/A

Footnotes

1. String length comparison vs a limit can often be calculated more efficiently than fully calculating the actual string length and comparing it to the limit. For example, for limit=1000, text.length == 2001 definitely exceeds it, while text.length == 999 is definitely lower (whereas text.length in the range 1000..2000 needs to be checked more carefully). ↩

2. Not sure how useful these are actually. Indexing is usually best done in code-units for compatibility with other JS in-built functions. However, there could still be interoperability use cases for them? ↩ ↩²

[BlackAsLight]

When talking about UTF8 string lengths in JavaScript, I think this issue is quite relevant in why it wasn't or at least hasn't been implemented natively.

whatwg/encoding#333

[lionel-rowe]

When talking about UTF8 string lengths in JavaScript, I think this issue is quite relevant in why it wasn't or at least hasn't been implemented natively.

whatwg/encoding#333

UTF-8 byte length could be useful too, but it's different from Unicode code point length. There's a good article called It’s Not Wrong that "🤦🏼‍♂️".length == 7 that goes into all the various length metrics that can be applied to a string, namely:

• Code unit length is 7, i.e. it takes up 7×2=14 bytes in UTF-16

  // ['\uD83E', '\uDD26', '\uD83C', '\uDFFC', '\U200D', '♂', '\UFE0F']
  '🤦🏼‍♂️'.split('')

• UTF-8 byte length is 17

  // Uint8Array(17) [240, 159, 164, 166, 240, 159, 143, 188, 226, 128, 141, 226, 153, 130, 239, 184, 143]
  new TextEncoder().encode("🤦🏼‍♂️")

• Unicode length is 5, i.e. it contains 5 separate code points/Unicode scalar values

  // ['🤦', '\U{1F3FC}', '\U200D', '♂', '\UFE0F']
  [...'🤦🏼‍♂️']

• Grapheme length is 1, i.e. all constituent chars are composed into a single grapheme

  // [{ segment: '🤦🏼‍♂️', index: 0, input: '🤦🏼‍♂️' }]
  [...new Intl.Segmenter('und', { granularity: 'grapheme' }).segment('🤦🏼‍♂️')]

• The poorly-named "East Asian Width" is 5, i.e. it takes up 5 columns in a TTY

  import { unicodeWidth } from 'jsr:@std/cli/unicode-width'
  import { assertEquals } from 'jsr:@std/assert'
  const str = '🤦🏼‍♂️'
  // 5
  const width = unicodeWidth(str)
  // [ 2, 2, 0, 1, 0 ]
  const widths = [...str].map(unicodeWidth)
  // true for any value of `str`, even if non-TTY layouts compose constituent chars into graphemes
  assertEquals(width, widths.reduce((a, b) => a + b, 0))

The article is very detailed but also very opinionated and claims code point length is essentially useless, which I obviously disagree with 😆