Skip to content

Tighter and broader VS16 handling #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
clipperhouse merged 10 commits into from
May 17, 2026
Merged

Tighter and broader VS16 handling #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b92f99d
first pass for simple case
clipperhouse May 16, 2026
80d2489
handle later VS16
clipperhouse May 16, 2026
079671b
fuzz and fix for hasEligibleVS16Pair
clipperhouse May 17, 2026
c7571d3
use IndexByte in hasEligibleVS16Pair
clipperhouse May 17, 2026
52119ba
bad parse should error
clipperhouse May 17, 2026
96122d8
rename & unused
clipperhouse May 17, 2026
a838e3d
oh, a type oversight
clipperhouse May 17, 2026
af90257
benches
clipperhouse May 17, 2026
ed69062
add fuzz to CI
clipperhouse May 17, 2026
6922f65
method instead of func
clipperhouse May 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/gofuzz.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
fuzzer: [FuzzBytesAndString, FuzzRune, FuzzTruncateStringAndBytes, FuzzControlSequences]
fuzzer: [FuzzBytesAndString, FuzzRune, FuzzTruncateStringAndBytes, FuzzControlSequences, FuzzHasEligibleVS16Pair]
steps:
- name: Check out code
uses: actions/checkout@v6
Expand Down
48 changes: 24 additions & 24 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,39 +152,39 @@ goarch: arm64
pkg: github.com/clipperhouse/displaywidth/comparison
cpu: Apple M2

BenchmarkString_Mixed/clipperhouse/displaywidth-8 6326 ns/op 266.66 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 9984 ns/op 168.97 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19602 ns/op 86.06 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/clipperhouse/displaywidth-8 6085 ns/op 277.23 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 9970 ns/op 169.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19060 ns/op 88.51 MB/s 0 B/op 0 allocs/op

BenchmarkString_EastAsian/clipperhouse/displaywidth-8 6167 ns/op 273.55 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 14022 ns/op 120.31 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19608 ns/op 86.04 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 6118 ns/op 275.76 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 13917 ns/op 121.22 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19263 ns/op 87.58 MB/s 0 B/op 0 allocs/op

BenchmarkString_ASCII/clipperhouse/displaywidth-8 60.62 ns/op 2111.60 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 122.3 ns/op 1047.01 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1490 ns/op 85.89 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/clipperhouse/displaywidth-8 54.54 ns/op 2347.10 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 125.5 ns/op 1020.32 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1478 ns/op 86.62 MB/s 0 B/op 0 allocs/op

BenchmarkString_Emoji/clipperhouse/displaywidth-8 3313 ns/op 218.51 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 5009 ns/op 144.55 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6868 ns/op 105.42 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3265 ns/op 221.74 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 5110 ns/op 141.69 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 7137 ns/op 101.44 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3430 ns/op 491.90 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 4833 ns/op 349.09 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3517 ns/op 479.72 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 4746 ns/op 355.48 MB/s 0 B/op 0 allocs/op

BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3494 ns/op 482.77 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 11724 ns/op 143.89 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3454 ns/op 488.36 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 11432 ns/op 147.56 MB/s 0 B/op 0 allocs/op

BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.0 ns/op 500.02 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 265.0 ns/op 483.01 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 255.5 ns/op 500.88 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 264.7 ns/op 483.48 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1381 ns/op 524.30 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2345 ns/op 308.70 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1320 ns/op 548.44 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2286 ns/op 316.72 MB/s 0 B/op 0 allocs/op

BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 2755 ns/op 64.24 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 4683 ns/op 37.80 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 2495 ns/op 70.94 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 4569 ns/op 38.74 MB/s 192 B/op 14 allocs/op

BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 2481 ns/op 92.30 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 5334 ns/op 42.93 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 2456 ns/op 93.25 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 5182 ns/op 44.19 MB/s 0 B/op 0 allocs/op
```

Here are some notes on [how to make Unicode things fast](https://clipperhouse.com/go-unicode/).
48 changes: 24 additions & 24 deletions comparison/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,37 +19,37 @@ goarch: arm64
pkg: github.com/clipperhouse/displaywidth/comparison
cpu: Apple M2

BenchmarkString_Mixed/clipperhouse/displaywidth-8 6326 ns/op 266.66 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 9984 ns/op 168.97 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19602 ns/op 86.06 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/clipperhouse/displaywidth-8 6085 ns/op 277.23 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 9970 ns/op 169.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19060 ns/op 88.51 MB/s 0 B/op 0 allocs/op

BenchmarkString_EastAsian/clipperhouse/displaywidth-8 6167 ns/op 273.55 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 14022 ns/op 120.31 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19608 ns/op 86.04 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 6118 ns/op 275.76 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 13917 ns/op 121.22 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19263 ns/op 87.58 MB/s 0 B/op 0 allocs/op

BenchmarkString_ASCII/clipperhouse/displaywidth-8 60.62 ns/op 2111.60 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 122.3 ns/op 1047.01 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1490 ns/op 85.89 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/clipperhouse/displaywidth-8 54.54 ns/op 2347.10 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 125.5 ns/op 1020.32 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1478 ns/op 86.62 MB/s 0 B/op 0 allocs/op

BenchmarkString_Emoji/clipperhouse/displaywidth-8 3313 ns/op 218.51 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 5009 ns/op 144.55 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6868 ns/op 105.42 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3265 ns/op 221.74 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 5110 ns/op 141.69 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 7137 ns/op 101.44 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3430 ns/op 491.90 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 4833 ns/op 349.09 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3517 ns/op 479.72 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 4746 ns/op 355.48 MB/s 0 B/op 0 allocs/op

BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3494 ns/op 482.77 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 11724 ns/op 143.89 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3454 ns/op 488.36 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 11432 ns/op 147.56 MB/s 0 B/op 0 allocs/op

BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.0 ns/op 500.02 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 265.0 ns/op 483.01 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 255.5 ns/op 500.88 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 264.7 ns/op 483.48 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1381 ns/op 524.30 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2345 ns/op 308.70 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1320 ns/op 548.44 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2286 ns/op 316.72 MB/s 0 B/op 0 allocs/op

BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 2755 ns/op 64.24 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 4683 ns/op 37.80 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 2495 ns/op 70.94 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 4569 ns/op 38.74 MB/s 192 B/op 14 allocs/op

BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 2481 ns/op 92.30 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 5334 ns/op 42.93 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 2456 ns/op 93.25 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 5182 ns/op 44.19 MB/s 0 B/op 0 allocs/op
```
74 changes: 74 additions & 0 deletions fuzz_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -443,3 +443,77 @@ func FuzzControlSequences(f *testing.F) {
}
})
}

// FuzzHasEligibleVS16Pair fuzzes byte-level VS16 detection against
// a slower UTF-8-decoding reference implementation.
func FuzzHasEligibleVS16Pair(f *testing.F) {
if testing.Short() {
f.Skip("skipping fuzz test in short mode")
}

seeds := []string{
"",
"a",
"a\uFE0F", // invalid VS16 base
"✡\uFE0F", // valid immediate VS16 pair
"👩‍❤️‍👨", // later VS16 in ZWJ sequence
"\u26F9\U0001F3FB\u200D\u2642\uFE0F", // later VS16 in skin-tone+gender sequence
"\u26F9\u0301\uFE0E\u200D\u2660\uFE0F", // non-FE0F 0xEF (FE0E) before the real FE0F
"\u26F9\uFE20\u200D\u2660\uFE0F", // non-FE0F 0xEF (FE20) before the real FE0F
"\xff\xfe\xfd", // invalid UTF-8
}
for _, s := range seeds {
f.Add([]byte(s), uint16(0))
f.Add([]byte(s), uint16(1))
f.Add([]byte(s), uint16(7))
}

f.Fuzz(func(t *testing.T, text []byte, startSeed uint16) {
// Exercise a range that includes in-bounds and out-of-bounds starts.
start := int(startSeed)
if len(text) > 0 {
start %= (len(text) + 3)
}

gotBytes := hasEligibleVS16Pair(text, start)
gotString := hasEligibleVS16Pair(string(text), start)
if gotBytes != gotString {
t.Errorf("hasEligibleVS16Pair bytes/string mismatch for %q start=%d: %v != %v", text, start, gotBytes, gotString)
}

want := hasEligibleVS16PairReference(text, start)
if gotBytes != want {
t.Errorf("hasEligibleVS16Pair(%q, start=%d) = %v, want %v", text, start, gotBytes, want)
}
})
}

func hasEligibleVS16PairReference(b []byte, start int) bool {
if len(b) == 0 || start >= len(b) {
return false
}
if start < 0 {
start = 0
}

i := 0
prevStart := -1
for i < len(b) {
r, sz := utf8.DecodeRune(b[i:])
if sz <= 0 {
break
}

if i >= start && r == '\uFE0F' && prevStart >= 0 {
p, rsz := lookup(b[prevStart:])
if rsz > 0 && prevStart+rsz == i && property(p).is(_VS16_Eligible) {
return true
}
}

prevStart = i
i += sz
}

return false
}
2 changes: 1 addition & 1 deletion graphemes.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
//
// Iterate using the Next method, and get the width of the current grapheme
// using the Width method.
type Graphemes[T ~string | []byte] struct {
type Graphemes[T ~string | ~[]byte] struct {
iter *graphemes.Iterator[T]
options Options
}
Expand Down
Loading