Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

std.unicode fixes #3987

Merged
merged 5 commits into from Dec 28, 2019
Merged

std.unicode fixes #3987

merged 5 commits into from Dec 28, 2019

Conversation

daurnimator
Copy link
Collaborator

Bug fixes, cleaner and faster :)

@daurnimator
std: fix unicode encoding of astral plane codepoints to utf16
a81c0ba
@daurnimator
std: add warm up phase to benchmark
8b72eed 
In my tests, whatever ran first was getting much better numbers.
Additionally, add alignment requirements so that comparison is fair.
@daurnimator
std: optimise utf8ByteSequenceLength
5843a6e 
Also tested (but not as fast):
```zig
pub fn utf8ByteSequenceLength(first_byte: u8) !u3 {
    const len = @clz(u8, ~first_byte);
    if (len == 0) return 1;
    if (len < 4) return @intcast(u3, len);
    return error.Utf8InvalidStartByte;
}
```
@daurnimator daurnimator added the standard library This issue involves writing Zig code for the standard library. label Dec 28, 2019
@daurnimator daurnimator mentioned this pull request Dec 28, 2019
Closed
@data-man
Copy link
Contributor

Proposal:

pub const CodePoint = u21;

@daurnimator
Copy link
Collaborator Author

Proposal:

pub const CodePoint = u21;

I thought about that but decided I'd like to see how #3806 panned out first.

@daurnimator
std: simplify utf8ToUtf16Le
ab60654 
Also faster, on my machine unicode/throughput_test.zig now gives e.g.
> original utf8ToUtf16Le: elapsed: 1048 ns (0 ms)
> new utf8ToUtf16Le: elapsed: 971 ns (0 ms)
@daurnimator
std: unicode codepoints are 21 bits
edb5deb
@andrewrk andrewrk merged commit cb02125 into ziglang:master Dec 28, 2019
@andrewrk
Copy link
Member

Nice!

@daurnimator daurnimator deleted the std.unicode-fixes branch December 28, 2019 23:16
@daurnimator daurnimator mentioned this pull request Nov 11, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
standard library This issue involves writing Zig code for the standard library.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@daurnimator @data-man @andrewrk