-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplify utf8.c #102424
Simplify utf8.c #102424
Conversation
9fc22ee
to
020c8ed
Compare
cc @AaronRobinsonMSFT, re: #100451 (comment) - this mainly reduces the assembly code with cl.exe -O2 (before 1091 lines, now 369 for |
} | ||
|
||
// May have a problem if we have to flush | ||
if (ch != 0) | ||
if (sourceIndex < sourceLength && (destinationIndex > destinationLength || (destinationIndex == destinationLength && sourceIndex + 1 < sourceLength))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lambdageek, just an FYI, I had to change this condition from
if (sourceIndex < sourceLength && (destinationIndex >= destinationLength)
for mono because this case was failing:
runtime/src/libraries/System.Runtime/tests/System.Runtime.Tests/System/AttributeTests.cs
Line 169 in 2bfd1b4
Assert.NotNull(stringArg); |
IMHO, the reflection stack of mono should use the same expectation from giconv about the expected lengths as coreclr's reflection stack makes from coreclr PAL's unicode.cpp, so this standalone component doesn't need to handle more edge cases than necessary.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/cc @fanyang-mono
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test that @am11 mentioned was testing the custom attribute value string "\uDFFF". For mono, it went through minipal_convert_utf8_to_utf16
to decode this string "\xed\xbf\xbf". It seems to me that CoreCLR went through a different mechanism to get the value of the custom attribute value.
Additionally, what Mono is doing in g_utf8_to_utf16_impl
of giconv.c isn't a lot different than what is done in MultiByteToWideChar
of unicode.cpp. If I understand correctly, the only difference is that in MultiByteToWideChar
, it makes source length and destination length one byte longer than it is. Doing this in mono won't stop this check ((sourceIndex < sourceLength) && (destinationIndex >= destinationLength))
either.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fanyang-mono, coreclr ultimately uses the same implementation (couple of wrappers later), but in reflection stack (metasig.h etc.), there it compensates for those off-by-one, zero-length kind of scenarios to match the expectation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@am11 Do you mind pointing me to the CoreCLR source code where they handles off-by-one, zero-length kind of scenarios?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fanyang-mono, good question. 😅 It is a bit tricky to discern the exact place without actually attaching a conditional debugger (or some tricky way), but I was debugging the similar tests for coreclr last year #85558 (comment) which points to
RegMeta::GetTypeDefProps( |
One thing I do remember; when all PAL tests were passing and 99% of managed tests were passingm those edge cases covered in reflection tests were still failing, which is why I had to add a similar special condition in unicode.cpp -> utf8.c conversion to balance varied expectations of mono and coreclr + reflection simultaneously.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is probably fine having this a little bit longer check, because to make mono align with CoreCLR on this is non-trivial work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. I will take a look.
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
MultiByteToWideChar: 220ms per iteration |
Ran some google benchmarks: https://gist.github.com/am11/8d949f6c112aaab1575e85f2f350c59e Long length: 140 main:
PR:
// Use vectorized operations for ASCII characters
typedef uint16_t v4ui __attribute__ ((vector_size (8)));
for (; sourceIndex < sourceLength && destinationIndex < destinationLength; sourceIndex += 4, destinationIndex += 4)
{
v4ui data = *(v4ui*)&source[sourceIndex];
v4ui mask4 = (v4ui){0x007F, 0x007F, 0x007F, 0x007F};
v4ui comparison = data <= mask4;
size_t remainingLength = sourceLength - sourceIndex;
if (remainingLength > 4 && __builtin_expect(comparison[0] & comparison[1] & comparison[2] & comparison[3], 1))
{
destination[destinationIndex] = (char)data[0];
destination[destinationIndex + 1] = (char)data[1];
destination[destinationIndex + 2] = (char)data[2];
destination[destinationIndex + 3] = (char)data[3];
continue;
}
else if (remainingLength <= 4)
{
// Handle the remaining 4 or less elements if ASCII
if (remainingLength == 4) {
if (__builtin_expect(comparison[0] & comparison[1] & comparison[2] & comparison[3], 1))
{
destination[destinationIndex++] = (char)data[0];
destination[destinationIndex++] = (char)data[1];
destination[destinationIndex++] = (char)data[2];
destination[destinationIndex++] = (char)data[3];
return destinationIndex;
}
break;
}
v4ui mask3 = (v4ui){0x007F, 0x007F, 0x007F};
v4ui mask2 = (v4ui){0x007F, 0x007F};
comparison = data <= mask3;
if (remainingLength == 3) {
if( __builtin_expect(comparison[0] & comparison[1] & comparison[2], 1)) {
destination[destinationIndex++] = (char)data[0];
destination[destinationIndex++] = (char)data[1];
destination[destinationIndex++] = (char)data[2];
return destinationIndex;
}
break;
}
comparison = data <= mask2;
if (remainingLength == 2) {
if(__builtin_expect(comparison[0] & comparison[1], 1)) {
destination[destinationIndex++] = (char)data[0];
destination[destinationIndex++] = (char)data[1];
return destinationIndex;
}
break;
}
if (source[sourceIndex] < 0x80){
destination[destinationIndex++] = (char)source[sourceIndex];
return destinationIndex;
}
}
// Break out of the outer loop here because we know it's non-ASCII
break;
}
// Handle non-ASCII and mixed cases
while (sourceIndex < sourceLength && destinationIndex < destinationLength)
{
... but that did not move the needle (+/- 2ns diff). Haven't given up on fine-tuning yet, but overall I think the new implementation is simple and more readable. So it's a balancing question of readability vs. performance (I'm pursuing both 🙂). |
@am11 I'm moving this to draft for now. |
Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it. |
diet version 🤸