-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deserializer doesn't support unicode identifiers #321
Comments
The bug is not that it cannot deserialize unicode identifiers, but that it doesn't use raw identifiers when serializing identifiers. EDIT: So far we don't support unicode in identifiers. I'll change this issue to be specific to unicode support; I've created one for incorrect serialization of raw idents: #322 |
All parsing operates on I see 3 possible implementations:
fn check_ident_other_char(&self, index: usize) -> bool {
// Safe because `index` always points at a valid char boundry
unsafe { from_utf8_unchecked(&self.bytes[index..]) }
.chars()
.next()
.map_or(false, is_xid_continue)
}
get_next_char(&self.bytes[index..]).map_or(false, is_xid_continue)
|
@ModProg Thank you for sharing your thoughts! I've been thinking about the UTF-8 support for a while too but haven't had the time to draft an implementation so far. When we serialise Option 2 has the appeal of still giving us the flexibility of non-UTF-8 documents but at the cost of extra maintenance on our side. I think I'd still prefer option (1) here unless there is a very simple implementation or small and well-supported crate that we could depend on. I have also thought about option (3) but the potential |
Well we actually cannot, every byte we parse, must be either inside a string and therefore valid utf-8, or outside a string and therefore ascii (subset of utf-8). Introducing the restriction of utf-8 for the whole file therefore does not change anything currently, it would only lock us out of the possibility to introduce some raw byte data later. But in case we really want to do that, we can still change our approach then. |
Do we have any benchmarks? To measure performance degradation? |
Oh ignore me, found them immediately after commenting: https://github.com/ron-rs/ron-bench |
Is this a bug of ron?
The text was updated successfully, but these errors were encountered: