You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[bug] Nokogiri::XML::Reader#inner_xml returns NCR encoded attributes even if the encoding is set to utf-8 in #from_memory call. #2891
Nokogiri::XML::Reader#inner_xml returns NCR encoded attributes even if the encoding is set to utf-8 in #from_memory call.
It does not happen if the XML input sets the encoding with <?xml version="1.0" encoding="UTF-8"?>.
It only happens to attributes, elements and text nodes are correctly encoded.
Sorry for not updating sooner. This appears to be intentional libxml2 behavior when a document does not declare an encoding.
However, I have some changes on a branch to work around this behavior within Nokogiri. Just need an hour or two to explore edge cases and build confidence that it's not breaking anything else encoding-related.
The behavior of the DOM parsing methods is slightly different, which is interesting:
The issue is in libxml2's serializer. If it wasn't told to use a specific encoding, it will use the one from the XML declaration. If there's no encoding in the XML declaration, it will encode non-ASCII characters with NCRs. (This is confusing since UTF-8 is the XML default and shouldn't make a difference. It probably comes from a time when UTF-8 wasn't as ubiquitous.) The best solution is use "UTF-8" instead of NULL when serializing documents with doc->encoding == NULL.
@nwellnhof Thanks for validating -- my working branch is doing exactly that: setting doc->encoding = xmlStrdup(BAD_CAST "UTF-8") if it's NULL. I'll rebase and merge it!
Activity
flavorjones commentedon May 25, 2023
Thanks for reporting this. The behavior of the DOM parsing methods is slightly different, which is interesting:
I'll investigate!
flavorjones commentedon May 30, 2023
Sorry for not updating sooner. This appears to be intentional libxml2 behavior when a document does not declare an encoding.
However, I have some changes on a branch to work around this behavior within Nokogiri. Just need an hour or two to explore edge cases and build confidence that it's not breaking anything else encoding-related.
nwellnhof commentedon Dec 27, 2023
The issue is in libxml2's serializer. If it wasn't told to use a specific encoding, it will use the one from the XML declaration. If there's no encoding in the XML declaration, it will encode non-ASCII characters with NCRs. (This is confusing since UTF-8 is the XML default and shouldn't make a difference. It probably comes from a time when UTF-8 wasn't as ubiquitous.) The best solution is use "UTF-8" instead of NULL when serializing documents with
doc->encoding == NULL.flavorjones commentedon Dec 27, 2023
@nwellnhof Thanks for validating -- my working branch is doing exactly that: setting
doc->encoding = xmlStrdup(BAD_CAST "UTF-8")if it'sNULL. I'll rebase and merge it!fix(libxml): default Reader node encoding to UTF-8
fix(libxml): default Reader node encoding to UTF-8
flavorjones commentedon Dec 28, 2023
See #3084
fix(libxml): default Reader node encoding to UTF-8
fix(libxml): default Reader node encoding to UTF-8
fix(libxml): default Reader node encoding to UTF-8
fix(libxml): default Reader node encoding to UTF-8
fix(libxml): default Reader node encoding to UTF-8 (#3084)