Skip to content

[bug] Nokogiri::XML::Reader#inner_xml returns NCR encoded attributes even if the encoding is set to utf-8 in #from_memory call. #2891

Closed

Description

@torce

Please describe the bug

Nokogiri::XML::Reader#inner_xml returns NCR encoded attributes even if the encoding is set to utf-8 in #from_memory call.
It does not happen if the XML input sets the encoding with <?xml version="1.0" encoding="UTF-8"?>.
It only happens to attributes, elements and text nodes are correctly encoded.

Help us reproduce what you're seeing

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'nokogiri', '1.13.8'
end

require 'nokogiri'

xml = <<~XML
  <test><anotación tipo="inspiración">(inspiración)</anotación></test>
XML

reader = Nokogiri::XML::Reader.from_memory(xml, nil, 'utf-8')
puts reader.inner_xml while reader.read && reader.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT

# Output with NCR encoded attributes:
# <anotación tipo="inspiraci&#xF3;n">(inspiración)</anotación>
# (inspiración)

xml_with_encoding = <<~XML
  <?xml version="1.0" encoding="UTF-8"?>
  <test><anotación tipo="inspiración">(inspiración)</anotación></test>
XML

reader = Nokogiri::XML::Reader.from_memory(xml_with_encoding, nil, 'utf-8')
puts reader.inner_xml while reader.read && reader.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT

# Output with correct encoding:
# <anotación tipo="inspiración">(inspiración)</anotación>
# (inspiración)

Environment

# Nokogiri (1.13.8)
    ---
    warnings: []
    nokogiri:
      version: 1.13.8
      cppflags:
      - "-I/home/david/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/nokogiri-1.13.8-x86_64-linux/ext/nokogiri"
      - "-I/home/david/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/nokogiri-1.13.8-x86_64-linux/ext/nokogiri/include"
      - "-I/home/david/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/nokogiri-1.13.8-x86_64-linux/ext/nokogiri/include/libxml2"
      ldflags: []
    ruby:
      version: 3.1.2
      platform: x86_64-linux
      gem_platform: x86_64-linux
      description: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
      engine: ruby
    libxml:
      source: packaged
      precompiled: true
      patches:
      - 0001-Remove-script-macro-support.patch
      - 0002-Update-entities-to-remove-handling-of-ssi.patch
      - 0003-libxml2.la-is-in-top_builddir.patch
      - 0004-use-glibc-strlen.patch
      - 0005-avoid-isnan-isinf.patch
      - 0006-update-automake-files-for-arm64.patch
      - '0008-htmlParseComment-handle-abruptly-closed-comments.patch'
      - '0009-allow-wildcard-namespaces.patch'
      libxml2_path: "/home/david/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/nokogiri-1.13.8-x86_64-linux/ext/nokogiri"
      memory_management: ruby
      iconv_enabled: true
      compiled: 2.9.14
      loaded: 2.9.14
    libxslt:
      source: packaged
      precompiled: true
      patches:
      - 0001-update-automake-files-for-arm64.patch
      datetime_enabled: true
      compiled: 1.1.35
      loaded: 1.1.35
    other_libraries:
      zlib: 1.2.12
      libgumbo: 1.0.0-nokogiri

Additional context

Activity

  1. added
    state/needs-triageInbox for non-installation-related bug reports or help requests
    on May 24, 2023
  2. flavorjones commented on May 25, 2023

    @flavorjones
    Member

    Thanks for reporting this. The behavior of the DOM parsing methods is slightly different, which is interesting:

    #! /usr/bin/env ruby
    
    require 'bundler/inline'
    
    gemfile do
      source 'https://rubygems.org'
      gem 'nokogiri', '1.13.8'
    end
    
    require 'nokogiri'
    
    xml = <<~XML
      <test><anotación tipo="inspiración">(inspiración)</anotación></test>
    XML
    
    Nokogiri::XML::Document.parse(xml).to_xml
    # => "<?xml version=\"1.0\"?>\n" +
    #    "<test>\n" +
    #    "  <anotación tipo=\"inspiraci&#xF3;n\">(inspiraci&#xF3;n)</anotación>\n" +
    #    "</test>\n"
    
    Nokogiri::XML::Document.parse(xml, nil, "UTF-8").to_xml
    # => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
    #    "<test>\n" +
    #    "  <anotación tipo=\"inspiración\">(inspiración)</anotación>\n" +
    #    "</test>\n"

    I'll investigate!

  3. added and removed
    state/needs-triageInbox for non-installation-related bug reports or help requests
    on May 26, 2023
  4. flavorjones commented on May 30, 2023

    @flavorjones
    Member

    Sorry for not updating sooner. This appears to be intentional libxml2 behavior when a document does not declare an encoding.

    However, I have some changes on a branch to work around this behavior within Nokogiri. Just need an hour or two to explore edge cases and build confidence that it's not breaking anything else encoding-related.

  5. nwellnhof commented on Dec 27, 2023

    @nwellnhof

    The behavior of the DOM parsing methods is slightly different, which is interesting:

    The issue is in libxml2's serializer. If it wasn't told to use a specific encoding, it will use the one from the XML declaration. If there's no encoding in the XML declaration, it will encode non-ASCII characters with NCRs. (This is confusing since UTF-8 is the XML default and shouldn't make a difference. It probably comes from a time when UTF-8 wasn't as ubiquitous.) The best solution is use "UTF-8" instead of NULL when serializing documents with doc->encoding == NULL.

  6. flavorjones commented on Dec 27, 2023

    @flavorjones
    Member

    @nwellnhof Thanks for validating -- my working branch is doing exactly that: setting doc->encoding = xmlStrdup(BAD_CAST "UTF-8") if it's NULL. I'll rebase and merge it!

  7. added a commit that references this issue on Dec 28, 2023
    d4a30b8
  8. added a commit that references this issue on Dec 28, 2023
    2c97b17
  9. flavorjones commented on Dec 28, 2023

    @flavorjones
    Member

    See #3084

  10. added 4 commits that reference this issue on Dec 28, 2023
    46276ef
    045d4f4
    35c8308
    26fcb96
  11. added a commit that references this issue on Dec 29, 2023
    bb114cf
  12. removed this from the v1.16.x patch releases milestone on Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions