v1.8.0 - Linear time processing of untrusted input

This is a recommended update if you use the HTML scanner (scan_html, replace_html, query_all) on untrusted input. The scanner could silently stop partway through input containing two < in a row or a < at the end, so elements after that point were never visited and replace_html returned the rest of the input unedited. A callback that removes elements, like images, could be bypassed this way.

luarocks install web_sanitize

https://luarocks.org/modules/leafo/web_sanitize

The HTML sanitizer and scanner now run in linear time on malformed and adversarial input, and several bugs that untrusted input could trigger are fixed. Editing nodes with replace_html is also stricter: some callbacks that relied on undefined behavior will now raise errors. See the new Processing untrusted input section of the README for the remaining costs to be aware of, and the replace_html docs for how edits are applied.

Changes

• Fix the scanner silently stopping at two < in a row, or a < at the end of the input. Elements after that point got no callbacks in scan_html, replace_html and query_all, and replace_html returned the rest of the input unedited
• Fix quadratic processing time for:
  • Many edits in one replace_html call
  • Deeply nested tags followed by closing tags that aren't open, in the sanitizer and the scanner
  • Deeply nested tags with optional closing tags, like thead, in the scanner
  • Selectors like a * checked on every node of a deeply nested document
  • Unterminated comments in extract_text and the sanitizer's strip_comments option, and unterminated CDATA sections in the scanner
• Fix a LuaJIT crash in sanitize_html on some inputs (seen with upstream LuaJIT 2.1, not with OpenResty's LuaJIT), and a too many results to unpack error when a closing tag closes thousands of open tags
• Raw text in elements like script now ends only at a complete closing tag for the element. A malformed one like </script! used to end it, and many <script> tags followed by one made scanning quadratic
• Fix an error when scanning a self closing raw text tag like <script /> at the top level. Markup after a self closing raw text tag is now scanned instead of being read as raw text
• Fix an error in update_attributes on elements with a valueless attribute like disabled, and in replace_attributes with a {"name"} tuple
• Fix attributes from one call being passed to add_attributes functions in the next call when a Sanitizer is reused
• Add node:unwrap() for removing an element's tags while keeping its content and the edits made to its children
• Fix replace_html losing or wrongly keeping edits to an empty element whose closing tag is left out, like an li closed by the next li, and replacing an empty element's content twice keeping both replacements
• Edit methods now only exist on nodes in replace_html callbacks. On scan_html nodes they silently did nothing
• Editing a node after replace_outer_html or unwrap raises an error. Previously it corrupted the output
• Replacing the HTML of an ancestor from a descendant's callback raises an "element is still open" error
• replace_html raises an error for edits it can't apply instead of producing corrupted output
• Document the cost of processing untrusted input, and how replace_html applies edits

Full Changelog: v1.7.0...v1.8.0