v1.3.0 - Large input sanitization

https://luarocks.org/modules/leafo/web_sanitize

This update includes a fix for the stack overflow (too many captures) error produced by LPeg when parsing too large of an input.

The structure of the parser has been modified slightly when handing inputs that are over 10kb.

Changes

• Add separate pattern for sanitizing HTML inputs that are over 10kb
• Add an assertion when parsing query selectors with the scanning library

Summary of stack overflow (too many captures) fix

Previously, the parser essentially boiled down to: html = Ct (open_tag + close_tag + html_entity + escaped_html_char + text)^0 * -1. Each thing that can be parsed (open_tag, etc.) is captured into a chunk of text that is placed into a table with Ct.

As Ct appends each parsed item into an array in a linear format, you would expect it to work for any size input, but it seems even a non-nested setup like this causes the stack overflow error to happen within LPeg. Each capture is left open until the entire string is finished parsing, so the capture limit is hit in larger inputs with a large number of tags.

In order to fix the error, I would have to reduce the number of active captures. My thought was to chunk the parsing reading X at a time, convert them to a string, then put that chunk into the buffer. (Ideally the chunking can happen directly as part of the LPeg pattern, and no Lua glue code is needed)

Initially I thought something like this would work:

html_chunk = open_tag + close_tag + html_entity + escaped_html_char + text

html = Ct (
  Cs(html_chunk * html_chunk^-1000)^0 -- Read up to 1000 html chunks in a loop
)

The idea being that Cs will convert up to 1000 of the html chunks to a plain string, and close out all those captures. The error still happened though, likely because the finalization of captures only happens at the very end of parsing. I remembered about Cmt which has to resolve the captures at run time to provide your callback function with arguments.

So the the final approach is to flatten a chunk of captures using Cmt as follows:

html_chunk = open_tag + close_tag + html_entity + escaped_html_char + text

flatten = (p) -> Cmt p, (s, p, c) -> true, table.concat c

html = Ct (
  flatten(Ct(html_chunk * html_chunk^-1000))^0
)

This can be a useful trick if you have to parse very large inputs with LPeg. You can chunk the parsing using Cmt to avoid hitting the stack limit. This approach only works for things you can parse linearly, so if you have nested structures you're probably out of luck. In order to parse something nested (html) linearly, you can use a separate stack that's checked against via Cmt patterns to verify if the nesting is valid.