Improve markdown compiler performance #2661
This PR fixes issue #2660.
The first commit eliminates pathological performance when compiling a large number of markdown files at one time but preventing the markdown extensions list from growing each time a markdown file was compiled. On my machine this reduced the time required to compile 1000 markdown files from over 11 minutes to under 55 seconds.
The second commit provides a further incremental improvement by creating and reusing a single Markdown object instead of creating a separate Markdown object for each file that is compiled. This further reduces the compilation time for 1000 markdown files to under 34 seconds.
The CompileMarkdown.compile_string() method was always adding the configured markdown extensions (from MARKDOWN_EXTENSIONS) to self.extensions. This meant that each time it was called, self.extensions grew. As a result, the markdown conversion grew slower each time due to the increasing number of extensions. Fix by adding the configured extensions once in the set_site() method instead.
In the PR description, I included some timings I measured (basically more than 11 minutes before my changes, then under 55 seconds with just the first commit, and finally under 34 seconds with both commits). Are you looking for something more than that? If so, could you please describe what you would like to see? I'm happy to try to accomodate
Stop using the markdown() convenience function in the CompileMarkdown.compile_string() method, which creates a new Markdown object each time it is called. Instead, create a new class, ThreadLocalMarkdown, which is a subclass of threading.local. This class has a single attribute, markdown, which is a per-thread Markdown instance. It also has a single method, convert(), which converts the data using the Markdown instance's convert() method, and then resets the Markdown instance's internal state. Then in the CompileMarkdown.set_site() method, instantiate a new ThreadLocalMarkdown instance (unless the markdown module is unavailable) and assign it to the plugin's converter attribute. In the compile_string() method, use that ThreadLocalMarkdown instance to perform the conversion. The end result is one Markdown object created per thread, instead of one for each markdown file compiled.