The title from toc_tokens ignores the smarty extension

[oprypin]

Current actual result:

>>> import markdown
>>> md = markdown.Markdown(extensions=['toc', 'smarty'])
>>> md.convert('# *Foo* --- `bar`')
'<h1 id="foo-bar"><em>Foo</em> &mdash; <code>bar</code></h1>'
>>> md.toc_tokens
[{'level': 1, 'id': 'foo-bar', 'name': 'Foo --- bar', 'children': []}]

Expected result:

[{'level': 1, 'id': 'foo-bar', 'name': 'Foo — bar', 'children': []}]

Background where I discovered this: mkdocs/mkdocs#3357 (comment)

This happens because smarty runs at a lower priority than toc (unlike most other treeprocessors) and it doesn't have a chance to kick in.

• markdown/markdown/extensions/toc.py

  Line 404 in a63e6f3

  md.treeprocessors.register(tocext, 'toc', 5)

• markdown/markdown/extensions/smarty.py

  Line 272 in a63e6f3

  md.treeprocessors.register(inlineProcessor, 'smarty', 2)

This other extension is presumably affected in the same way, probably among others:
https://github.com/facelessuser/pymdown-extensions/blob/8f5283f71e8833f1ba0fa0bbe2680e1ad2c6cb19/pymdownx/smartsymbols.py#L167

Something should be done so that all the rest of the treeprocessors are also applied before saving the title. Maybe postprocessors too, rather than applying the unescape functionality out-of-band.

[oprypin]

By the way, please don't solve this by lowering the priority of the toc extension, so much stuff I made already depends on its precise priority 🥲

[waylan]

I didn't have much involvement in the smarty extension, which was added much later than TOC. I wonder why it was given a lower priority. @mitya57 do you recall? I seem to recall you being involved in the development of the smarty extension.

I ask why because, barring any good reason not to, I have no object in raising the priority to be before TOC.

[mitya57]

When I added smarty treeprocessor in 47ec0cb, I probably just pushed it to the end without thinking too much.

The tests pass when changing its priority, so I think it's fine to do that. Submitted PR as #1440.