How do you get the parsed token tree?

[swquinn]

I'm looking at Mistune, and it appears that you can get the token tree but I can't figure out how.

There are a few issues that ask a similar question (e.g. #98, #25, and #29), and some reference PR #20. Looking at these issues, however, it isn't clear to me how you can use Mistune for getting the parse tree. I was not able to identify how to do this from the documentation either.

My guess is that I'm using Mistune incorrectly. I've provided an example from the Python REPL how I'm trying to use Mistune to get the parse tree. Could someone help me with this?

>>> import mistune
>>> md = mistune.Markdown()
>>> md('# This is a test\n[this](is-a-link)')
'<h1>This is a test</h1>\n<p><a href="is-a-link">this</a></p>\n'
>>> md.tokens
[]
>>> md.token
{'type': 'paragraph', 'text': '[this](is-a-link)'}

In the above, observe that md.token is only showing the last token. Also, I would expect to see tokens represent the whole token tree.

It looks like as Mistune renders the output it also pops elements off of the token tree. Is there any reason that, after parsing the elements, it can't also track the tree as an immutable list (e.g. tuple) that won't be mutated as the tokens are processed for rendering?

[michaeljoseph]

@swquinn I think you need a custom Renderer.
Here's an example from the tests:
https://github.com/lepture/mistune/blob/master/tests/test_subclassing.py#L146-L191