Redo pygments tokenization using Mathics tokenizer.

[rocky]

Right now:

$ mathicsscript --pygments-tokens

Mathicscript: 1.1.2, Mathics 2.0.0dev
...
In[1]:= à2
[(Token.Error, 'à'), (Token.Literal.Number.Integer, '2'), (Token.Text.Whitespace, '\n')]
Out[1]= à2

[GarkGarcia]

I'll look into this. I imagine there's a way to customize the tokenizer used by pygments (?) If so, we'll only have to deal with the conversion between our internal token format to the one used by pygments.

[rocky]

I imagine there's a way to customize the tokenizer used by pygments (?)

Yes. And I imagine that if you look at the pygments docs it will be clear how to do. Or look at other Mathematica Pygments formatters.

Just to set expectations here, after doing this you'll probably want to rethink the token name mappings to style names is handled in mathicsscript. Or keep that in mind when doing this.

But as with other things you are interested in, it is a not a small bugfix kind of project, and I have little time during the week if you get stuck.

I think the pygments module should be done outside of core as it is done in the link given above. Down the line we can and probably should split off the tokenizer and parser into its own module that core pulls in.

[rocky]

This has been addressed in version 3.0.0