Document the tokeniser #11
Conversation
| """ | ||
| A tokeniser for the Wolfram Language. | ||
|
|
||
| When subclassing ``Tokeniser``, custom tokenisation rules can be defined by |
There was a problem hiding this comment.
I don't think this mechanism should be exposed in the public API (and therefore it shouldn't be documented in the docstring). If you think about it, all consumers of this library want is to have a functioning WL tokeniser that they can use as a black-box (that's what I think at-least).
This definitively should be documented somewhere though. @rocky I'd apprciate if we could merge #8 before this, so that I can move this information to implementation.rst. I also plan to convert implementation.rst and the rest of the documentation to a proper Sphinx document before we release the library (which should be pretty easy to do, so it's not gonna take too much time).
This methods are only useful internally and are not used by core anywhere
|
@rocky Please wait for me to finish document the tokeniser before the release. It's going to be very fast and it won't break anything since we're only changing the documentation. I'd be very disappointed if you went through with the release without giving me time to fix the documentation, since I've been waiting the entire week for you to merge #8 so that I could proceed with the changes in here. |
|
A release is already done. Sorry. We can make another realase. I don't understand why fixing documentation couldn't have been done previously. |
I needed
Since you didn't reply to my comment, I assumed you were going to wait for me to make the changes. |
|
Sorry about the release. |
|
Rocky, please, before de release check that mathics master pass the CI |
|
@mmatera Have been working on that right now. Looks like there was longstanding breakage I am finding out about now. See mathics/Mathics#1146 The failing PossbleQ tests I think are related to something else. |
This is a follow up to #9. I documented everything I could find about the bits of the tokeniser used by core.