Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

More specific docs for contexts and tokenizer.

  • Loading branch information...
commit b2b49ebd80c217d6cf79de2f18b2ada94ccf4324 1 parent e787c97
@earwig authored
View
24 mwparserfromhell/parser/contexts.py
@@ -29,10 +29,30 @@
current point and also if the current parsing route is invalid.
The tokenizer stores context as an integer, with these definitions bitwise OR'd
-to add them, AND'd to check if they're set, and XOR'd to remove them.
+to add them, AND'd to check if they're set, and XOR'd to remove them. The
+advantage of this is that contexts can have sub-contexts (as FOO == 0b11 will
+cover BAR == 0b10 and BAZ == 0b01).
+
+Local (stack-specific) contexts:
+
+* TEMPLATE
+** TEMPLATE_NAME
+** TEMPLATE_PARAM_KEY
+** TEMPLATE_PARAM_VALUE
+* HEADING
+** HEADING_LEVEL_1
+** HEADING_LEVEL_2
+** HEADING_LEVEL_3
+** HEADING_LEVEL_4
+** HEADING_LEVEL_5
+** HEADING_LEVEL_6
+
+Global contexts:
+
+* GL_HEADING
"""
-# Local (stack-specific) contexts:
+# Local contexts:
TEMPLATE = 0b000000111
TEMPLATE_NAME = 0b000000001
View
26 mwparserfromhell/parser/tokens.py
@@ -28,6 +28,32 @@
:py:class:`~mwparserfromhell.parser.tokenizer.Tokenizer`, and then converted
into the :py:class`~mwparserfromhell.wikicode.Wikicode` tree by the
:py:class:`~mwparserfromhell.parser.builder.Builder`.
+
+Tokens:
+
+* Text = make("Text")
+* *Templates*
+** TemplateOpen
+** TemplateParamSeparator
+** TemplateParamEquals
+** TemplateClose
+** HTMLEntityStart
+** HTMLEntityNumeric
+** HTMLEntityHex
+** HTMLEntityEnd
+* *Headings*
+** HeadingStart
+** HeadingEnd
+* *Tags*
+** TagOpenOpen
+** TagAttrStart
+** TagAttrEquals
+** TagAttrQuote
+** TagCloseOpen
+** TagCloseSelfclose
+** TagOpenClose
+** TagCloseClose
+
"""
from __future__ import unicode_literals
Please sign in to comment.
Something went wrong with that request. Please try again.