Skip to content
This repository
Browse code

More specific docs for contexts and tokenizer.

  • Loading branch information...
commit b2b49ebd80c217d6cf79de2f18b2ada94ccf4324 1 parent e787c97
Ben Kurtovic authored August 21, 2012
24  mwparserfromhell/parser/contexts.py
@@ -29,10 +29,30 @@
29 29
 current point and also if the current parsing route is invalid.
30 30
 
31 31
 The tokenizer stores context as an integer, with these definitions bitwise OR'd
32  
-to add them, AND'd to check if they're set, and XOR'd to remove them.
  32
+to add them, AND'd to check if they're set, and XOR'd to remove them. The
  33
+advantage of this is that contexts can have sub-contexts (as FOO == 0b11 will
  34
+cover BAR == 0b10 and BAZ == 0b01).
  35
+
  36
+Local (stack-specific) contexts:
  37
+
  38
+* TEMPLATE
  39
+** TEMPLATE_NAME
  40
+** TEMPLATE_PARAM_KEY
  41
+** TEMPLATE_PARAM_VALUE
  42
+* HEADING
  43
+** HEADING_LEVEL_1
  44
+** HEADING_LEVEL_2
  45
+** HEADING_LEVEL_3
  46
+** HEADING_LEVEL_4
  47
+** HEADING_LEVEL_5
  48
+** HEADING_LEVEL_6
  49
+
  50
+Global contexts:
  51
+
  52
+* GL_HEADING
33 53
 """
34 54
 
35  
-# Local (stack-specific) contexts:
  55
+# Local contexts:
36 56
 
37 57
 TEMPLATE =              0b000000111
38 58
 TEMPLATE_NAME =         0b000000001
26  mwparserfromhell/parser/tokens.py
@@ -28,6 +28,32 @@
28 28
 :py:class:`~mwparserfromhell.parser.tokenizer.Tokenizer`, and then converted
29 29
 into the :py:class`~mwparserfromhell.wikicode.Wikicode` tree by the
30 30
 :py:class:`~mwparserfromhell.parser.builder.Builder`.
  31
+
  32
+Tokens:
  33
+
  34
+* Text = make("Text")
  35
+* *Templates*
  36
+** TemplateOpen
  37
+** TemplateParamSeparator
  38
+** TemplateParamEquals
  39
+** TemplateClose
  40
+** HTMLEntityStart
  41
+** HTMLEntityNumeric
  42
+** HTMLEntityHex
  43
+** HTMLEntityEnd
  44
+* *Headings*
  45
+** HeadingStart
  46
+** HeadingEnd
  47
+* *Tags*
  48
+** TagOpenOpen
  49
+** TagAttrStart
  50
+** TagAttrEquals
  51
+** TagAttrQuote
  52
+** TagCloseOpen
  53
+** TagCloseSelfclose
  54
+** TagOpenClose
  55
+** TagCloseClose
  56
+
31 57
 """
32 58
 
33 59
 from __future__ import unicode_literals

0 notes on commit b2b49eb

Please sign in to comment.
Something went wrong with that request. Please try again.