most pre-defined tokens (i.e. Grammar methods) aren't documented

[smls]

/language/grammars#Special_Tokens documents only two pre-defined tokens: <ws> and <!>.¹

But there are actually quite a few more.

S05#Predefined_Subrules lists these:

<!> <?> alnum alpha blank cntrl digit graph ident lower print punct space upper wb ws ww xdigit

And Rakudo introspection reveals:

➜  say Grammar.^methods».name.sort;
(!APPEND_TO_ORIG !BACKREF !BACKREF-LATEST-CAPTURE !DELEGATE_ACCEPTS !DYNQUANT_LIMITS !LITERAL !alt !alt_nfa !clear_highwater !cursor_capture !cursor_fail !cursor_init !cursor_more !cursor_next !cursor_pass !cursor_pos !cursor_push_cstack !cursor_start !cursor_start_all !cursor_start_cur !cursor_start_fail !cursor_start_subcapture !dba !fresh_highexpect !highexpect !highwater !precompute_nfas !protoregex !protoregex_nfa !protoregex_table !reduce !reduce_with_match !set_highexpect !shared Bool CALL_SUBRULE CAPHASH DYNQUANT_LIMITS FAILGOAL INDMETHOD INDRULE INTERPOLATE MATCH MATCH_SAVE OTHERGRAMMAR RECURSE after alnum alpha at before blank cntrl digit from graph ident lower orig parse parsefile pos print prune punct same space subparse target upper wb ws ww xdigit)

A bunch of those are Rakudo internals, but the intersection of that list with the S05 list, are the ones that should probably be officially documented... :p

Since they're technically methods of class Grammar, I think they should be listed as routines (with keyword token instead of method) on /type/Grammar, but it probably wouldn't hurt mentioning them on /language/grammars as well.

---

¹ TOP is special, since it isn't a pre-defined token, but rather one that user code is expected to define.

[moritz]

They are now mentioned in doc/Language/regexes.pod6.