It is a wrapper around nqp Perl6 parsing methods.
Known problem: if it is compiled into pir. (Panda does that), use Perl6::Parsing fails. It seems to be a Rakudo bug.
Workaround: delete Parsing.pir.
use Rakudo::Perl6::Parsing; my $p= Rakudo::Perl6::Parsing.new(); # create a new object $p.parse("my \$p=3;"); # let us parse this text <br> say $p.parser.dump; # dumps parse tree $p.printree(); #prints the parse tree using a different format my @tokens = $p.tokenise(); # extract tokens , requires $p.parse... Instead of tokens, it would be more accurate to say parsed texts. It may be useful for all kind of Perl 6 analysis. @tokens is a array of [hash (keys are tokentypes or parsing events, values are charpos where the token may ends), startpos in text, endpos in text ]. There are overlapping tokens but no overlaps are returned. Look at the values of the hash to determine overlaps. <br> E.g. values in the previous line is not equal to endpos in text. say $p.text.substr(@tokens,@tokens-@tokens); # prints first token Tokens are derived from parse tree. It means the token boundaries may not be where you expect them to be. <br> For example, two consecutive comments may be returned as one token. Token boundaries are derived from what the parser considered to be important: code mainly. say @tokens.perl; # look at the structure say $p.dumptokens(); # shows better view