Add the ability to parse just until the end of the first directive

[rynowak]

This is needed to delete MVC's "is it a page or not" functionality.

This will work by replacing the tokenizer with an implementation that will tokenize up to the first directive, and then to the end of the line before ending the token sequence.

This means we don't have to many any modifications to the parser, we just have to decorate the tokenizer.

This is a cheaper plan for implementing this functionality than we earlier planned, and should make it so that MVC deals with important things like a license header, which should be allowed.