Better lexing(and scanning) and parsing

[lestherll]

There is currently an issue with the lexing strategy for unary operators. Because i have only partially implemented it, constructs that uses unary operators are note exactly recognised. For example

> -(1)
1  # should be -1 

This is problematic because it affects expressions such as to give wrong the answer

> -1^2
1  # should be -1
> -(1^2)
1  # should be -1

The main reason for this is that unary operators are not actually lexed themselves, when an expression such as -1 is passed, the program checks if the immediately succeeding character is a number and immediately binds the sign to the number after it, a bracket/parentheses is not a number and thus the operator doesn't get lexed independently at all disabling the parser to not recognise have any knowledge of it.

Also note that + and - are always initially tokenised as Token.BINARY_OPs and I believe this should change.

see this line

[lestherll]

One solution I can think of is to do a scanning pass to the whole program/expression, separating each valid characters one by one, before converting them into tokens. This would make lexing easier as now there are only valid characters to lex/tokenise.

This will also allow for the easier separation of binary and unary operators.

[lestherll]

3be3d92 now fixes unary operator precedence bug