429 Add hex and binary literals and allow underscores

[michaelhkay]

Addresses issue #429. The grammar is extended to allow hex and binary integer literals, and all numeric literals may contain underscores for readability.

[ndw]

👍

[ChristianGruen]

Much appreciated (and no need for me to offer a pull request for this one anymore).

Thoughts:

• Most languages seem to also support both 0X and 0B, but as X isn’t legal in character references either, I agree it’s probably better to only offer the lower-case options.
• The new grammar does not allow integer literals to start with underscores (as _1 is a valid name test). To be consistent with other languages, maybe we should also disallow underscores at the end of Digits and the start/end of HexDigits and BinaryDigits?

    (: legal :)  0b0_1, 0x2_3, 4_5, 6_7.8_9,
  (: illegal :)  _1, 2._3, 4e_5, 6.7e_8
(: disallow? :)  0b_0, 0b1_, 0x_2, 0x3_, 4_, 5.6_, 7.8e9_

[ChristianGruen]

Suggested grammar rules:

Digits       ::= Digit ((Digit | "_")* Digit)?
Digit        ::= [0-9]
HexDigits    ::= HexDigit ((HexDigit | "_")* HexDigit)?
HexDigit     ::= [0-9a-fA-F]
BinaryDigits ::= BinaryDigit ((BinaryDigit | "_")* BinaryDigit)?
BinaryDigit  ::= [01]

[ChristianGruen]

Observation from Reece (QT4 CG Meeting 032):

Note that JavaScript disallows consecutive underscores, so 1_000 is valid, but 1__000 is invalid.

[ndw]

Accepted at meeting 032

[michaelhkay]

Both Java and C# appear to allow adjacent underscores (and both prohibit terminal underscores).