Prefixed string/char literals

[lbajolet]

Unicode is good, unicode is great, but it has a few undesirable pitfalls sometimes:

• Byte strings: low-level C users know what I'm talking about, a Nit Text object is necessarily Unicode, which leaves the guy wanting to use a string as a byte sequence sadly coping with invalid byte sequences (i.e. cannot do shellcoding in nit without expressing all, including text, in a hexdigest)
• Chars used as their value as explained in We need an easy service to get the ascii code point from a literal char #1718

For these uses, no real easy solution exists now in Nit.
As such, this issue proposes the introduction of prefixed strings and chars.
It will first be recognized grammatically, and some prefixes will be implemented in the compiler later.

Some candidate prefixes on strings are:

• b"Byte\xfe" => for a byte string with potentially invalid unicode chars in it.
• re"\w*?+" => for Regular Expressions
• raw"\n\rEscaped" => For raw strings, where escape sequences are treated as normal characters

Some candidate prefixes on chars are:

• a'\n' => ASCII value of char '\n' as a Byte
• u'𐏓' => Unicode code-point of char '𐏓' as an Int

This issue is also a feature request for people wishing for more specific or exotic prefixes, this is the place and the moment to express your needs :)

[xymus]

What is the use of the Byte string with invalid Unicode chars? I would prefer an ASCII string to go with the ASCII Byte.

[lbajolet]

I'm not sure I understood what you meant by "ASCII string to go with the ASCII Byte", could you clarify ?

Since an 0xFE byte is invalid in a string, the only use for such byte strings is to ease the writing of byte sequences with lots of text in it but with occasional delimiters or non-valid characters.

This is relatively common when dealing with protocols, shellcodes, and such.

[xymus]

About the ASCII string, I would have expected a a"ASCII string" alongside the ASCII byte a'\n'. But it's pretty useless, the literal unicode char with only ASCII characters is the same. So it's OK as proposed.

I had misinterpreted the goal of b"", it looked illogical to write invalid unicode string literals for a unicode parser, but now I see the escape sequences. So what would be the byte string, a Bytes?

The re prefix is great. You should also prepare for additional pre/suffixes such as i for ignore_case, m for newline and maybe something for not extented.

[lbajolet]

Bytes is a good idea for the return of the byte strings yeah.

The idea of this PR is to submit idea for prefixes and suffixes so the i, m and not extended (whatever the suffix could be, I'll take any input into consideration at the moment) can be added to the list of what to implement :)

I'll do this tomorrow maybe, be ready for the PR then !

[privat]

I agree the the intention and the need.

Two (detail i.e. devil) things:

• What is the real/fundamental difference between re"(a|b)*" and "(a|b)*".to_re? Shouldn't we expect, by the philosophy of POLA, that they are strictly equivalent, thus, shouldn't we assume, by the philosophy of KISS, that one is superfluous?
• Should we go with prefixes, suffixes or both? Should we try to be coherent with literal integers that have both, the prefix being the representation and the suffix being the type (eg. 0x7Fu8)?

[lbajolet]

Following the reasoning, suffixes like u8 for Bytes should be suppressed since we can do 0x4F.to_b :)

I don't think that adding a layer of syntactic sugar is a bad idea for common cases like these, and I do believe that we should add both prefixes and suffixes as it might be useful.

[privat]

closed by #1851 #1991 and #1992