implement "use source::encoding"

[rjbs]

RFC 7 is the "source encoding pragma", meant to allow the programmer to declare the encoding of the source document. Right now, we have two options for this:

• use utf8 which says "this source document is UTF-8 encoded, decode it before compiling"
• not using use utf8 which says "this source document is bytes"

Both of these options lead to confusion of one sort or another, and I am not here (today, anyway) to argue which is absolutely better. But what is lacking is:

• use ascii to say "if it's not ASCII, which is unambiguous in meaning, reject compilation"

The new pragma is meant to be a one-stop-shop for declaring any of the above, to be called as:

• use source::encoding 'ascii'
• use source::encoding 'utf8'
• no source::encoding (but this one is probably better avoided)

I proposed that the utf8 form do the same thing as utf8, and that over time we tighten up the strictness of utf8.pm, and then also of the exact codepoint sequences permitted in Perl source -- but those enhancements can happen later than and separately from this pragma.

[khwilliamson]

And my question is should we exclude some ASCII non-printables under the 'ascii' option? Conceivably a backsapce could be used to sneak something by a reader. Should we require those to be specified as escape sequences? There are octal, hex, and \cX versions of those. Even tab can cause confusion. Or maybe we accept some in single quoted context, but not in double-quotish?

For your convience, here is a complete list of the ASCII non-printables:
U+0000 NULL
U+0001 START OF HEADING
U+0002 START OF TEXT # 2 ** 1
U+0003 END OF TEXT
U+0004 END OF TRANSMISSION # 2 ** 2
U+0005 ENQUIRY
U+0006 ACKNOWLEDGE
U+0007 ALERT
U+0008 BACKSPACE # 2 ** 3
U+0009 CHARACTER TABULATION
U+000A LINE FEED
U+000B LINE TABULATION
U+000C FORM FEED
U+000D CARRIAGE RETURN
U+000E SHIFT OUT
U+000F SHIFT IN
U+0010 DATA LINK ESCAPE # 2 ** 4
U+0011 DEVICE CONTROL ONE
U+0012 DEVICE CONTROL TWO
U+0013 DEVICE CONTROL THREE
U+0014 DEVICE CONTROL FOUR
U+0015 NEGATIVE ACKNOWLEDGE
U+0016 SYNCHRONOUS IDLE
U+0017 END OF TRANSMISSION BLOCK
U+0018 CANCEL
U+0019 END OF MEDIUM
U+001A SUBSTITUTE
U+001B ESCAPE
U+001C INFORMATION SEPARATOR FOUR
U+001D INFORMATION SEPARATOR THREE
U+001E INFORMATION SEPARATOR TWO
U+001F INFORMATION SEPARATOR ONE
U+007F DELETE

[demerphq]

On Thu, 3 Feb 2022, 10:40 Ricardo Signes, ***@***.***> wrote: RFC 7 <https://github.com/Perl/RFCs/blob/master/rfcs/rfc0007.md> is the "source encoding pragma", meant to allow the programmer to declare the encoding of the source document. Right now, we have two options for this: - use utf8 which says "this source document is UTF-8 encoded, decode it before compiling" - not using use utf8 which says "this source document is bytes" Both of these options lead to confusion of one sort or another, and I am not here (today, anyway) to argue which is absolutely better. But what is lacking is: - use ascii to say "if it's not ASCII, which is unambiguous in meaning, reject compilation" The new pragma is meant to be a one-stop-shop for declaring any of the above, to be called as: - use source::encoding 'ascii' - use source::encoding 'utf8' - no source::encoding (but this one is probably better avoided) I proposed that the utf8 form do the same thing as utf8, and that over time we tighten up the strictness of utf8.pm, and then also of the exact codepoint sequences permitted in Perl source -- but those enhancements can happen later than and separately from this pragma.

Fwiw, I am strongly in support of this with the minor exception that I think we should leave the "utf8"/"UTF-8" distinction alone, however I do think we should forbid "utf8" as a source encoding and either specify that saying "utf8" in /this/ context really means strict UTF-8, or forbid the use of "utf8" in this context. Imo it can be very useful to use "utf8" as a way to represent arbitrary sequences of packed integers, but this means allowing codepoints that would be absolutely illegal in UTF-8. But I see no need to support this in a perl source code context. Having said all that it's not a *huge* deal for me. I think the proposal is good regardless. Thanks for your work putting it forward. Yves

…

[demerphq]

On Thu, 3 Feb 2022, 10:40 Ricardo Signes, ***@***.***> wrote: RFC 7 <https://github.com/Perl/RFCs/blob/master/rfcs/rfc0007.md> is the "source encoding pragma", meant to allow the programmer to declare the encoding of the source document. Right now, we have two options for this: - use utf8 which says "this source document is UTF-8 encoded, decode it before compiling" - not using use utf8 which says "this source document is bytes" Both of these options lead to confusion of one sort or another, and I am not here (today, anyway) to argue which is absolutely better. But what is lacking is: - use ascii to say "if it's not ASCII, which is unambiguous in meaning, reject compilation" The new pragma is meant to be a one-stop-shop for declaring any of the above, to be called as: - use source::encoding 'ascii' - use source::encoding 'utf8' - no source::encoding (but this one is probably better avoided) I proposed that the utf8 form do the same thing as utf8, and that over time we tighten up the strictness of utf8.pm, and then also of the exact codepoint sequences permitted in Perl source -- but those enhancements can happen later than and separately from this pragma.

I have a question regarding BOMs and UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE. Imo it would seem reasonable and sane to treat a BOM at index 0 in a source file as being exactly equivalent as saying use source::encoding $type; etc. Remember that of the Unicode encodings utf8/UTF-8 is the only one that does not need to know the endianness of the data, all the rest do. In windows land this is normally solved with a BOM at the start of the file. What do you think? What is your position on the use of BOM's for UTF-16 and UTF-32? Cheers, Yves

…

[demerphq]

On Thu, 3 Feb 2022 at 04:22, demerphq ***@***.***> wrote: On Thu, 3 Feb 2022, 10:40 Ricardo Signes, ***@***.***> wrote: > RFC 7 <https://github.com/Perl/RFCs/blob/master/rfcs/rfc0007.md> is the > "source encoding pragma", meant to allow the programmer to declare the > encoding of the source document. Right now, we have two options for this: > > - use utf8 which says "this source document is UTF-8 encoded, decode > it before compiling" > - not using use utf8 which says "this source document is bytes" > > Both of these options lead to confusion of one sort or another, and I am > not here (today, anyway) to argue which is absolutely better. But what is > lacking is: > > - use ascii to say "if it's not ASCII, which is unambiguous in > meaning, reject compilation" > > The new pragma is meant to be a one-stop-shop for declaring any of the > above, to be called as: > > - use source::encoding 'ascii' > - use source::encoding 'utf8' > - no source::encoding (but this one is probably better avoided) > > I proposed that the utf8 form do the same thing as utf8, and that over > time we tighten up the strictness of utf8.pm, and then also of the exact > codepoint sequences permitted in Perl source -- but those enhancements can > happen later than and separately from this pragma. > I have a question regarding BOMs and UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE. Imo it would seem reasonable and sane to treat a BOM at index 0 in a source file as being exactly equivalent as saying use source::encoding $type; etc. Remember that of the Unicode encodings utf8/UTF-8 is the only one that does not need to know the endianness of the data, all the rest do. In windows land this is normally solved with a BOM at the start of the file. What do you think? What is your position on the use of BOM's for UTF-16 and UTF-32?

BTW, keep in mind that if the source code is encoded in a fixed width widechar encoding it would look like: UTF-16LE -> qq<u\0s\0e\0 \0s\0o\0u\0r\0c\0e\0:\0:\0e\0n\0c\0o\0d\0i\0n\0g\0 \0"\0U\0T\0F\0-\01\06\0L\0E\0"\0;\0> UTF-16BE -> qq<\0u\0s\0e\0 \0s\0o\0u\0r\0c\0e\0:\0:\0e\0n\0c\0o\0d\0i\0n\0g\0 \0"\0U\0T\0F\0-\01\06\0B\0E\0"\0;> UTF-32LE -> qq<u\0\0\0s\0\0\0e\0\0\0 \0\0\0s\0\0\0o\0\0\0u\0\0\0r\0\0\0c\0\0\0e\0\0\0:\0\0\0:\0\0\0e\0\0\0n\0\0\0c\0\0\0o\0\0\0d\0\0\0i\0\0\0n\0\0\0g\0\0\0 \0\0\0"\0\0\0U\0\0\0T\0\0\0F\0\0\0-\0\0\03\0\0\02\0\0\0L\0\0\0E\0\0\0"\0\0\0;\0\0\0> UTF-32BE -> qq<\0\0\0u\0\0\0s\0\0\0e\0\0\0 \0\0\0s\0\0\0o\0\0\0u\0\0\0r\0\0\0c\0\0\0e\0\0\0:\0\0\0:\0\0\0e\0\0\0n\0\0\0c\0\0\0o\0\0\0d\0\0\0i\0\0\0n\0\0\0g\0\0\0 \0\0\0"\0\0\0U\0\0\0T\0\0\0F\0\0\0-\0\0\03\0\0\02\0\0\0B\0\0\0E\0\0\0"\0\0\0;> The fact that we can avoid these issues with UTF-8 and ASCII/Latin-1/Binary is a byproduct of UTF-8's design to be codepoint and octet equivalent to ASCII for codepoints 0-127, and to be codepoint equivalent to Latin-1 for codepoints 0-255 (Latin-1 is in turn codepoint and octet equivalent to ASCII for codepoints 0-127). But this does not apply for other multi-byte encodings like UTF-16 or UTF-32 where byte order is critical. BOM's exist *exactly* for this purpose and I think we should take advantage of them. So I propose that we define that if a perl source code file starts with the bytes FE FF that it is functionally equivalent to saying q<use source::encoding "UTF-16BE";>, and if it starts with 00 00 FE FF that it is exactly equivalent to saying q<use source::encoding "UTF-32BE"> and if it starts with the byte sequence FF FE 00 00 that it is exactly equivalent to saying q<use source::encoding "UTF-32LE";> otherwise if it starts with the bytes FF FE (and is NOT followed by two null bytes), that is exactly equivalent to saying q<use source::encoding "UTF-16LE";> This would make it possible to use standard Windows editors to work on perl source code without needing to be explicit about the encoding, and would resolve the question of "how do you even say 'use source::encoding "UTF-16BE";' in a document so encoded?". This could perhaps be extended to the entire set of BOM's documented at https://en.wikipedia.org/wiki/Byte_order_mark cheers, Yves

…

-- perl -Mre=debug -e "/just|another|perl|hacker/"

[Leont]

Imo it would seem reasonable and sane to treat a BOM at index 0 in a source
file as being exactly equivalent as saying

One side-note is that is this would only work for modules, not for scripts (at least not on Unix) because it breaks shebangs.

AFAIK we already handle utf-8 BOMs. And I'm not sure there's really value in UTF-16 and UTF-32 support. In practice they're quite uncommon for text files IME.

[khwilliamson]

On 2/3/22 17:05, Leon Timmermans wrote: Imo it would seem reasonable and sane to treat a BOM at index 0 in a source file as being exactly equivalent as saying One side-note is that is this would only work for modules, not for scripts (at least not on Unix) because it breaks shebangs. AFAIK we already handle utf-8 BOMs. And I'm not sure there's really value in UTF-16 and UTF-32 support. In practice they're quite uncommon for text files IME.

I'm pretty sure we read UTF-16 BOMs properly in parsing perl scripts. There is code to do that in toke.c

[ikegami]

• not using use utf8 which says "this source document is bytes"

It means "this document is ASCII (with 8-bit clean literals)" on ASCII-based perls.

It means "this document is EBCDIC (with 8-bit clean literals)" on EBCDIC-based perls.

[ikegami]

I'm pretty sure we read UTF-16 BOMs properly in parsing perl scripts. There is code to do that in toke.c

IIRC, we do on Windows, but only on Windows?

[khwilliamson]

AFAICT we don't restrict it to Windows; The code in toke.c is turned off by defining PERL_NO_UTF16_FILTER and we don't do that anywhere

[khwilliamson]

    not using use utf8 which says "this source document is bytes"

It means "this document is ASCII (with 8-bit clean literals)" on ASCII-based perls.

It means "this document is EBCDIC (with 8-bit clean literals)" on EBCDIC-based perls.

You can have the 8th bit set on ASCII platforms in comments and literals out of the box. (And you pretty much always do with EBCDIC as the letters and digits that correspond to ASCII are all above 127. (The reason this is transparent, mostly, in C is that 'char' is typically signed on ASCII platforms and unsigned on EBCDIC ones)

[khwilliamson]

This was implemented by 45e4ded in #20976