Normalizing line endings in ixml inputs

[ndw]

In a comment on issue #176 , @johnwcowan writes:

I take a radical (and perhaps radically annoying) perspective, namely that the five line-end sequences #D, #A, #D#A, #85, #2028 never mean different things in text (binary data is another matter) and that the ixml "input hopper" should be responsible for blindly normalizing them all to #A. Consequently, rules with #D, #85, or #2028 in them would never match anything (and it is reasonable to provide a warning if one is seen). This would be performed just after encoding conversion and before matching.

This amounts to a claim that different line endings never have any differential semantics. I believe this to be true of all text, not just XML, and assuming it takes the burden of dealing with junk off both the rule author and the consumer of the XML output by an ixml processor, just as it is not necessary to write extra rules to deal with input encodings (which would be equivalent to treating all input as binary).

In short, we should stop distinguishing once and for all between ASR/KSR 33 (CRLF) and ASR 37 (LF) Teletypes. Their day is done.

The CG discussed this briefly today and decided it warrants further discussion.

Among the possible options are:

1. Do nothing
2. Leave line endings unchanged and suggest that implementations should provide an option to normalize line endings
3. Normalize line endings by default (and suggest that implementations provide a way to disable this functionality)

[ndw]

Per an action I took to write up a proposal: https://lists.w3.org/Archives/Public/public-ixml/2023Oct/0021.html

[spemberton]

Apparently I'm not receiving all mail from W3C, because I didn't receive the referenced email, but I see a 4th alternative, namely to have a specific notation for newline, that implementations would map to the local machine's newline convention. That way there is a generalised notation of newlines, without excluding other uses of the characters involved, nor requiring the implementations to mess with the input stream. Using / as a strawperson, since that is what is used in poetry: file = line++/, /*. line = ~[/]+. where / stands for the character or characters used in the newline sequence on the implementation, with the understanding that [/] may need special treatment. input: "Host: ", host, ":", port, /, "Connection: keep-alive", /, "Accept: ", mediatype++",", /. Steven On Monday 23 October 2023 19:07:11 (+02:00), Norman Walsh wrote: Per an action I took to write up a proposal: https://lists.w3.org/Archives/Public/public-ixml/2023Oct/0021.html — Reply to this email directly, view it on GitHub <#192 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNIXL5HGVNIYEWN5PISVVTYA2P37AVCNFSM6AAAAAA2XMAHS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZVGYZTQNRUG4> . You are receiving this because you are subscribed to this thread.

[johnwcowan]

The advantages of #A is that it has no other semantics (unless you are talking to a Model 33 TTY) and it is already used in plain text by most systems, the exceptions being the Internet and protocol transactions such as HTTP, FTP, SMTP, Gemini, etc.

In any case, this is not a "fourth option"; it is orthogonal to the choice between 2 and 3 (but not 1). As should be obvious by now, I support option 3 with #A.