Concatenating two ISO-2022-JP outputs from a conforming encoder doesn't result in conforming input

[hsivonen]

Encodings other than ISO-2022-JP have the property that if you concatenate two outputs from a conforming encoder and decode them together, you get the same result as when decoding them separately and then concatenating.

ISO-2022-JP lacks this property, because despite the encoder making an effort into this direction by emitting a transition to the ASCII state at the end, if the next segment being concatenated starts with a transition to a non-ASCII state, the concatenation results in zero ASCII bytes between two escapes.

Is there a strong reason for treating a transition to the ASCII state immediately followed by another escape as non-conforming? Or put the other way, what purpose does the transition to the ASCII state at the end of encode serve if not achieving the above-mentioned concatenation property that other encodings have?

This is relevant to RFC 2047 header decoding.

[vyv03354]

It is my understanding that the reason is to prevent XSS attacks. Consider "<\u001b(B\u001b$Bscript" for example.

[hsivonen]

It is my understanding that the reason is to prevent XSS attacks. Consider "<\u001b(B\u001b$Bscript" for example.

Why is that worth protecting against if we can't protect against "<\x1b(Js\x1b(Bcript"?

That is, if we can't generate U+FFFD for all of these, is it worth generating it for any of these?

• Escape immediately followed by another escape.
• Transition from the ASCII state to the ASCII state.
• Useless transitions between the ASCII state and the Roman state.

The last one seems the hardest to prevent without potentially breaking some legitimate inputs.

[annevk]

Does anyone wish to drive a change proposal here? Or should we just accept this and maybe document the issues a bit further?

cc @jungshik

[annevk]

I decided to simply document this quirk in #155. If there are any concerns with that approach let me know.

[hsivonen]

Another bug reported about Gecko conforming to the spec.

[hsivonen]

I still have trouble seeing the value of generating an error when an escape is followed by another escape when we don't generate errors for the other two cases I mentioned on July 3 2017.

[annevk]

FWIW, if you want to reopen this and remove that error, that's fine by me. It'd be good to know what other implementations do at this point, as changing such details is never fun for anyone involved.

[hsivonen]

I noticed that Unicode Security Considerations say that what the Encoding Standard requires for ISO-2022-JP should be done. If we reopen this, we should ask Mark Davis and Michel Suignard for their rationale.

[hsivonen]

Regarding the previous Gecko bug from a week ago, the reporter provides more info in a Thunderbird bug.