gh-85287: Change codecs to raise precise UnicodeEncodeError and UnicodeDecodeError

[jjsloboda]

Change codecs to raise UnicodeEncodeError and UnicodeDecodeError, depending on what operation the API user is attempting, instead of just UnicodeError. This involved providing more information to the exceptions, including the problematic string and start and end offsets within the string, which are not always relevant depending on the nature of the exception.

Some of the codecs will perform multiple encodes and decodes during an operation as part of their implementation. In these cases, I've opted to throw the exception that matches the API function the error is occurring in.

A tricky thing to watch out for while reviewing is that UnicodeEncodeError takes a str and UnicodeDecodeError takes a bytes, which is supposed to be the original object being en/decoded. However, most of these codec functions modify their arguments, so by the time the exception block is reached, the original argument is no longer available and needs to be reconstructed. Alternatively, I could change the PR to save the argument, but I don't think it's worth the extra memory just for the exception case (may be more pythonic though).

Another thing to watch for in the review is to make sure I'm not doing anything in the exception blocks that could trigger more en/decoding exceptions when I'm reconstructing the argument strings.

• Issue: Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError #85287

[cpython-cla-bot]

All commit authors signed the Contributor License Agreement.

[methane]

Suggested change

	raise UnicodeEncodeError("idna", input, 0, 1, f"unsupported error handling {errors}")
	raise UnicodeEncodeError("idna", input, 0, 0, f"unsupported error handling {errors}")

[serhiy-storchaka]

UnicodeEncodeError implies that there is an error in the encoded string.

Other codecs usually raise LookupError for unknown error handlers, but here an exception is raised even for known error handlers.

Perhaps ValueError better suits here. UnicodeError is a subclass of ValueError, so we can keep it for now.

[jjsloboda]

Changed the cases where there was not an issue with the input string back to UnicodeError.

[methane]

Suggested change

	raise UnicodeEncodeError("idna", input, index, index+len(label), "label empty or too long")
	raise UnicodeEncodeError(
	"idna", input, index, index+len(label),
	"label empty or too long")

[serhiy-storchaka]

Agree we using absolute offset.

But it still reports weird range for empty label. I think that it is better to include an opening or closing dot.

Also, I think that it is better to use different messages for empty and too long labels.

[jjsloboda]

Split out the empty vs too long cases. Note that UnicodeEncodeError takes a python-style [start, end) range, but outputs a (start)-(end-1) inclusive range in the error message, so using something like 0, 0, gives the weird-looking 0--1 error message whereas 0, 1, gives position 0.

[methane]

Suggested change

	raise UnicodeEncodeError("idna", input, index, len(input), "label too long")
	raise UnicodeEncodeError("idna", input, index, len(input),
	"label too long")

[jjsloboda]

Fixed the linebreaks of some of the longer exceptions to look like this

[jjsloboda]

Thanks for the reviews @methane and @serhiy-storchaka, comments addressed and ready for another look

[serhiy-storchaka]

We cannot simply translate offsets from a label to the full input, because the label is transformed several times during encoding: some characters are ignored, others are combined in the normalization or transformed in the punycode encoding. It is difficult to track offsets of original characters that cannot be encoded. So I suggest to simply report the range of the whole label that cannot be encoded. The main goal is changing the type of the exception to more specific, and it should be enough for now. Later we can narrow the range of error if it is useful and possible.

[jjsloboda]

I decided to add some tests for the error offset in one more hope to get the offsets figured out properly, or at least document the current behavior. They seem to be working now at least for those cases, but I'm not an expert in IDNA or Punycode so I'm not sure if there are important edgecases I'm missing.

My goal was to avoid the need to revisit this in the future to further tighten up the offsets, but @serhiy-storchaka let me know if you'd still prefer I switch it to use the whole range of the problematic label.

Ready for another look @methane @serhiy-storchaka

[methane]

Do we need copy inbuf? If no, use just inbuf with O format.

[jjsloboda]

Done 👍 I think I tried this originally, but I must have been doing something different because it seems to work fine now.

[methane]

same here.

[methane]

missing if (excobj == NULL) goto errorexit;

[jjsloboda]

Done, for all new and pre-existing code

[methane]

same here.

[jjsloboda]

Thanks @methane , ready for another look

[methane]

I don't feel this is valid UnicodeEncodeError.
Should we really avoid UnicodeError here?

[jjsloboda]

If UnicodeEncodeError is because the user passed an input that directly causes the encoding problem, rather than any problem that occurs during encoding, then UnicodeError makes sense here. Changed this one back to UnicodeError.

[jjsloboda]

Comment addressed @methane , thanks, ready for review

@serhiy-storchaka I think the new IDNA errors work now at least for the main cases, but let me know if you still want me to go back and change the IDNA encoding errors to report the whole label as the problematic range

[methane]

punycode looks ugly. Although it doesn't have __all__, I think only punycode_decode, punycode_encode, Codec, IncrementalEncoder, IncrementalDecoder, StreamWriter, StreamReader, and getregentry are public APIs.
(Strictly speaking, getentry is the only API used from external other than tests.)
So I will change where to encode/decode in them.

[methane]

LGTM. But I will wait for a few weeks to have chance other core developer review my changes.