StreamReader.readline doesn't advance on decode errors

[donut]

BPO	1379393
Nosy	@doerwalter, @birkenfeld
Files	python-test_streamreader.py: script to demonstrate the problem skipbadlines.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/doerwalter'
closed_at = <Date 2006-02-19.00:58:54.000>
created_at = <Date 2005-12-13.10:35:47.000>
labels = ['library']
title = "StreamReader.readline doesn't advance on decode errors"
updated_at = <Date 2006-02-19.00:58:54.000>
user = 'https://bugs.python.org/donut'

bugs.python.org fields:

activity = <Date 2006-02-19.00:58:54.000>
actor = 'georg.brandl'
assignee = 'doerwalter'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2005-12-13.10:35:47.000>
creator = 'donut'
dependencies = []
files = ['1850', '1851']
hgrepos = []
issue_num = 1379393
keywords = []
message_count = 4.0
messages = ['27059', '27060', '27061', '27062']
nosy_count = 3.0
nosy_names = ['doerwalter', 'georg.brandl', 'donut']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1379393'
versions = []

[donut]

In previous versions of python, when there was a
unicode decode error, StreamReader.readline() would
advance to the next line. In the current version(2.4.2
and trunk), it doesn't. Testing under Linux AMD64
(Ubuntu 5.10)

Attaching an example script. In python2.3 it prints:

hi~
hi
error: 'utf8' codec can't decode byte 0x80 in position
2: unexpected code byte
error: 'utf8' codec can't decode byte 0x81 in position
2: unexpected code byte
all done

In python2.4 and trunk it prints:
hi~
hi
error: 'utf8' codec can't decode byte 0x80 in position
0: unexpected code byte
error: 'utf8' codec can't decode byte 0x80 in position
0: unexpected code byte
error: 'utf8' codec can't decode byte 0x80 in position
0: unexpected code byte
[repeats forever]

Maybe this isn't actually supposed to work (the docs
don't mention what should happen with strict error
checking..), but it would be nice, given the alternatives:

1. use errors='replace' and then search the result for
   the replacement character. (ick)
2. define a custom error handler similar to ignore or
   replace, that also sets some flag. (slightly less ick,
   but more work.)

[birkenfeld]

Logged In: YES
user_id=1188172

I don't know what should be correct. Walter?

[doerwalter]

Logged In: YES
user_id=89016

IMHO the current behaviour is more consistent. To read the
broken utf-8 stream from the test script the appropriate
error handler should be used. What is the desired outcome?
If only the broken byte sequence should be skipped
errors="replace" is appropriate. To skip a complete line
that contains a broken byte sequence do something like in
the attached skipbadlines.py. The StreamReader can't know
which behaviour is wanted.

[birkenfeld]

Logged In: YES
user_id=1188172

Closing as Won't Fix, then.