universal newlines edge case: trailing \r\r treated as a single newline

[gitlab-importer]

In Heptapod by @dairiki on Nov 23, 2023, 07:11

I just stumbled across this by chance:

>>>> import io
>>>> io.TextIOWrapper(io.BytesIO(b"\r\r"), newline="").readlines()
['\r\r']  # wrong — this should be read as two lines
>>>> io.TextIOWrapper(io.BytesIO(b"\r\r\r\r"), newline="").readlines()
['\r', '\r', '\r\r']  # this should be four lines
>>>> io.TextIOWrapper(io.BytesIO(b"a\r\rb\r\r"), newline="").readlines()
['a\r', '\r', 'b\r\r']
>>>> io.TextIOWrapper(io.BytesIO(b"a\r\rb\r\rc"), newline="").readlines()
['a\r', '\r', 'b\r', '\r', 'c']  # ← this one is correct

I see the same behavior with both pypy3.9-7.3.13 and pypy3.10-7.3.13 (installed via pyenv)

The same bug happens when reading from a file.

It only happens with newline="" (universal newlines without newline translation). If newline is set to None, things seem to work as they are supposed to.

[gitlab-importer]

In Heptapod by @cfbolz on Nov 23, 2023, 11:39

thanks for the report. ouch, this is annoying. from your examples it looks like only the last \r\r at the end are treated incorrectly, earlier in the file it seems to be ok? I'll look into it

[gitlab-importer]

In Heptapod by @cfbolz on Nov 23, 2023, 15:17

mentioned in commit 5b31c9b32ef9c204de9b0671cdcb542a8c6f2e25

[gitlab-importer]

In Heptapod by @cfbolz on Nov 23, 2023, 15:19

fixed in pypy2 in the above commit, now merging to pypy3

[gitlab-importer]

In Heptapod by @cfbolz on Nov 28, 2023, 17:32

mentioned in commit 4e87fc16efa440c7368282311b0e5df6ebe4f237

[mattip]

Closing, should be fixed. Please open a new issue if needed.