Add doctest option for allowing splitting strings without whitespaces

[picnixz]

Feature or enhancement

Proposal:

The NORMALIZE_WHITESPACE option allows to split long output into multiple lines but this doesn't work with strings that do not have whitespaces. For instance:

>>> ok = "foo bar"
>>> ok  # doctest: +NORMALIZE_WHITESPACE
'foo
bar'

>>> fail = "foobar"
>>> fail  # doctest: +NORMALIZE_WHITESPACE
'foo
bar'

I suggest adding an option to handle very long strings (I don't think we should extend NORMALIZE_WHITESPACE as someone might want the new line in the output to match a whitespace). It would be possible to use +ELLIPSIS but sometimes I find myself wanting to match the entire string (e.g., when the output is deterministic but looks random).

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-138135: allow ignoring line breaks in doctest output #138136

[serhiy-storchaka]

A simple workaround (used also for unstable output) is to use something like:

>>> fail == "foobar"
True

It could be better to split the tested string on the parts and test them separately. Or use pprint.

If the output is too long, then perhaps doctest is not appropriate tool. The idea of doctests that if tests fail because the expected output should be changed, you simply copy what you got to expected output, with minimal changes.

[AA-Turner]

One example where this might be useful:

>>> seq = [...]
>>> seq
[
    'item 1, item 2', 'item 3',
    'item 4, item 5', 'item 6',
]

Currently, this fails with normalise whitespace, as there is no whitespace between [ and ', or ' and ]. When writing doctest output, it is often nice to intentionally make the lines shorter and more readable, but doctest makes this harder.

A

[picnixz]

Oh yes, being able to put [ on its own line would be good.

[picnixz]

EDIT: my current PR fails to work for Adam's example, so I'm investigating to see how I can make it work properly and not without increasing complexity too much.

[picnixz]

Update: I don't think my approach would really be possible without making guesses. If the expected output is:

[
    'a', ..., 'z',
    'A', ..., 'Z'
]

Then, ignoring linebreaks actually mean ignoring the linebreak after [, after 'z', and after 'Z'. However, if I do that, the collapsed string is.

[<TAB>'a', ..., 'z',<TAB>'A', ..., 'Z']

As you can, there will be extra tabs. It's unfortunately impossible to decide whether the leading spaces in each splitted lines should be removed entirely or not. Consider the following case:

'abc
123'

In this case, the expected string would be 'abcdef' and not 'abc def'. So there is an ambiguity that we cannot resolve. Adam's example would however work if users were to write

['a', ..., 'z',
 'A', ..., 'Z']

Notice that the second lines starts with a whitespace. Unfortunately, this is only a partial solution (the flag works as it should work, namely it removes line breaks), but it is probably not the format a user would want. Maybe we could first improve the docs concerning NORMALIZE_WHITESPACE and recommend using pprint for long outputs?