ASCII color chars leaking into xml report file when using verbose flag

[ro-oliveira95]

Problem description

A bug was noticed when displaying failing tests on CI with the -vv flag: the ASCII color characters are being printed inside the detailed diff.

The displayed output comes directly from xml file (generated with the --junitxml flag), which in fact carries the undesired characters as shown in the minimal example below. This happens only when pygments is also installed, with both -v and -vv flags.

This undesired behavior was noticed in all 8.* versions but not in 7.4, so it probably was introduced in 8.0 together with the improved verbose diffs change (changelog).

Output of pip list

Package        Version
-------------- -------
exceptiongroup 1.2.1
iniconfig      2.0.0
packaging      24.0
pip            21.2.4
pluggy         1.5.0
Pygments       2.18.0
pytest         8.2.1
setuptools     58.1.0
tomli          2.0.1

pytest and operating system versions

• SO (uname -a): Linux FLDEVWSV032 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
• Python: 3.10.2
• pytest: 8.0.0, 8.1.0, 8.2.1

Minimal example

test_foo.py:

def test_foo() -> None:
    assert [1,2,3] == [3,2,1]

Running it:

foo.xml:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
	<testsuite name="pytest" errors="0" failures="1" skipped="0" tests="1" time="0.082" timestamp="2024-05-24T14:19:42.491202" hostname="FLDEVWSV032">
		<testcase classname="test_foo" name="test_foo" time="0.055">
			<failure message="assert [1, 2, 3] == [3, 2, 1]&#10;  &#10;  At index 0 diff: #x1B[0m#x1B[94m1#x1B[39;49;00m#x1B[90m#x1B[39;49;00m != #x1B[0m#x1B[94m3#x1B[39;49;00m#x1B[90m#x1B[39;49;00m&#10;  &#10;  Full diff:&#10;  #x1B[0m#x1B[90m #x1B[39;49;00m [#x1B[90m#x1B[39;49;00m&#10;  #x1B[92m+     1,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m&#10;  #x1B[92m+     2,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m&#10;  #x1B[90m #x1B[39;49;00m     3,#x1B[90m#x1B[39;49;00m&#10;  #x1B[91m-     2,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m&#10;  #x1B[91m-     1,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m&#10;  #x1B[90m #x1B[39;49;00m ]#x1B[90m#x1B[39;49;00m">def test_foo() -&gt; None:
&gt;       assert [1,2,3] == [3,2,1]
E       assert [1, 2, 3] == [3, 2, 1]
E         
E         At index 0 diff: #x1B[0m#x1B[94m1#x1B[39;49;00m#x1B[90m#x1B[39;49;00m != #x1B[0m#x1B[94m3#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         
E         Full diff:
E         #x1B[0m#x1B[90m #x1B[39;49;00m [#x1B[90m#x1B[39;49;00m
E         #x1B[92m+     1,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[92m+     2,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     3,#x1B[90m#x1B[39;49;00m
E         #x1B[91m-     2,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[91m-     1,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m ]#x1B[90m#x1B[39;49;00m

test_foo.py:2: AssertionError</failure>
		</testcase>
	</testsuite>
</testsuites

For comparison, without having pygments installed the xml looks like this:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
	<testsuite name="pytest" errors="0" failures="1" skipped="0" tests="1" time="0.028" timestamp="2024-05-24T14:22:36.093080" hostname="FLDEVWSV032">
		<testcase classname="test_foo" name="test_foo" time="0.001">
			<failure message="assert [1, 2, 3] == [3, 2, 1]&#10;  &#10;  At index 0 diff: 1 != 3&#10;  &#10;  Full diff:&#10;    [&#10;  +     1,&#10;  +     2,&#10;        3,&#10;  -     2,&#10;  -     1,&#10;    ]">def test_foo() -&gt; None:
&gt;       assert [1,2,3] == [3,2,1]
E       assert [1, 2, 3] == [3, 2, 1]
E         
E         At index 0 diff: 1 != 3
E         
E         Full diff:
E           [
E         +     1,
E         +     2,
E               3,
E         -     2,
E         -     1,
E           ]

test_foo.py:2: AssertionError</failure>
		</testcase>
	</testsuite>
</testsuites>

If confirmed, I'd be happy to submit a PR fixing this.

[nicoddemus]

Thanks @ro-oliveira95!

[nicoddemus]

I investigated this, unfortunately it seems it is not simple to fix, or at least I cannot figure out a simple solution.

#11661 was implemented so that it is generating the color codes directly as part of the exception message, which is represented just as a string. The JUnitXML plugin then cannot safely remove the color codes from the exception message before writing the XML, because at that point we cannot be sure the color codes in the message were included by pytest or by the code under testing -- for example a color-related library might be testing the strings they output (which include color codes), so stripping them away would definitely be a bad thing to do.

The only solution I think (other than ugly workarounds like stripping color codes from junitxml based on a flag) is to refactor how error messages are produced, so instead of plain strings we can use better structures, for example we could instead of returning only list[str], we could return list[str | ColoredLine], so one can handle ColoredLine explicitly to strip colored codes as needed.

[ro-oliveira95]

Alright @nicoddemus, thanks for digging into this.

Since the fix is not as simple as we'd like, I'll leave my first contribution for another time/issue =)

[leonarduschen]

Hello, I'd like to give this a go.

I'll dig into this and see if I can come up with a quick workaround.

While I'm at it, I'll also think of how I can do the refactoring like @nicoddemus suggested, and will proceed with it if I can't come up with a good workaround for this specific issue.

[prusse-martin]

I got curious (we had some output wit those escapes recently, not the first time).
Apparently we can't add the actual ansi escapes in the xml the string "\x1b[31mA\x1b[0m" (that is a red "A"), that "could" be escaped as &#1B;[31mA&#1B;[0m (currently pytest is escaping it as #1B[31mA#1B[0m).
I say could because \x1b (along with other control characters) is an invalid character to have in xml.
I did test and element tree (and at least a browser) would fail to parse a xml using that character reference (&#1B;).

(I kind of remember to try and use "cdata" with same bad results)

The most obvious options that came to my mind are:

• let it as it is today (the invalid characters are being mangled to be represented in the xml);
• just strip them and get just the "visible" characters (I find some, like delete, is allowed), but @nicoddemus already pointed out how this could be bad;
• strip (as last option) but somewhere add the base64 encoded version of the original output;
• come up and document in pytest docs a way to escape/encode those invalid characters;