Test failure summary at end of test no longer shows up failure contents?

[juj]

I started seeing a week or two back, that when multiple tests fail in a suite, the end-of-test-run log summary has started printing out empty lines.

This is seen e.g. at http://clbri.com:8010/api/v2/logs/224467/raw_inline

where the output print looks like

======================================================================
FAIL: test_html5_webgl_create_context_closure (test_browser.browser.test_html5_webgl_create_context_closure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/common.py", line 291, in resulting_test
    return func(self, *args)
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/test_browser.py", line 223, in decorated
    f(self, *args, **kwargs)
    ~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/test_browser.py", line 2579, in test_html5_webgl_create_context
    self.btest_exit('webgl_create_context.cpp', cflags=args + ['-lGL'])
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/browser_common.py", line 904, in btest_exit
    return self.btest(filename, *args, **kwargs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/browser_common.py", line 945, in btest
    self.run_browser(html_file, expected=['/report_result?' + e for e in expected], timeout=timeout)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/browser_common.py", line 857, in run_browser
    raise e
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/browser_common.py", line 848, in run_browser
    self.assertContained(expected, output)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/clb/buildbot/h12dsi-linux-mint22/emscripten_linux_x64/build/emscripten/main/test/common.py", line 1069, in assertContained
    self.fail("Expected to find '%s' in '%s', diff:\n\n%s\n%s" % (
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      limit_size(values[0]), limit_size(string), limit_size(diff),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      additional_info,
      ^^^^^^^^^^^^^^^^
    ))
    ^^
AssertionError: Expected to find '/report_result?exit:0
' in '[no http server activity]
', diff:

--- expected
+++ actual
@@ -1 +1 @@
-/report_result?exit:0
+[no http server activity]



======================================================================
FAIL: test_cubegeom_glew_wasmfs (test_browser.browser.test_cubegeom_glew_wasmfs)
----------------------------------------------------------------------

======================================================================
FAIL: test_webgl_with_closure (test_browser.browser.test_webgl_with_closure)
----------------------------------------------------------------------

======================================================================
FAIL: test_glfw3_closure (test_browser.browser.test_glfw3_closure)
----------------------------------------------------------------------

======================================================================
FAIL: test_gles2_emulation_normal (test_browser.browser.test_gles2_emulation_normal)
----------------------------------------------------------------------

======================================================================
FAIL: test_html5_webgl_create_context_no_antialias_closure (test_browser.browser.test_html5_webgl_create_context_no_antialias_closure)
----------------------------------------------------------------------

======================================================================
FAIL: test_closure_in_web_only_target_environment_webgl (test_browser.browser.test_closure_in_web_only_target_environment_webgl)
----------------------------------------------------------------------

======================================================================
FAIL: test_webgl2_closure (test_browser.browser.test_webgl2_closure)
----------------------------------------------------------------------

======================================================================
FAIL: test_glfw_joystick (test_browser.browser.test_glfw_joystick)















Ran 370 tests in 693.492s

FAILED (failures=32, skipped=185)

Some of the tests print the error reason like they used to, but some of the tests are now displaying an empty error log, and the last test a cut-off error log. This looks like a regression of some sorts?

[sbc100]

Does running with --vebose (to disable the SingleLineTestRunner) or CI=1 to enable the XMLTestRunner fix the issue?

Do you know if those other tests actually failing or are they being cut short due to --failfast or --max-failures?

[juj]

Good questions - I'll try some of these flags on my CI to see if those have an effect.