Test setup failures with Bash 5.1: AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''

[hroncok]

Hello. In Fedora Rawhide (development branch), we have updated to Bash 5.1. The tests here fail to setup with:

======================================================================
FAIL: test_comp_point (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_completion_environment (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_continuation (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_debug_output (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_double_quoted_completion (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_exclamation_in_double_quotes (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_one_space_after_exact (test.test.TestBash)
Test exactly one space is appended after an exact match.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_parse_special_characters (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_parse_special_characters_dollar (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_partial_completion (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_quoted_exact (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_quoted_space (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_quotes (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_simple_completion (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_single_quoted_completion (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_single_quotes_in_double_quotes (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_single_quotes_in_single_quotes (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_special_characters (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_special_characters_double_quoted (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_special_characters_single_quoted (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_temp_file (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_unquoted_space (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_wordbreak_chars (test.test.TestBash)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_comp_point (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_completion_environment (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_console_script_module (test.test.TestBashGlobal)
Test completing a console_script for a module.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_console_script_module_wheel (test.test.TestBashGlobal)
Test completing a console_script for a module from a wheel.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_console_script_package (test.test.TestBashGlobal)
Test completing a console_script for a package.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_console_script_package_wheel (test.test.TestBashGlobal)
Test completing a console_script for a package from a wheel.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_continuation (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_debug_output (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_double_quoted_completion (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_exclamation_in_double_quotes (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_one_space_after_exact (test.test.TestBashGlobal)
Test exactly one space is appended after an exact match.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_parse_special_characters (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_parse_special_characters_dollar (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_partial_completion (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_python_completion (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_python_filename_completion (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_python_module (test.test.TestBashGlobal)
Test completing a module run with python3 -m.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_python_not_executable (test.test.TestBashGlobal)
Test completing a script that cannot be run directly.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_quoted_exact (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_quoted_space (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_quotes (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_simple_completion (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_single_quoted_completion (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_single_quotes_in_double_quotes (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_single_quotes_in_single_quotes (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_special_characters (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_special_characters_double_quoted (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_special_characters_single_quoted (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_temp_file (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_unquoted_space (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

======================================================================
FAIL: test_wordbreak_chars (test.test.TestBashGlobal)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/argcomplete-1.12.0/test/test.py", line 1206, in setUp
    self.assertEqual(output, '')
AssertionError: '\x1b[?2004l\r\x1b[?2004h' != ''
- 

----------------------------------------------------------------------
Ran 144 tests in 84.144s

FAILED (failures=54, skipped=2, expected failures=9)

I unfortunately fail to reproduce this outside of our build environment :(

We have 1.12.0.

[hroncok]

Apparently, this has something to do with bracketed paste mode which has been enabled by default in Bash 5.1.

[hroncok]

Most likely a problem in pexpect pexpect/pexpect#669

[kislyuk]

Thanks for tracking this, we'll wait to see how the pexpect maintainers react and bump the required version as appropriate. If there's a workaround available for how to configure pexpect, let us know.

[hroncok]

In Fedora, I've applied this workaround:

echo "set enable-bracketed-paste off" > .inputrc
export INPUTRC=$PWD/.inputrc

Since pexpect is used in tests only, I know that this is safe: Users of argcomplete with bracketed-paste enabled won't get the extra escape sequences from argcomplete.

[tony]

Similarly, I ran into this.

This workaround helped

echo "set enable-bracketed-paste off" > .inputrc
export INPUTRC=$PWD/.inputrc

But that's really not enough. This should be working out of the box for users.

This is creating a huge barrier for users who want to contribute basic things. They'll be blocked by this and burn precious time - if that lost time can be avoided - we should make it a default setting in the test