make test fails on master

[liushapku]

System Infomation

commit: bd97577
system: ubuntu 18.04
shell: both bash and zsh
sympton: make test hangs after printing traceback of lua
details: see output at the end.

Questions

1. how to solve it?
2. If not easy without more investigation, how to disable tests from system_spec.lua

Log

[----------] Running tests from .../neovim/test/functional/eval/system_spec.lua
[ RUN      ] system() command passed as a List sets v:shell_error if cmd[0] is not executable: 3.05 ms OK
[ RUN      ] system() command passed as a List parameter validation does NOT modify v:shell_error: 4.28 ms OK
[ RUN      ] system() command passed as a List quotes arguments correctly #5280: 6.06 ms OK
[ RUN      ] system() command passed as a List calls executable in $PATH: 25.89 ms OK
[ RUN      ] system() command passed as a List does NOT run in shell: 3.92 ms OK
[ RUN      ] system() sets v:shell_error: 18.30 ms OK
[ RUN      ] system() executes shell function `echo` and waits for its return: 3.96 ms OK
[ RUN      ] system() executes shell function prints verbose information: ERR
test/functional/ui/screen.lua:531: Failed to match any screen lines.
Expected (anywhere): "Executing command: "'/[^']*sh' '%-c' 'echo hi'""
Actual:
  |                                                                        |
  |~                                                                       |
  |~                                                                       |
  |~                                                                       |
  |~                                                                       |
  |~                                                                       |
  |~                                                                       |
  |~                                                                       |
  |                                                                        |
  |Executing command: "'zsh' '-c' 'echo hi'"                               |
  |                                                                        |
  |hi                                                                      |
  |                                                                        |
  |Press ENTER or type command to continue^                                 |

stack traceback:
	test/functional/ui/screen.lua:531: in function '_wait'
	test/functional/ui/screen.lua:336: in function 'expect'
	.../neovim/test/functional/eval/system_spec.lua:211: in function <...pos/neovim_official/test/functional/eval/system_spec.lua:205>

[liushapku]

If I use v0.3.1, it also fails but the test continues after detecting the mismatch and can run to the end.

Three tests fails:

[  FAILED  ] 3 tests, listed below:
[  FAILED  ] .../neovim/test/functional/eval/executable_spec.lua @ 56: executable() exec-bit set, unqualified and not in $PATH
.../neovim/test/functional/eval/executable_spec.lua:58: Expected objects to be the same.
Passed in:
(number) 1
Expected:
(number) 0

stack traceback:
	.../neovim/test/functional/eval/executable_spec.lua:58: in function <.../neovim/test/functional/eval/executable_spec.lua:56>

[  FAILED  ] .../neovim/test/functional/eval/system_spec.lua @ 351: system() with a program that doesn't close stdout will exit properly after passing input
.../neovim/test/functional/eval/system_spec.lua:353: Expected objects to be the same.
Passed in:
(string) 'Error: target STRING not available
'
Expected:
(string) 'clip-data'

stack traceback:
	.../neovim/test/functional/eval/system_spec.lua:353: in function <.../neovim/test/functional/eval/system_spec.lua:351>

[  FAILED  ] .../neovim/test/functional/eval/system_spec.lua @ 536: systemlist() with a program that doesn't close stdout will exit properly after passing input
.../neovim/test/functional/eval/system_spec.lua:539: Expected objects to be the same.
Passed in:
(table) {
 *[1] = 'Error: target STRING not available' }
Expected:
(table) {
 *[1] = 'clip'
  [2] = 'data' }

[blueyed]

The second failure looks like #7958.

@liushapku
Does the first still happen for you?

[blueyed]

For the first issue:

The following patch triggers a test failure, where it then hangs.

after_each is not called then.

diff --git i/test/functional/eval/system_spec.lua w/test/functional/eval/system_spec.lua
index 5cbf34365..31cda2b99 100644
--- i/test/functional/eval/system_spec.lua
+++ w/test/functional/eval/system_spec.lua
@@ -129,7 +129,9 @@ describe('system()', function()
     end)

     after_each(function()
+      print("after_each")
       screen:detach()
+      print("detached")
     end)

     if iswin() then
@@ -205,10 +207,12 @@ describe('system()', function()
     it('prints verbose information', function()
       screen:try_resize(72, 14)
       feed(':4verbose echo system("echo hi")<cr>')
+      -- feed('<cr>')
+      eq(1, 2)
       if iswin() then
         screen:expect{any=[[Executing command: "'cmd.exe' '/s' '/c' '"echo hi"'"]]}
       else

% TEST_FILE=test/functional/eval/system_spec.lua make functionaltest
[ RUN      ] system() executes shell function `echo` and waits for its return: 6.89 ms OK
after_each
detached
[ RUN      ] system() executes shell function prints verbose information: FAIL
test/functional/eval/system_spec.lua:211: Expected objects to be the same.
Passed in:
(number) 2
Expected:
(number) 1

stack traceback:
        (tail call): ?
        test/functional/eval/system_spec.lua:211: in function <test/functional/eval/system_spec.lua:207>

I could not find where this is hanging.

(But I've digged into this after running in an actual failure there (in Docker))

[blueyed]

First issue should be fixed in #10532.

[blueyed]

If I use v0.3.1, it also fails but the test continues after detecting the mismatch and can run to the end.

In v0.3.1 the test does not exist, so cannot hang. It was added in a225374 (included in v0.3.2).

The hang can be reproduced there also already, using just:

    it('prints verbose information', function()
      feed(':4verbose echo system("echo hi")<cr>')
      eq(1,2)
    end)