Some tests can not be run

[dvzrv]

Hi! When packaging 1.1.1 for Arch Linux I attempted to run the test suite.
As I'm providing all the ruby dependencies, I don't have to build anything before testing and just start rspec directly:

export EDITORCONFIG_VIM_EXTERNAL_CORE=/usr/bin/editorconfig
export DISPLAY=:99.0
xvfb-run rspec tests/plugin/spec/editorconfig_spec.rb

Unfortunately a lot of the tests are unable to run because the files do not exist. I have already copied a version of the editorconfig-plugin-tests in place, so that the missing tests due to the gitmodules are not a problem.
Anyways, I still have to apply the following patch to be able to run (the remaining) test suite successfully (it otherwise gets stuck at high CPU load and never returns and has to be killed manually):

diff -ruN a/tests/plugin/spec/editorconfig_spec.rb b/tests/plugin/spec/editorconfig_spec.rb
--- a/tests/plugin/spec/editorconfig_spec.rb	2020-06-02 03:15:12.000000000 +0200
+++ b/tests/plugin/spec/editorconfig_spec.rb	2020-06-03 20:40:18.039124754 +0200
@@ -30,14 +30,14 @@
       vim.kill
     end
 
-    describe '#all' do
-      it '3_space.py' do
-        test_editorconfig vim, '3_space.txt',
-          expandtab: '1',
-          shiftwidth: '3',
-          tabstop: '3'
-      end
-    end
+#    describe '#all' do
+#      it '3_space.py' do
+#        test_editorconfig vim, '3_space.txt',
+#          expandtab: '1',
+#          shiftwidth: '3',
+#          tabstop: '3'
+#      end
+#    end
 
     it '4_space.py' do
       test_editorconfig vim, '4_space.py',
@@ -76,55 +76,55 @@
         fileformat: 'unix'
     end
 
-    it 'crlf.txt' do
-      test_editorconfig vim, 'crlf.txt',
-        fileformat: 'dos'
-    end
-
-    it 'cr.txt' do
-      test_editorconfig vim, 'cr.txt',
-        fileformat: 'mac'
-    end
-
-    it 'utf-8.txt' do
-      test_editorconfig vim, 'utf-8.txt',
-        fileencoding: 'utf-8',
-        bomb: '0'
-    end
-
-    it 'utf-8-bom.txt' do
-      test_editorconfig vim, 'utf-8-bom.txt',
-        fileencoding: 'utf-8',
-        bomb: '1'
-    end
-
-    it 'utf-16be.txt' do
-      test_editorconfig vim, 'utf-16be.txt',
-        fileencoding: 'utf-16'
-    end
-
-    it 'utf-16le.txt' do
-      test_editorconfig vim, 'utf-16le.txt',
-        fileencoding: 'utf-16le'
-    end
-
-    it 'latin1.txt' do
-      test_editorconfig vim, 'latin1.txt',
-        fileencoding: 'latin1'
-    end
+#    it 'crlf.txt' do
+#      test_editorconfig vim, 'crlf.txt',
+#        fileformat: 'dos'
+#    end
+
+#    it 'cr.txt' do
+#      test_editorconfig vim, 'cr.txt',
+#        fileformat: 'mac'
+#    end
+
+#    it 'utf-8.txt' do
+#      test_editorconfig vim, 'utf-8.txt',
+#        fileencoding: 'utf-8',
+#        bomb: '0'
+#    end
+
+#    it 'utf-8-bom.txt' do
+#      test_editorconfig vim, 'utf-8-bom.txt',
+#        fileencoding: 'utf-8',
+#        bomb: '1'
+#    end
+
+#    it 'utf-16be.txt' do
+#      test_editorconfig vim, 'utf-16be.txt',
+#        fileencoding: 'utf-16'
+#    end
+
+#    it 'utf-16le.txt' do
+#      test_editorconfig vim, 'utf-16le.txt',
+#        fileencoding: 'utf-16le'
+#    end
+
+#    it 'latin1.txt' do
+#      test_editorconfig vim, 'latin1.txt',
+#        fileencoding: 'latin1'
+#    end
 
     # insert_final_newline by PreserveNoEOL tests are omitted, since they are not supported
-    if vim.echo("exists('+fixendofline')") == '1'
-      it 'with_newline.txt' do
-        test_editorconfig vim, 'with_newline.txt',
-          fixendofline: '1'
-      end
-
-      it 'without_newline.txt' do
-        test_editorconfig vim, 'without_newline.txt',
-          fixendofline: '0'
-      end
-    end
+#    if vim.echo("exists('+fixendofline')") == '1'
+#      it 'with_newline.txt' do
+#        test_editorconfig vim, 'with_newline.txt',
+#          fixendofline: '1'
+#      end
+#
+#      it 'without_newline.txt' do
+#        test_editorconfig vim, 'without_newline.txt',
+#          fixendofline: '0'
+#      end
+#    end
   end
 end

[cxw42]

1.

That is very odd! What results do you get from

$ git submodule update --init --recursive
$ ./tests/travis-test.sh plugin

? That's what Travis does. I don't see any obvious reason why those particular tests should hang.

2.

export EDITORCONFIG_VIM_EXTERNAL_CORE=/usr/bin/editorconfig

What core have you installed in /usr/bin? The editorconfig executable that comes with this plugin (tests/core/editorconfig) should never be installed --- it is only for testing.

3.

Do the core tests pass?

[cxw42]

Any updates? If not, I will close this issue. You can always reopen later if necessary!

[dvzrv]

Argh, sorry, I forgot about this ticket!

1.

That is very odd! What results do you get from

$ git submodule update --init --recursive
$ ./tests/travis-test.sh plugin

? That's what Travis does. I don't see any obvious reason why those particular tests should hang.

I build from the source tarball, not from a git clone.
As mentioned above I provide the ruby dependencies myself, so that I don't have to build them during packaging (package build and test environments are supposed to work without internet connection).

So in this case I specifically only run rspec. Unfortunately some of the files seem to be missing.

2.

export EDITORCONFIG_VIM_EXTERNAL_CORE=/usr/bin/editorconfig

What core have you installed in /usr/bin? The editorconfig executable that comes with this plugin (tests/core/editorconfig) should never be installed --- it is only for testing.

The vim-editorconfig package relies on the editorconfig-core-c package.
I don't build the included editorconfig binary, because we have it as a package.

3.

Do the core tests pass?

It doesn't look like we are running any tests for the editorconfig-core-c package. I can get in touch with the package maintainer to ask about that though.

[cxw42]

No worries :) .

I build from the source tarball

How do you make your source tarball, or where do you download it from? Perhaps the way we make the source tarball needs to change.

The vim-editorconfig package relies on the editorconfig-core-c package

As of v1.0.0-beta, this is no longer the case. The released v1.1.1 uses the internal Vimscript core by default. Therefore, I believe you can safely remove that dependency from the Arch package.

Do the core tests pass?

It doesn't look like we are running any tests for the editorconfig-core-c package

Since editorconfig-vim includes its own core, the tests test both the plugin and that core. Sorry for the confusion! I was referring to the core tests in editorconfig-vim itself, not any tests of editorconfig-core-c.

[cxw42]

@dvzrv Do you need further support for this issue?

[dvzrv]

How do you make your source tarball, or where do you download it from? Perhaps the way we make the source tarball needs to change.

I don't. I'm relying on the ones that are auto-generated by this project's github repository upon tagging.
I can not rely on the .zip that is attached to the 1.1.1 release, because it does not extract to a directory but extracts all files into a top-level directory. If you can, please create a release source tarball that contains a directory (e.g. editorconfig-vim-1.1.1) and below that the files of the repository including submodules.

When it comes to the tests I'm using the specific commit mentioned in the initial ticket description of https://github.com/editorconfig/editorconfig-plugin-tests/
As there hasn't been any new development on that repository since then, I don't think anything has changed there.

As of v1.0.0-beta, this is no longer the case. The released v1.1.1 uses the internal Vimscript core by default. Therefore, I believe you can safely remove that dependency from the Arch package.

Thanks! I'll do that!

Since editorconfig-vim includes its own core, the tests test both the plugin and that core. Sorry for the confusion! I was referring to the core tests in editorconfig-vim itself, not any tests of editorconfig-core-c.

Ah interesting!

[cxw42]

Weird! Sorry to say I do not know off-hand. If it occurs again, please post the exact error messages, and exact package versions, as well as the commands you ran.

Random thoughts:

• I see that we are using xvfb start& in .travis.yml, and that you used xvfb-run. Maybe there is some difference between those.
• I wonder if perhaps Vim is displaying a "press Enter" message, and that's what's causing the hang. We could perhaps try setting lines=999 cols=999 cmdheight=10 before calling vim.edit from the spec file.

[dvzrv]

I have now also tried running (similar to the CI script) you use from a clone of this repository and the test still hangs.

  export EDITORCONFIG_VIM_EXTERNAL_CORE=/usr/bin/editorconfig
  # export EDITORCONFIG_VIM_EXTERNAL_CORE=tests/core/editorconfig  <- also hangs
  # export DISPLAY=:99.0 <- makes no difference
  xvfb-run rspec tests/plugin/spec/editorconfig_spec.rb

Not really sure how to further proceed here, as it seems that this can not be run elsewhere but CI 😢
We have:

• ruby-rake 13.0.6
• ruby-rspec 3.12.0
• ruby-vimrunner 0.3.4
• xorg-server-xvfb 21.1.8
• editorconfig-core-c 0.12.6
• gvim (pulled in by ruby-vimrunner) 9.0.1420

Tests always hang and I have to disable them to be able to create the package:

Testing default
Testing with express vim_core mode
Testing with fallback to vim_core mode
Testing with external_command /usr/bin/editorconfig
......

[cxw42]

I just tried with ZIP files, and it worked fine when using the deps I got using Bundler. I'm afraid I can't help you if there's a dependency issue :( --- I don't have enough Ruby experience.

This worked for me:

1. Log in to X and open a terminal.
2. At https://github.com/editorconfig/editorconfig-vim, hit Code | Download ZIP to download editorconfig-vim-master.zip. (at 7f4e4df)
3. At https://github.com/editorconfig/editorconfig-plugin-tests, hit Code | Download ZIP to download editorconfig-plugin-tests-master.zip (at cb7ae15)
4. unzip ~/Downloads/editorconfig-vim-master.zip
5. cd editorconfig-vim-master/tests/plugin/spec/
6. unzip ~/Downloads/editorconfig-plugin-tests-master.zip
7. rmdir plugin_tests/
8. mv editorconfig-plugin-tests-master/ plugin_tests
9. cd ../../.. (go back to editorconfig-vim-master)
10. ./tests/travis-test.sh plugin

All tests run, and pass.

Note: I don't even have xvfb installed on my system!

$ apt list --installed xvfb
Listing... Done

editorconfig-vim-master$ ./tests/travis-test.sh plugin

<snip source>
+ [[ ! -n '' ]]
+ [[ -n plugin ]]
+ export TEST_WHICH=plugin
+ TEST_WHICH=plugin
<snip the `cat` contents>
+ [[ ! -n plugin ]]
<snip source>
+ [[ plugin = \p\l\u\g\i\n ]]
+ [[ ! -n '' ]]
+++ dirname ./tests/travis-test.sh
++ cd ./tests
++ pwd
+ here=/<snip>/editorconfig-vim-master/tests
+ export BUNDLE_GEMFILE=/<snip>/editorconfig-vim-master/tests/plugin/Gemfile
+ BUNDLE_GEMFILE=/<snip>/editorconfig-vim-master/tests/plugin/Gemfile
+ bundle install --jobs=3 --retry=3 --deployment
[DEPRECATED] The `--deployment` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set deployment 'true'`, and stop using this flag                                                                                                                               
Fetching gem metadata from https://rubygems.org/.
Fetching rake 12.3.3
Installing rake 12.3.3
Using bundler 2.1.4
Fetching diff-lcs 1.2.5
Fetching rspec-support 3.4.1
Installing diff-lcs 1.2.5
Fetching vimrunner 0.3.1
Installing vimrunner 0.3.1
Installing rspec-support 3.4.1
Fetching rspec-expectations 3.4.0
Fetching rspec-core 3.4.1
Installing rspec-expectations 3.4.0
Fetching rspec-mocks 3.4.0
Installing rspec-core 3.4.1
Installing rspec-mocks 3.4.0
Fetching rspec 3.4.0
Installing rspec 3.4.0
Bundle complete! 3 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `./tests/plugin/vendor/bundle`
+ export EDITORCONFIG_VIM_EXTERNAL_CORE=tests/core/editorconfig
+ EDITORCONFIG_VIM_EXTERNAL_CORE=tests/core/editorconfig
+ bundle exec rspec tests/plugin/spec/editorconfig_spec.rb
Testing default
Testing with express vim_core mode
Testing with fallback to vim_core mode
Testing with external_command tests/core/editorconfig
Testing with express vim_core mode
................................................................................

Finished in 10.25 seconds (files took 1.18 seconds to load)
80 examples, 0 failures

[dvzrv]

The 1.2.0 release appears to have fixed this for me finally.