Test failures on on Windows: editorconfig returned a nonzero exit code

[xuhdev]

After 3d439da, multiline tests start to fail.

Example:

118/190 Testing: tab_width_default_ML
118/190 Test: tab_width_default_ML
Command: "C:/Program Files (x86)/CMake/bin/cmake.exe" "-D" "EDITORCONFIG_CMD=C:/projects/editorconfig-core-c/bin/x64-static/core/bin/editorconfig" "-D" "ECARGS:LIST=-f;tab_width_default.in;C:/projects/editorconfig-core-c/tests/properties/test.c" "-P" "C:/projects/editorconfig-core-c/tests/cmake/ec_sort.cmake"
Directory: C:/projects/editorconfig-core-c/bin/x64-static/core/tests/properties
"tab_width_default_ML" start time: Jul 01 01:11 Coordinated Universal Time
Output:
----------------------------------------------------------
CMake Error at C:/projects/editorconfig-core-c/tests/cmake/ec_sort.cmake:55 (message):
  C:/projects/editorconfig-core-c/bin/x64-static/core/bin/editorconfig
  -f;tab_width_default.in;C:/projects/editorconfig-core-c/tests/properties/test.c
  returned a nonzero exit code
<end of output>
Test time =   0.02 sec
----------------------------------------------------------
Test Fail Reason:
Required regular expression not found.Regex=[^[
]*indent_size=4[ 	]*[
]+indent_style=space[ 	]*[
]+tab_width=4[	
]*$
]
"tab_width_default_ML" end time: Jul 01 01:11 Coordinated Universal Time
"tab_width_default_ML" time elapsed: 00:00:00
----------------------------------------------------------

Here shows two Windows CI tests on editorconfig-core-c, one with the test submodule right before the commit, and one right after the commit.

@cxw42 Could you look into this, please?

[xuhdev]

@cxw42 #9 is likely a different issue, but this issue seems more urgent (CMake starts to fail on Windows after this commit)

[cxw42]

@xuhdev I see this is on the test-windows branch. What is the purpose of this branch? I'd like to know what the end goal is.

Both of the links you sent show as failures when I look at them -

• https://ci.appveyor.com/project/xuhdev/editorconfig-core-c/builds/25649628/job/jebgc8mbey5imlbg?fullLog=true#L2296
• https://ci.appveyor.com/project/xuhdev/editorconfig-core-c/builds/25649754/job/f8rlmy0rek0ptc9h?fullLog=true#L2770

I see the tab_width_default_ML failure in the second, but I don't see that test in the first at all.

Would you please double-check and let me know if those are the correct builds to look at? Thanks!

[xuhdev]

@cxw42 Because tab_width_default_ML was renamed from tab_width_default in 3d439da -- The first one succeeded but the second one failed. These two links correspond to two PRs:

• [DO NOT MERGE] Test Windows CI: before "Rewrote multiline tests to use new harness" editorconfig-core-c#59
• [DO NOT MERGE] Test Windows CI: after "Rewrote multiline tests to use new harness" editorconfig-core-c#60

Now it looks like the first link also failed one multiline test (which I did not notice before), but the second failed a lot of them.

[cxw42]

Thanks for the extra info. Is the test-windows branch just updating editorconfig-core-c to the latest tests?

[xuhdev]

@cxw42 No, the master has the latest test code. test-windows is a branch used in one of the above PRs.

[cxw42]

@xuhdev Would you please assign this to me? Thanks!

[xuhdev]

@cxw42 Done. Would like to be a developer of EditorConfig? I can add you in the Developer Team.

[cxw42]

@xuhdev Thank you for asking! I would be honored.

This problem seems to be related to the path to editorconfig.exe in some tests - it is in build\bin and core\bin\Release, but the meta test is looking in core\bin (example). I will see if I can figure out why.

[cxw42]

@xuhdev I have a proposed fix at https://github.com/cxw42/editorconfig-core-test/tree/cxwtest . It is not ready for a PR yet. However, would you be willing to try it out with a few cores and see what you find?

• C core CI pass at https://ci.appveyor.com/project/cxw42/editorconfig-core-c/builds/26125944 . (With changes discussed below.)
• Passes on editorconfig-vim on Cygwin and Windows on my local system.
• Vim tests on CI in progress at https://ci.appveyor.com/project/cxw42/editorconfig-vim/builds/26127101

In the C core and core tests, I had to change the way EDITORCONFIG_CMD works in order to handle Debug vs. Release builds. In the new tests, =foo tells the core tests to use the result of CMake target foo as the editorconfig executable. Any name not starting with = has the old behaviour. I then use that = to switch to the COMMAND mode of add_test(), in which CMake itself figures out the executable name. The C core now uses the '=' form (https://github.com/cxw42/editorconfig-core-c/tree/cxwtest).

I also had to change the backslash escaping in the filetree test path_separator_backslash_in_cmd_line_regex. Edit That at least needs some sort of fixing (see here), but I am not completely sure yet what the right fix should be.

If you have any questions or comments, please let me know. Otherwise, I will clean up the repos in question and submit PRs (core-test first, then core-c).

[xuhdev]

@cxw42 Sure. Could you send a WIP PR? We can look at the changes together there and it's probably easier.

[cxw42]

An example of the backslash problem - with cmake 3.14.5, on Cygwin, CMake generates the test differently for the C core than for the Vim plugin+core (!) (scroll right):

editorconfig-vim/tests/core/build/tests/filetree/CTestTestfile.cmake
36:add_test(path_separator_backslash_in_cmd_line "/home/cxw/proj/editorconfig-vim/tests/core/editorconfig" "-f" "path_separator.in" "/home/cxw/proj/editorconfig-vim/tests/core/tests/filetree\\path\\separator")
                                                                                                                                                                                              ^^ two backslashes
37:set_tests_properties(path_separator_backslash_in_cmd_line PROPERTIES  PASS_REGULAR_EXPRESSION "^[

editorconfig-core-c/build/tests/filetree/CTestTestfile.cmake
36:add_test(path_separator_backslash_in_cmd_line "/home/cxw/proj/editorconfig-core-c/build/bin/editorconfig.exe" "-f" "path_separator.in" "/home/cxw/proj/editorconfig-core-c/tests/filetree\\\\path\\\\separator")
                                                                                                                                                                                            ^^^^ four backslashes
37:set_tests_properties(path_separator_backslash_in_cmd_line PROPERTIES  PASS_REGULAR_EXPRESSION "^[
(END)

I am going to try generating the path string by concatenating strings with fixed ASCII values and see what happens.

[xuhdev]

Fixed by 48610d4