Tests do not work under Windows

[MartinJohns]

I'm trying to run the unit tests for step 0 under Windows, using cygwin. But there seems to be an issue with the line endings. I'm doing what is described in step 0 - I write the prompt, read the input, write the input out and repeat.

Here's the output of the test call:

$ make test^quux^step0
----------------------------------------------
Testing test^quux^step0, step file: quux/step0_repl.exe, test file: tests/step0_repl.mal
Running: ../runtest.py  ../tests/step0_repl.mal -- ../quux/step0_repl.exe
 -> ['','hello world\r'] -> FAIL (line 2):
    Expected : ['hello world\r\r\nhello world\r', 'hello world\r\r\nhello world\r\r\nhello world\r']
    Got      : 'hello world\r\n\r\nhello world\r'
 -> ['','abcABC123\r'] -> FAIL (line 5):
    Expected : ['abcABC123\r\r\nabcABC123\r', 'abcABC123\r\r\nabcABC123\r\r\nabcABC123\r']
    Got      : 'user> abcABC123\r\n\r\nabcABC123\r'
 -> ['',';:() []{}"\'*\r'] -> FAIL (line 8):
    Expected : [';:() []{}"\'*\r\r\n;:() []{}"\'*\r', ';:() []{}"\'*\r\r\n;:() []{}"\'*\r\r\n;:() []{}"\'*\r']
    Got      : 'user> ;:() []{}"\'*\r\n\r\n;:() []{}"\'*\r'
 -> ['','hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r'] -> FAIL (line 13):
    Expected : ['hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\r\nhello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r', 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\r\nhello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\r\nhello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r']
    Got      : 'user> hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\n\r\nhello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r'
FAILURES: 4
Makefile:156: recipe for target 'test^quux^step0' failed
make: *** [test^quux^step0] Error 2

Got any idea on how to solve this?

[kanaka]

@MartinJohns I can't easily test this, but can you try the following hack to see if it makes things work for you?

diff --git a/runtest.py b/runtest.py
index 436b769..c9d2ea8 100755
--- a/runtest.py
+++ b/runtest.py
@@ -12,9 +12,12 @@ import pty, array, fcntl, termios

 IS_PY_3 = sys.version_info[0] == 3

-# TODO: do we need to support '\n' too
-sep = "\r\n"
-#sep = "\n"
+import platform
+if platform.system().find("CYGWIN_NT") >= 0:
+    # TODO: this is weird, is this really right on Cygwin?
+    sep = "\n\r\n"
+else:
+    sep = "\r\n"
 rundir = None

 parser = argparse.ArgumentParser(

[kanaka]

@MartinJohns had a chance to test the above change?

[epylar]

@kanaka This patch (off master) works on Cygwin 64 (and probably 32, but I
didn't check); however, due to the nature of the change, I'd recommend you
test it on non-Cygwin. This is for make test^python^step0 -- confirmed
that without the change, the test fails; with the change, it passes.

diff --git a/runtest.py b/runtest.py
index 436b769..932dd82 100755
--- a/runtest.py
+++ b/runtest.py
@@ -12,9 +12,11 @@ import pty, array, fcntl, termios

 IS_PY_3 = sys.version_info[0] == 3

-# TODO: do we need to support '\n' too
-sep = "\r\n"
-#sep = "\n"
+import platform
+if platform.system().find("CYGWIN_NT") >= 0:
-    sep = "\r\n"
  +else:
-    sep = "\n"
  rundir = None
  
  parser = argparse.ArgumentParser(
  @@ -120,7 +122,7 @@ class Runner():
  
  args = parser.parse_args(sys.argv[1:])
  -test_data = args.test_file.read().split('\n')
  +test_data = args.test_file.read().split(sep)
  
  if args.rundir: os.chdir(args.rundir)

[gilch]

The "\n\r\n" doesn't seem right. Git for Windows will automatically convert line endings to "\r\n", so the .mal test files themselves will change when cloned on a Windows machine. If the test runner is assuming these files always have the Unix-style "\n" endings, that could explain the problem.

I tried manually changing the tests/step0_repl.mal file to Unix line endings and then the make test^python^step0 tests pass if I use Cygwin's python. Tests using Windows' py -2 and py -3 still aren't passing on my setup, but this appears to be for unrelated reasons.

[gilch]

The unrelated reason seems to be that Python for Windows doesn't include the readline module, but the Python MAL implementation requires it. An example without this dependency could be illuminating.

This one-liner will pass the step0 test using Cygwin's python, given a Unix-terminated test file. (It does not require readline.)

while 1:print raw_input("user> ")

This still does not work when using Python for Windows:

----------------------------------------------
Testing test^python^step0, step file: python/step0_repl.py, test file: tests/ste
p0_repl.mal
Running: ../runtest.py   ../tests/step0_repl.mal -- py -2 ../python/step0_repl.p
y
Did not one of following prompt(s): ['user> ', 'mal-user> ']
    Got      : ''
Makefile:199: recipe for target 'test^python^step0' failed
make: *** [test^python^step0] Error 1

Cygwin's python3 passes using

while 1:print(input("user> "))

But and Windows' py -3 has a different error:

----------------------------------------------
Testing test^python^step0, step file: python/step0_repl.py, test file: tests/ste
p0_repl.mal
Running: ../runtest.py   ../tests/step0_repl.mal -- py -3 ../python/step0_repl.p
y
TEST: hello world -> ['',hello world] -> FAIL (line 2):
    Expected : ['hello world\r\nhello world', 'hello world\r\nhello world\r\nhel
lo world']
    Got      : 'hello world\r\nhello world\r'
TEST: abcABC123 -> ['',abcABC123] -> FAIL (line 5):
    Expected : ['abcABC123\r\nabcABC123', 'abcABC123\r\nabcABC123\r\nabcABC123']

    Got      : 'abcABC123\r\nabcABC123\r'
TEST: []{}"'* ;:() -> ['',[]{}"'* ;:()] -> FAIL (line 8):
    Expected : ['[]{}"\'* ;:()\r\n[]{}"\'* ;:()', '[]{}"\'* ;:()\r\n[]{}"\'* ;:(
)\r\n[]{}"\'* ;:()']
    Got      : '[]{}"\'* ;:()\r\n[]{}"\'* ;:()\r'
TEST: hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 01234567
89 (;:() []{}"'* ;:() []{}"'* ;:() []{}"'*) -> ['',hello world abcdefghijklmnopq
rstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:()
[]{}"'*)] -> FAIL (line 13):
    Expected : ['hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWX
YZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\nhello world abcdefg
hijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []
{}"\'* ;:() []{}"\'*)', 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOP
QRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\nhello world
 abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'*
 ;:() []{}"\'* ;:() []{}"\'*)\r\nhello world abcdefghijklmnopqrstuvwxyz ABCDEFGH
IJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)']
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXY
Z 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r\nhello world abcdefgh
ijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{
}"\'* ;:() []{}"\'*)\r'
TEST RESULTS (for ../tests/step0_repl.mal):
    0: soft failing tests
    4: failing tests
    0: passing tests
    4: total tests

Makefile:199: recipe for target 'test^python^step0' failed
make: *** [test^python^step0] Error 1

This time it does appear to be an issue with line terminators, but changing back to "\r\n" only makes it worse.

[nukecoder]

I got tests working on Windows Under Cygwin64 withe following couple of changes. (This is a diff from Master)

@@ -125,11 +125,11 @@ class Runner():
                 debug(new_data)
                 if self.no_pty:
                     self.buf += new_data.replace("\n", "\r\n")
                 else:
                     self.buf += new_data
-                self.buf = self.buf.replace("\r\r", "\r")
+                self.buf = self.buf.replace("\r\r", sep)
                 for prompt in prompts:
                     regexp = re.compile(prompt)
                     match = regexp.search(self.buf)
                     if match:
                         end = match.end()
@@ -191,17 +191,18 @@ class TestReader:
                     return True
                 continue
             elif line[0:1] == ";":         # unexpected comment
                 log("Test data error at line %d:\n%s" % (self.line_num, line))
                 return None
-            self.form = line   # the line is a form to send
+            # remove the last character. Writeline will insert the line ending.
+            self.form = line[0:len(line)-1]   # the line is a form to send
 
             # Now find the output and return value
             while self.data:
                 line = self.data[0]
                 if line[0:3] == ";=>":
-                    self.ret = line[3:]
+                    self.ret = line[3:].rstrip() + sep
                     self.line_num += 1
                     self.data.pop(0)
                     break
                 elif line[0:2] == "; ":
                     self.out = self.out + line[2:] + sep

I think part of the problem, as stated above, is that when the repo is clones with Git on Windows, the line endings are changed to \r\n. Adding the conditional sep assignment was useful, but was not carried through on the reads and writes. All tests are working for me now. Not sure how this would affect non-windows users.

[kanaka]

@nukecoder that patch breaks tests on Linux.

If somebody suggests a clean patch that will work on Linux, OS X and Windows (cygwin or native but both would be a plus) I'll be happy to merge it. I don't have a convenient way to test all three (only Linux).

[dubek]

According to @nukecoder 's message another thing we should pay attention to is Git's translation of line endings - apparently on Windows git will auto-translate line endings to \r\n - just to add some more noise to the party.

[inkydragon]

It seems like sep = "\r\n" works well on cygwin now.

cygwin version

$ uname -a
CYGWIN_NT-10.0 wos-ThinkPad 2.11.2(0.329/5/3) 2018-11-08 14:34 x86_64 Cygwin

Tested (passed all tests): awk, bash, make, perl, python, powershell

[kanaka]

Okay, seems like this has been resolved and tested by @inkydragon in #396 (thanks!) so I'm going to close it.