non-zero exit code for failures

[meisterluk]

Rationale:
By convention non-zero exit codes indicate failure of terminating programs (Wikipedia: Exit status). This is generally done on all systems, but especially important on UNIX systems using a lot of shell scripting. When automating tasks, instead of parsing stdout and stderr, exit codes often give sufficient information for the system administrator ("did my intent work or not?!").

Now let's consider sikuli.

1. Consider the following sikuli script:
   click("does_not_exist.png")

   Now let's run it:

   % sikuli exitcode.sikuli/
   [info] Sikuli vision engine loaded.
   [info] VDictProxy loaded.
   [error] does_not_exist.png looks like a file, but can't be found on the disk. Assume it's text.
   [info] Text Recognizer inited.
   
   Error: Illegal min or max specification!
   "Fatal error encountered!" == NULL:Error:Assert failed:in file globaloc.cpp, line 75
   #
   # A fatal error has been detected by the Java Runtime Environment:
   #
   #  SIGSEGV (0xb) at pc=0x00007f6259fc465a, pid=16138, tid=140061131474688
   #
   # JRE version: OpenJDK Runtime Environment (7.0_101) (build 1.7.0_101-b00)
   # Java VM: OpenJDK 64-Bit Server VM (24.95-b01 mixed mode linux-amd64 compressed oops)
   # Derivative: IcedTea 2.6.6
   # Distribution: Debian GNU/Linux 8.2 (jessie), package 7u101-2.6.6-1~deb8u1
   # Problematic frame:
   # C  [libtesseract.so.3+0x24465a]  ERRCODE::error(char const*, TessErrorLogCode, char const*, ...) const+0x13a 
   #
   # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
   #
   # An error report file with more information is saved as:
   # /.../poc-tests/hs_err_pid16138.log
   #
   # If you would like to submit a bug report, please include
   # instructions on how to reproduce the bug and visit:
   #   http://icedtea.classpath.org/bugzilla
   # The crash happened outside the Java Virtual Machine in native code.
   # See problematic frame for where to report the bug.
   #
   [1]    16138 abort      sikuli exitcode.sikuli/
   sikuli exitcode.sikuli/  11,29s user 1,04s system 97% cpu 12,677 total
   

   So if the PNG file does not exist in the file system, it starts tesseract and somehow an assertion in C is thrown. What about the exit code? 134. So this test case is fine.

2. Now consider a case where the PNG file exists:

   click("exists_but_cannot_find.png")
   

   What about stdout and stderr?

   % sikuli exitcode.sikuli/
   [info] Sikuli vision engine loaded.
   [info] VDictProxy loaded.
   [error] Can't run this Sikuli script: exitcode.sikuli/
   Traceback (most recent call last):
    File "/.../poc-tests/exitcode.sikuli/exitcode.py", line 1, in <module>
      click("exists_but_cannot_find.png")
   Line 2, in file /.../poc-tests/exitcode.sikuli/exitcode.py
   
      at org.sikuli.script.Region.handleFindFailed(Region.java:421)
      at org.sikuli.script.Region.wait(Region.java:512)
      at org.python.proxies.sikuli.Region$Region$0.super__wait(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
   
   org.sikuli.script.FindFailed: FindFailed: can not find exists_but_cannot_find.png on the screen.
   Line 2, in file /.../poc-tests/exitcode.sikuli/exitcode.py
   
   sikuli exitcode.sikuli/  13,15s user 0,95s system 94% cpu 14,992 total
   

   What about the exit code? 0. Even though user's initial intent verify correctness of my application could not be met.

3. Now let a PNG file exist and make it findable on the GUI. Intentionally exit with exit code 3.

   click("exists_and_found.png")
   exit(3)
   

   And let's see:

   % sikuli exitcode.sikuli/
   [info] Sikuli vision engine loaded.
   [info] VDictProxy loaded.
   [log] CLICK on (135,64)
   [error] Can't run this Sikuli script: exitcode.sikuli/
   sikuli exitcode.sikuli/  11,18s user 0,70s system 93% cpu 12,743 total
   

   What about the exit code? 3. If you remove exit(3) from the code, it is zero and if you add exit(3), it is 3. Seems legit.

4. And finally consider something like a wrong CLI parameter:

   % sikuli --wrongparameter exitcode.sikuli/
   [error] Can't run this Sikuli script: --wrongparameter
   java.lang.StringIndexOutOfBoundsException: String index out of range: -1
      at java.lang.String.substring(String.java:1911)
      at org.sikuli.script.ScriptRunner.getPyFrom(ScriptRunner.java:109)
      at org.sikuli.script.ScriptRunner.runPython(ScriptRunner.java:103)
      at org.sikuli.script.SikuliScript.main(SikuliScript.java:111)
   [info] Sikuli vision engine loaded.
   ...
   
   sikuli --wrongparameter exitcode.sikuli/  13,09s user 1,26s system 95% cpu 15,095 total
   

   What about the exit code? It is 0.

My point of this issue: Please fix cases 2 and 4 to return non-zero exit codes in case of failure.

Design: Be aware that AFAICS exit codes of exit(...) are passed over to Java and System.Exit is called correspondingly. Hence exit codes might collide (i.e. 1 means test case failed as well as exit(1) was called). Nevertheless, this issue should be resolved by convention. For example all sikuli scripts shall not exit with exit code 255, because it is reserved for failing testcases. You might want to consider this backwards-incompatible, but sikuli goes against conventions established by any other established test framework.

Code: If think most of this fix needs to be done in SikuliScript. For cases 2 and 4 System.exit(1) suffices in case of an exception.

Btw, I think message Can't run this Sikuli script for a script exiting with exit(N) where N!=0 is very non-intuitive, because the script was actually run.

[mika]

ACK, what might be worth consideration is what e.g. autopkgtest does, quoting from its manpage:

EXIT STATUS
       0    all tests passed
       2    at least one test skipped
       4    at least one test failed
       6    at least one test failed and at least one test skipped
       8    no tests in this package
       12   erroneous package
       16   testbed failure
       20   other unexpected failures including bad usage

[RaiMan]

Thanks for the elaboration.

This repo is no longer alive.

The follow up now is at https://github.com/RaiMan/SikuliX-2014
homepage http://sikulix.com

If you ever have the chance to check the same with SikuliX version 1.1.1 and report back, I would highly appreciate that.

I will try to consolidate the exit behavior with SikuliX version 2.