Separate exit code for test parsing errors?

[Hoodoo]

Thanks for all the hard work on this utility, I've found it really easy to use.

I use a simple bash script to run tests depending on different facter facts on hosts. However, I'd like to be able to tell failing tests from synctactically incorrect tests, e.g.:

#============ The test which should fail:
lan1:~ # cat goss_incorrect.yaml
port:
  tcp:22:
listening: false

lan1:~ # goss --gossfile goss_incorrect.yaml validate -f tap
Error: Could not parse resource Port:tcp:22

lan1:~ # echo $?
1

#============ The incorrect test:
lan1:~ # cat goss_failing.yaml 
port:
  tcp:22:
    listening: false

lan1:~ # goss --gossfile goss_failing.yaml validate -f tap
1..1
not ok 1 - Port: tcp:22: listening: doesn't match, expect: [false] found: [true]

lan1:~ # echo $?
1

I believe having a separate exit code for YAML parsing/template rendering would make scripting easier.

[kwaio]

I was looking for documentation/code that would explain the different possible error values... and this proves that's no specific policy.
I would love to have different error codes to process whether there was gossfile errors or validation errors. (wish I had any Go skill to suggest a PR)

[aelsabbahy]

Agree, this would be useful for scripting.

[kevinreddot]

I was looking for documentation/code that would explain the different possible error values... and this proves that's no specific policy.

It is. E.g., FreeBSD documents it in sysexits man page, Linux has /usr/include/sysexits.h too. In Go there is https://godoc.org/sny.no/os/sysexit.

So, for incorrect yaml the proper exit code would probably be EX_CONFIG while failed test could still be 1.

[naftulikay]

I'm also affected by this. I maintain the degoss Ansible role (which installs, runs, and removes Goss and test cases on a machine), and I have to write some very interesting logic to catch these kinds of errors. Goss returns 0 as a return code when it fails to load or parse the Goss file.

My workaround is to set --format to json and if I'm unable to deserialize JSON, I assume that there's an error, regardless of return code.

[marcinpraczko]

I would say that this is also importnat for check over HTTP?
As I can see goss returns proper result (HTTP BODY) with errors - but then returns 503 as HTTP return code.
So is really hard to know if connection with goss failed or whether checks failed:

http http://goss.zone/healthz
HTTP/1.1 503 Service Unavailable   # !!!! <- 503 - which is not correct - service IS working, checks fails.
Content-Length: 1289
Content-Type: text/plain; charset=utf-8
Date: Thu, 08 Aug 2019 10:31:29 GMT

GOSS CRITICAL - Count: 60, Failed: 9, Skipped: 0, Duration: 0.219s