RFE: Disabling the exit failure when tests failing

[ukulekek]

When I'm running the rpminspect in a pipeline, the pipeline can't distinguish between failing tests and the failing execution of rpminspect. In both cases, rpminspect returns exit code 1.
I propose to add an additional tag for the threshold option to disable the exit failure(e.g. -t NO) or change the exit code for failing tests.

[thrix]

@ukulekek can you post here an example pls? Feel free to share the NVR, should not be a problem ...

[ukulekek]

Example of valid execution of failing tests with "BAD" result:

$ rpminspect -o results symlinks-1.4-10.el7
$ echo $?
1

Example of invalid execution:

$ rpminspect not-existing-nvr.el7
*** Unable to find after build: none-existing-nvr.el7
*** Failed to gather specified builds.
$ echo $?
1

[dcantrell]

I think this is a reasonable idea. For the example given, running rpminspect in Fedora on symlinks-1.4-10.el7 will fail unless you are providing an rpminspect.conf file which directs rpminspect to the internal brew instance so it can look up that build (the '-o results' captures the message, but that's my guess as to what's happening).

Rather than extending the -t option, I think there should be 3 exit codes for rpminspect. Similar to a program like grep. In the case of rpminspect:

0=passes inspection
1=fails inspection
2=program error occurred

Thoughts?

[ukulekek]

Yes, I like it. Moreover, this is the best option for our infrastructure.

[dcantrell]

Just pushed a change for this. Exit 0 means inspections passed, exit 1 means inspections failed, and exit 2 means a program error occurred.