Exit status is 0 when scanner fails to find selector ID in Benchmark

[xprazak2]

When running a scan against profile from a tailoring file and the tailoring file references a benchmark that is not in the XCCDF file, the scanner outputs the rules that could not be found:

OpenSCAP Error: Selector ID(xccdf_org.ssgproject.content_rule_firefox_preferences-on-secure_page_warning) does not exist in Benchmark.

The resulting report has all rules from XCCDF file marked as 'notselected':
arf_report.xml.gz, which makes sense, but the exit status is 0.

Steps to reproduce:

1. Use SCAP Workbench to create a tailoring file (based on Upstream Firefog STIG), unselect at least one rule.
2. run a scan with tailoring file created in the previous step. Supply a XCCDF file different from what was used to create tailoring file (ssg-jave-ds.xml):

oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_stig-firefox-upstream_customized --tailoring-file ssg-firefox-ds-tailored.xml --results-arf arf.xml ssg-java-ds.xml

3. Notice there will be errors (for each rule that was deselected in tailoring).
4. run echo $?, it will be 0

Is this intended bahavior? Why would exit status be 0, if there were clearly errors?

[jan-cerny]

I can reproduce this with the latest upstream OpenSCAP. I think this behaviour is wrong.

I thought OpenSCAP should check if tailoring file references the same benchmark as was provided. It seems to be easily achived by checking xccdf:Tailoring/xccdf:benchmark@href attribute.
However, the XCCDF language specification says on page 46 following:

Note, however, that this is a purely informative field. Benchmark consumers are
not required to perform any lookup or even verify that a tailoring document is
being applied to the correct Benchmark.

So this does not seem to be a way.

But the actual error reported by OpenSCAP is

OpenSCAP Error: Selector ID(xccdf_org.ssgproject.content_rule_firefox_preferences-on-secure_page_warning) does not exist in Benchmark

The ID menitioned here is the ID in xccdf:select@idref attribute in the tailoring document.

And here, the specification says:

The xccdf:select element’s @idref attribute (identifier) SHALL specify the designated Group, Rule, or cluster of Rules and Groups; it MUST match the @id attribute of a Group or Rule in the
Benchmark, or the cluster id assigned to one or more Rules or Groups.

Based on that, I would vote for terminating and returning 1 in this case.

But what if an user creates tailoring for a profile, the tailored content updates and some rule that he tailored is renamed or removed in updated version? Then he will have to create a new tailoring file, which is obviously wrong. That brings me back to my first solution, but there, the specification was against me 😬

@mpreisler Do you have any idea on this?

[mpreisler]

I agree but keep in mind that these issues are common and several releases of SSG are affected. If we change the exit code we may be breaking existing use-cases.

[mpreisler]

Discussed at today's meeting. A good compromise would be to change the wording to "Warning: Selector ID was not found in benchmark. Ignoring...". We are scared of exiting with 1 because of user tailoring files and changing IDs of Rules / Groups in SSG releases.