extend misleading Automatus error message

[vojtapolasek]

Description:

• The error message is extended and the oscap output is appended

Rationale:

• The condition checks that the output does not contain the rule id. This can mean that the rule has not been selected. But it can mean also other errors, e.g. there has been problem starting evaluation.

I think this manifests also here: #10895

Review Hints:

I managed to reproduce it by editing /etc/nsswitch.conf, removing all lines starting with "passwd" and replacing them with:

passwd nis

This is invalid on RHEL 9 and mostly on RHEL 8. It seems this prevents oscap-ssh from logging in and the error message is encountered.

[vojtapolasek]

Please @matusmarhefka or @mildas have look at this, maybe you will have a better solution.

[github-actions]

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

[github-actions]

🤖 A k8s content image for this PR is available at:
ghcr.io/complianceascode/k8scontent:11658
This image was built from commit: eacecbf

Click here to see how to deploy it

If you alread have Compliance Operator deployed:
utils/build_ds_container.py -i ghcr.io/complianceascode/k8scontent:11658

Otherwise deploy the content and operator together by checking out ComplianceAsCode/compliance-operator and:
CONTENT_IMAGE=ghcr.io/complianceascode/k8scontent:11658 make deploy-local

[codeclimate]

Code Climate has analyzed commit eacecbf and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 59.8% (0.0% change).

View more on Code Climate.

[jan-cerny]

I have reproduced the problem by breaking the tested data stream by changing the value of @test_ref in the OVAL definition in my favorite rule. This triggered your improved message. It no longer points people to a wrong direction concerning the profile which is in practice unlikely to happen. Also, I really love that it points me to the right log.

jcerny@fedora:~/work/git/scap-security-guide (pr/11658)$ python3 tests/automatus.py rule --libvirt qemu:///system ssgts_rhel9 --scenario selinux_enforcing.pass.sh selinux_state
Setting console output to log level INFO
INFO - The base image option has not been specified, choosing libvirt-based test environment.
INFO - Logging into /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2024-03-12-1145/test_suite.log
INFO - xccdf_org.ssgproject.content_rule_selinux_state
ERROR - Script selinux_enforcing.pass.sh using profile xccdf_org.ssgproject.content_profile_ospp found issue:
ERROR - Rule xccdf_org.ssgproject.content_rule_selinux_state has not been evaluated! Wrong profile selected in test scenario or there has been problem starting the evaluation. Please inspect the log file /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2024-03-12-1145/selinux_state-selinux_enforcing.pass.sh-initial.verbose.log for details.
ERROR - The initial scan failed for rule 'xccdf_org.ssgproject.content_rule_selinux_state'.
jcerny@fedora:~/work/git/scap-security-guide (pr/11658)$ cat /home/jcerny/work/git/scap-security-guide/logs/rule-custom-2024-03-12-1145/selinux_state-selinux_enforcing.pass.sh-initial.verbose.log
Warning: Permanently added '192.168.124.235' (ED25519) to the list of known hosts.
I: oscap: Identified document type: data-stream-collection [oscap(2070):oscap(7f0358df7940):doc_type.c:96:oscap_determine_document_type_reader]
I: oscap: Created a new XCCDF session from a SCAP Source Datastream '/tmp/tmp.LXPFPqJLYF/input.xml'. [oscap(2070):oscap(7f0358df7940):xccdf_session.c:179:xccdf_session_new_from_source]
D: oscap: Validating SCAP Source Datastream (1.3) document from /tmp/tmp.LXPFPqJLYF/input.xml. [oscap(2070):oscap(7f0358df7940):oscap_source.c:360:oscap_source_validate]
OpenSCAP Error: File '/tmp/tmp.LXPFPqJLYF/input.xml' line 65535: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}criterion': No match found for key-sequence ['oval:ssg-test_etc_selinux_config:tst:2'] of keyref '{http://oval.mitre.org/XMLSchema/oval-definitions-5}testKeyRef'.
 [/builddir/build/BUILD/openscap-1.3.8/src/XCCDF/xccdf_session.c:726]
Invalid SCAP Source Datastream (1.3) content in /tmp/tmp.LXPFPqJLYF/input.xml. [/builddir/build/BUILD/openscap-1.3.8/src/source/oscap_source.c:363]
Invalid SCAP Source Datastream (1.3) content in /tmp/tmp.LXPFPqJLYF/input.xml [/builddir/build/BUILD/openscap-1.3.8/src/XCCDF/xccdf_session.c:839]
scp: /tmp/tmp.LXPFPqJLYF/results-arf.xml: No such file or directory
Failed to copy the ARF file back to local machine!
jcerny@fedora:~/work/git/scap-security-guide (pr/11658)$