Make validate on RHEL7 content fails

[jan-cerny]

I tried to run make validate in RHEL/7 directory. The output is below.
I also found out that in SSG Makefile, make validate target is commented out for RHEL7, Openstack and RHEVM3 content. This must be fixed because the content should be validated.

[jcerny@t440s 7{master}]$ make validate
oscap xccdf validate-xml output/ssg-rhel7-xccdf.xml
oscap oval validate-xml --schematron output/ssg-rhel7-oval.xml
oscap cpe validate-xml output/ssg-rhel7-cpe-dictionary.xml
oscap oval validate-xml --schematron output/ssg-rhel7-cpe-oval.xml
oscap ds sds-validate output/ssg-rhel7-ds.xml
oscap xccdf validate-xml output/ssg-centos7-xccdf.xml
oscap ds sds-validate output/ssg-centos7-ds.xml
oscap xccdf validate-xml output/ssg-sl7-xccdf.xml
oscap ds sds-validate output/ssg-sl7-ds.xml
cd output; ../utils/verify-references.py --rules-with-invalid-checks --ovaldefs-unused ssg-rhel7-xccdf.xml
Invalid OVAL definition referenced by XCCDF Rule: rpm_verify_permissions
Invalid OVAL definition referenced by XCCDF Rule: rpm_verify_hashes
Invalid OVAL definition referenced by XCCDF Rule: mount_option_nodev_nonroot_local_partitions
Invalid OVAL definition referenced by XCCDF Rule: mount_option_nodev_removable_partitions
Invalid OVAL definition referenced by XCCDF Rule: mount_option_noexec_removable_partitions
Invalid OVAL definition referenced by XCCDF Rule: mount_option_nosuid_removable_partitions
Invalid OVAL definition referenced by XCCDF Rule: mount_option_tmp_nodev
Invalid OVAL definition referenced by XCCDF Rule: mount_option_tmp_noexec
Invalid OVAL definition referenced by XCCDF Rule: mount_option_tmp_nosuid
Invalid OVAL definition referenced by XCCDF Rule: mount_option_dev_shm_nodev
Invalid OVAL definition referenced by XCCDF Rule: mount_option_dev_shm_noexec
Invalid OVAL definition referenced by XCCDF Rule: mount_option_dev_shm_nosuid
Invalid OVAL definition referenced by XCCDF Rule: mount_option_var_tmp_bind
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_usb-storage_disabled
Invalid OVAL definition referenced by XCCDF Rule: bootloader_nousb_argument
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_cramfs_disabled
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_freevxfs_disabled
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_jffs2_disabled
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_hfs_disabled
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_hfsplus_disabled
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_squashfs_disabled
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_udf_disabled
Invalid OVAL definition referenced by XCCDF Rule: sticky_world_writable_dirs
Invalid OVAL definition referenced by XCCDF Rule: world_writeable_files
Invalid OVAL definition referenced by XCCDF Rule: no_files_unowned_by_user
Invalid OVAL definition referenced by XCCDF Rule: no_files_unowned_by_group
Invalid OVAL definition referenced by XCCDF Rule: world_writable_files_system_ownership
Invalid OVAL definition referenced by XCCDF Rule: disable_users_coredumps
Invalid OVAL definition referenced by XCCDF Rule: sysctl_fs_suid_dumpable
Invalid OVAL definition referenced by XCCDF Rule: enable_dmesg_restriction
Invalid OVAL definition referenced by XCCDF Rule: enable_selinux_bootloader
Invalid OVAL definition referenced by XCCDF Rule: selinux_all_devicefiles_labeled
Invalid OVAL definition referenced by XCCDF Rule: accounts_passwords_pam_faillock_interval
Invalid OVAL definition referenced by XCCDF Rule: root_path_no_dot
Invalid OVAL definition referenced by XCCDF Rule: disable_interactive_boot
Invalid OVAL definition referenced by XCCDF Rule: network_disable_zeroconf
Invalid OVAL definition referenced by XCCDF Rule: network_sniffer_disabled
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_default_send_redirects
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_all_send_redirects
Invalid OVAL definition referenced by XCCDF Rule: sysctl_ipv4_ip_forward
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_all_accept_source_route
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_all_accept_redirects
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_all_secure_redirects
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_all_log_martians
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_default_accept_source_route
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_default_accept_redirects
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_default_secure_redirects
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_icmp_echo_ignore_broadcasts
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_icmp_ignore_bogus_error_responses
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_tcp_syncookies
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_all_rp_filter
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv4_conf_default_rp_filter
Invalid OVAL definition referenced by XCCDF Rule: deactivate_wireless_interfaces
Invalid OVAL definition referenced by XCCDF Rule: service_bluetooth_disabled
Invalid OVAL definition referenced by XCCDF Rule: network_ipv6_disable_rpc
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv6_conf_default_accept_ra
Invalid OVAL definition referenced by XCCDF Rule: sysctl_net_ipv6_conf_default_accept_redirects
Invalid OVAL definition referenced by XCCDF Rule: network_ipv6_static_address
Invalid OVAL definition referenced by XCCDF Rule: network_ipv6_privacy_extensions
Invalid OVAL definition referenced by XCCDF Rule: network_ipv6_default_gateway
Invalid OVAL definition referenced by XCCDF Rule: service_ip6tables_enabled
Invalid OVAL definition referenced by XCCDF Rule: set_iptables_default_rule
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_rds_disabled
Invalid OVAL definition referenced by XCCDF Rule: kernel_module_tipc_disabled
Invalid OVAL definition referenced by XCCDF Rule: userowner_rsyslog_files
Invalid OVAL definition referenced by XCCDF Rule: groupowner_rsyslog_files
Invalid OVAL definition referenced by XCCDF Rule: rsyslog_file_permissions
Invalid OVAL definition referenced by XCCDF Rule: rsyslog_send_messages_to_logserver
Invalid OVAL definition referenced by XCCDF Rule: rsyslog_accept_remote_messages_none
Invalid OVAL definition referenced by XCCDF Rule: configure_logwatch_hostlimit
Invalid OVAL definition referenced by XCCDF Rule: configure_logwatch_splithosts
Invalid OVAL definition referenced by XCCDF Rule: bootloader_audit_argument
Invalid OVAL definition referenced by XCCDF Rule: configure_auditd_num_logs
Invalid OVAL definition referenced by XCCDF Rule: configure_auditd_max_log_file
Invalid OVAL definition referenced by XCCDF Rule: configure_auditd_max_log_file_action
Invalid OVAL definition referenced by XCCDF Rule: auditd_data_retention_space_left_action
Invalid OVAL definition referenced by XCCDF Rule: auditd_data_retention_admin_space_left_action
Invalid OVAL definition referenced by XCCDF Rule: auditd_data_retention_action_mail_acct
Invalid OVAL definition referenced by XCCDF Rule: audit_logs_rootowner
Invalid OVAL definition referenced by XCCDF Rule: disable_xinetd
Invalid OVAL definition referenced by XCCDF Rule: service_rexec_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_rsh_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_rlogin_disabled
Invalid OVAL definition referenced by XCCDF Rule: no_rsh_trust_files
Invalid OVAL definition referenced by XCCDF Rule: disable_ypbind
Invalid OVAL definition referenced by XCCDF Rule: service_tftp_disabled
Invalid OVAL definition referenced by XCCDF Rule: tftpd_uses_secure_mode
Invalid OVAL definition referenced by XCCDF Rule: service_acpid_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_certmonger_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_cgconfig_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_cgred_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_cpuspeed_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_irqbalance_enabled
Invalid OVAL definition referenced by XCCDF Rule: service_kdump_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_mdmonitor_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_messagebus_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_netconsole_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_portreserve_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_psacct_enabled
Invalid OVAL definition referenced by XCCDF Rule: service_quota_nld_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_rhnsd_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_rhsmcertd_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_saslauthd_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_smartd_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_sysstat_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_crond_enabled
Invalid OVAL definition referenced by XCCDF Rule: ssh_server_iptables_exception
Invalid OVAL definition referenced by XCCDF Rule: disable_xwindows_with_target
Invalid OVAL definition referenced by XCCDF Rule: packagegroup_xwindows_remove
Invalid OVAL definition referenced by XCCDF Rule: disable_avahi
Invalid OVAL definition referenced by XCCDF Rule: service_cups_disabled
Invalid OVAL definition referenced by XCCDF Rule: cups_disable_browsing
Invalid OVAL definition referenced by XCCDF Rule: cups_disable_printserver
Invalid OVAL definition referenced by XCCDF Rule: disable_dhcp_server
Invalid OVAL definition referenced by XCCDF Rule: service_postfix_enabled
Invalid OVAL definition referenced by XCCDF Rule: postfix_network_listening_disabled
Invalid OVAL definition referenced by XCCDF Rule: postfix_server_banner
Invalid OVAL definition referenced by XCCDF Rule: ldap_client_start_tls
Invalid OVAL definition referenced by XCCDF Rule: ldap_client_tls_cacertpath
Invalid OVAL definition referenced by XCCDF Rule: service_nfslock_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_rpcgssd_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_rpcidmapd_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_netfs_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_nfs_disabled
Invalid OVAL definition referenced by XCCDF Rule: service_rpcsvcgssd_disabled
Invalid OVAL definition referenced by XCCDF Rule: use_nodev_option_on_nfs_mounts
Invalid OVAL definition referenced by XCCDF Rule: use_nosuid_option_on_nfs_mounts
Invalid OVAL definition referenced by XCCDF Rule: disable_dns_server
Invalid OVAL definition referenced by XCCDF Rule: disable_vsftpd
Invalid OVAL definition referenced by XCCDF Rule: disable_httpd
Invalid OVAL definition referenced by XCCDF Rule: httpd_logs_permissions
Invalid OVAL definition referenced by XCCDF Rule: httpd_conf_dir_permissions
Invalid OVAL definition referenced by XCCDF Rule: httpd_conf_files_permissions
Invalid OVAL definition referenced by XCCDF Rule: disable_dovecot
Invalid OVAL definition referenced by XCCDF Rule: dovecot_enable_ssl
Invalid OVAL definition referenced by XCCDF Rule: dovecot_disable_plaintext_auth
Invalid OVAL definition referenced by XCCDF Rule: disable_smb_server
Invalid OVAL definition referenced by XCCDF Rule: require_smb_client_signing
Invalid OVAL definition referenced by XCCDF Rule: require_smb_client_signing_mount.cifs
Invalid OVAL definition referenced by XCCDF Rule: disable_squid
Invalid OVAL definition referenced by XCCDF Rule: disable_snmpd
OVAL Check is not referenced by XCCDF: oval:ssg:def:188
Makefile:170: návod pro cíl „validate“ selhal
make: *** [validate] Chyba 1

[iankko]

Hello Jan,

thank you for your report. This is a known issue - the RHEL/7 SCAP content was never ever been in the state, the make validate target could be switched on for it. As you correctly pointed out in:
https://lists.fedorahosted.org/pipermail/scap-security-guide/2015-August/006546.html

it has been commented out in August 2014. If you would checkout the RHEL/7 git repository content from August 2014, you would find out the situation now is better than it was before (IOW make validate now shows less rules failing that it has in August 2014). So it's not the case the make validate would be failing just for couple of rules, and instead of fixing it, we would comment it out. The situation is different - the RHEL/7 content was never in shape, the make validate target could be switched on. We are progressing (meaning during the time less and less rules are reported to be invalid), but still didn't reach the state RHEL/7 content could have the make validate target switched on.

For RHEVM3 and Openstack cases you refer above the situation is even worse -- if you would have a look at the actual RHEVM3 and Openstack content, you would notice there are no OVAL rules / remediations defined at all for these products (what is in the repository now for both products being just "template" scripts for both products, but no content at all). Therefore it does not make sense to run make validate target for these two products. And that's also the reason, why we do not ship these benchmarks in downstreams RPM packages.

Jan.

[mpreisler]

@iankko, can we comment the references in the XCCDF rules to fix make validate? Including RHEL7 content in make validate gives us a lot more sanity checks when reviewing pull requests and doing continuous integration.

I don't see why we should do this slowly and keep this commented out. In my eyes this should be done ASAP so that we stop regressing and have some assurance that quality of RHEL7 content is going up with every commit.

[mpreisler]

make validate also doesn't run RHEL/5 validation.