Build HTML guide for every profile in input XCCDF or datastream #578

Merged
merged 26 commits into from Jul 3, 2015

Conversation

Projects
None yet
5 participants
@mpreisler
Member

mpreisler commented Jun 11, 2015

Implemented a script that takes an XCCDF 1.1, 1.2 or source datastream, scrapes all profiles and builds a guide for every profile not in the blacklist. All the guides are placed next to the input file. This solves the issues with allrules which is no longer an option with new OpenSCAP versions. It also is the right thing IMO because every profile we ship is probably important enough to have a HTML guide.

I am proposing a pull request at this stage to discuss how I should continue. I would like to make this part of the build process and install all the HTML guide when installing the scap-security-guide package. The script also generates an index HTML file, right now it just links to all the profile guides but in the future it may have description of the benchmark and allow switching the profiles. This may be a good thing to publish on websites.

Usage example:

cd RHEL/6/output
../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

Generated 'ssg-rhel6-guide-CSCF-RHEL6-MLS.html' for profile ID 'xccdf_org.ssgproject.content_profile_CSCF-RHEL6-MLS'.
Generated 'ssg-rhel6-guide-usgcb-rhel6-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_usgcb-rhel6-server'.
Generated 'ssg-rhel6-guide-common.html' for profile ID 'xccdf_org.ssgproject.content_profile_common'.
Generated 'ssg-rhel6-guide-pci-dss.html' for profile ID 'xccdf_org.ssgproject.content_profile_pci-dss'.
Generated 'ssg-rhel6-guide-CS2.html' for profile ID 'xccdf_org.ssgproject.content_profile_CS2'.
Generated 'ssg-rhel6-guide-C2S.html' for profile ID 'xccdf_org.ssgproject.content_profile_C2S'.
Generated 'ssg-rhel6-guide-stig-rhel6-server-upstream.html' for profile ID 'xccdf_org.ssgproject.content_profile_stig-rhel6-server-upstream'.
Generated 'ssg-rhel6-guide-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_server'.
Generated 'ssg-rhel6-guide-rht-ccp.html' for profile ID 'xccdf_org.ssgproject.content_profile_rht-ccp'.

I should note at this point that building all the guides takes some time. The SSG build system right now is very stupid and builds everything all the time, regardless of whether the input files changed or not. Generating guides every time will slow the build process.

time ../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

...

real    0m8.802s
user    0m8.619s
sys     0m0.189s

I thought about being smart and generating several guides in parallel in the script but in the end I think it's a job of the build system to be smart, not the build scripts. But I will say here that it's possible to bring the times close to 3 seconds on quad core processors with very little work.

@mpreisler mpreisler changed the title from Build guides for every profile in input XCCDF or datastream to Build HTML guide for every profile in input XCCDF or datastream Jun 11, 2015

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jun 11, 2015

Contributor

Note: I didn't test your proposed change yet (need to reinstall RHEL-6 guest first to deal with some parallel issue). So expect limited feedback just on text (more detailed feedback based on testing later).

DO NOT MERGE YET

Implemented a script that takes an XCCDF 1.1, 1.2 or source datastream, scrapes all profiles and builds a guide for every profile not in the blacklist. All the guides are placed next to the input file. This solves the issues with allrules which is no longer an option with new OpenSCAP versions. It also is the right thing IMO because every profile we ship is probably important enough to have a HTML guide.

While I agree that each profile we ship is important enough to have it's own guide generated, assuming the guide being put next to the input file is just temporary build dist solution, right?
When shipping those profiles would it be possible to place them under e.g. /usr/share/scap-security-guide/guides/product location?

Also (since not have had tested yet) asking, would this generate custom profiles just for RHEL/6, RHEL/7, and Fedora products? Or is the code universal enough to be applied against any benchmark, e.g. Firefox or Java product content for example?

I am proposing a pull request at this stage to discuss how I should continue. I would like to make this part of the build process and install all the HTML guide when installing the scap-security-guide package.

Agree with this. As said I would suggest these HTML files to be placed under /usr/share/scap-security-guide/guides/product location (since AFAIK the scap-security-guide.spec %doc section can't contain whole directory as %doc specification, just file). So the alternative / proposal how to workaround this seems to be to place all HTML guides for that product under:
/usr/share/scap-security-guide/guides/product location, and have one e.g.
/usr/share/doc/scap-security-guide-0.1.22/guides-index.html location, that would serve as access points to these particular / concrete guides.

The script also generates an index HTML file, right now it just links to all the profile guides but in the future it may have description of the benchmark and allow switching the profiles. This may be a good thing to publish on websites.

IMHO it would be good to include it into the scap-security-guide RPM package too under the /usr/share/doc/scap-security-guide-NVR/guides-index.html location to serve like entry point to access concrete guides.

Taking this even further, we could have a scap-security-guide RPM %post-installation script, which would after scap-security-guide RPM installation pass that guide-index.html file to user's browser, and the browser would open that index for further investigation (yet before first actual system scan is run).

Usage example:

cd RHEL/6/output
../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

Generated 'ssg-rhel6-guide-CSCF-RHEL6-MLS.html' for profile ID 'xccdf_org.ssgproject.content_profile_CSCF-RHEL6-MLS'.
Generated 'ssg-rhel6-guide-usgcb-rhel6-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_usgcb-rhel6-server'.
Generated 'ssg-rhel6-guide-common.html' for profile ID 'xccdf_org.ssgproject.content_profile_common'.
Generated 'ssg-rhel6-guide-pci-dss.html' for profile ID 'xccdf_org.ssgproject.content_profile_pci-dss'.
Generated 'ssg-rhel6-guide-CS2.html' for profile ID 'xccdf_org.ssgproject.content_profile_CS2'.
Generated 'ssg-rhel6-guide-C2S.html' for profile ID 'xccdf_org.ssgproject.content_profile_C2S'.
Generated 'ssg-rhel6-guide-stig-rhel6-server-upstream.html' for profile ID 'xccdf_org.ssgproject.content_profile_stig-rhel6-server-upstream'.
Generated 'ssg-rhel6-guide-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_server'.
Generated 'ssg-rhel6-guide-rht-ccp.html' for profile ID 'xccdf_org.ssgproject.content_profile_rht-ccp'.

Thank you, I will check that.

I should note at this point that building all the guides takes some time. The SSG build system right now is very stupid and builds everything all the time, regardless of whether the input files changed or not. Generating guides every time will slow the build process.

time ../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

...

real    0m8.802s
user    0m8.619s
sys     0m0.189s

I thought about being smart and generating several guides in parallel in the script but in the end I think it's a job of the build system to be smart, not the build scripts. But I will say here that it's possible to bring the times close to 3 seconds on quad core processors with very little work.

Sure, feel free to optimize the code if you are of the opinion it would be improvement. There are lot more issues with SSG build process than just this one (IIRC correctly there's a ticket that for Fedora some files are included twice when building datastream etc.). I am aware of these deficiencies, just don't have time right now to invest into fixing them. So the more of the issues you notice && fix, the better for SSG as a whole.

Thanks, Jan.

Contributor

iankko commented Jun 11, 2015

Note: I didn't test your proposed change yet (need to reinstall RHEL-6 guest first to deal with some parallel issue). So expect limited feedback just on text (more detailed feedback based on testing later).

DO NOT MERGE YET

Implemented a script that takes an XCCDF 1.1, 1.2 or source datastream, scrapes all profiles and builds a guide for every profile not in the blacklist. All the guides are placed next to the input file. This solves the issues with allrules which is no longer an option with new OpenSCAP versions. It also is the right thing IMO because every profile we ship is probably important enough to have a HTML guide.

While I agree that each profile we ship is important enough to have it's own guide generated, assuming the guide being put next to the input file is just temporary build dist solution, right?
When shipping those profiles would it be possible to place them under e.g. /usr/share/scap-security-guide/guides/product location?

Also (since not have had tested yet) asking, would this generate custom profiles just for RHEL/6, RHEL/7, and Fedora products? Or is the code universal enough to be applied against any benchmark, e.g. Firefox or Java product content for example?

I am proposing a pull request at this stage to discuss how I should continue. I would like to make this part of the build process and install all the HTML guide when installing the scap-security-guide package.

Agree with this. As said I would suggest these HTML files to be placed under /usr/share/scap-security-guide/guides/product location (since AFAIK the scap-security-guide.spec %doc section can't contain whole directory as %doc specification, just file). So the alternative / proposal how to workaround this seems to be to place all HTML guides for that product under:
/usr/share/scap-security-guide/guides/product location, and have one e.g.
/usr/share/doc/scap-security-guide-0.1.22/guides-index.html location, that would serve as access points to these particular / concrete guides.

The script also generates an index HTML file, right now it just links to all the profile guides but in the future it may have description of the benchmark and allow switching the profiles. This may be a good thing to publish on websites.

IMHO it would be good to include it into the scap-security-guide RPM package too under the /usr/share/doc/scap-security-guide-NVR/guides-index.html location to serve like entry point to access concrete guides.

Taking this even further, we could have a scap-security-guide RPM %post-installation script, which would after scap-security-guide RPM installation pass that guide-index.html file to user's browser, and the browser would open that index for further investigation (yet before first actual system scan is run).

Usage example:

cd RHEL/6/output
../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

Generated 'ssg-rhel6-guide-CSCF-RHEL6-MLS.html' for profile ID 'xccdf_org.ssgproject.content_profile_CSCF-RHEL6-MLS'.
Generated 'ssg-rhel6-guide-usgcb-rhel6-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_usgcb-rhel6-server'.
Generated 'ssg-rhel6-guide-common.html' for profile ID 'xccdf_org.ssgproject.content_profile_common'.
Generated 'ssg-rhel6-guide-pci-dss.html' for profile ID 'xccdf_org.ssgproject.content_profile_pci-dss'.
Generated 'ssg-rhel6-guide-CS2.html' for profile ID 'xccdf_org.ssgproject.content_profile_CS2'.
Generated 'ssg-rhel6-guide-C2S.html' for profile ID 'xccdf_org.ssgproject.content_profile_C2S'.
Generated 'ssg-rhel6-guide-stig-rhel6-server-upstream.html' for profile ID 'xccdf_org.ssgproject.content_profile_stig-rhel6-server-upstream'.
Generated 'ssg-rhel6-guide-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_server'.
Generated 'ssg-rhel6-guide-rht-ccp.html' for profile ID 'xccdf_org.ssgproject.content_profile_rht-ccp'.

Thank you, I will check that.

I should note at this point that building all the guides takes some time. The SSG build system right now is very stupid and builds everything all the time, regardless of whether the input files changed or not. Generating guides every time will slow the build process.

time ../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

...

real    0m8.802s
user    0m8.619s
sys     0m0.189s

I thought about being smart and generating several guides in parallel in the script but in the end I think it's a job of the build system to be smart, not the build scripts. But I will say here that it's possible to bring the times close to 3 seconds on quad core processors with very little work.

Sure, feel free to optimize the code if you are of the opinion it would be improvement. There are lot more issues with SSG build process than just this one (IIRC correctly there's a ticket that for Fedora some files are included twice when building datastream etc.). I am aware of these deficiencies, just don't have time right now to invest into fixing them. So the more of the issues you notice && fix, the better for SSG as a whole.

Thanks, Jan.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 11, 2015

Code Health
Repository health decreased by 0.21% when pulling da5e8f0 on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

Code Health
Repository health decreased by 0.21% when pulling da5e8f0 on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 11, 2015

Member

The script is generic, can be used for every XCCDF or datastream SSG ships. Even for content outside of SSG.

The location of the guides can be changed and I am prepared to change it, depending on the discussion here.

What about the "allrules" profile? I can change the script to generate this profile and generate a guide for it. Is there demand for guide with every rule in the benchmark?

Member

mpreisler commented Jun 11, 2015

The script is generic, can be used for every XCCDF or datastream SSG ships. Even for content outside of SSG.

The location of the guides can be changed and I am prepared to change it, depending on the discussion here.

What about the "allrules" profile? I can change the script to generate this profile and generate a guide for it. Is there demand for guide with every rule in the benchmark?

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jun 11, 2015

Contributor

The script is generic, can be used for every XCCDF or datastream SSG ships. Even for content outside of SSG.

Brilliant to hear. Just asked to be sure. Thanks.

The location of the guides can be changed and I am prepared to change it, depending on the discussion here.

Ok, great. So let's wait for the other opinions prior making this change final.
@isimluk , @shawndwells , @redhatrises Can you comment on -^?

What about the "allrules" profile? I can change the script to generate this profile and generate a guide for it. Is there demand for guide with every rule in the benchmark?

IMHO we should continue generating it (even when this would mean we would do this not by using oscap tool like till now, but via this script). The motivation for being it available is that users might be interested in having a chance to look at all the uniq rules that are available for concrete product without the need to get such a list by combining the different profiles for that product.

Contributor

iankko commented Jun 11, 2015

The script is generic, can be used for every XCCDF or datastream SSG ships. Even for content outside of SSG.

Brilliant to hear. Just asked to be sure. Thanks.

The location of the guides can be changed and I am prepared to change it, depending on the discussion here.

Ok, great. So let's wait for the other opinions prior making this change final.
@isimluk , @shawndwells , @redhatrises Can you comment on -^?

What about the "allrules" profile? I can change the script to generate this profile and generate a guide for it. Is there demand for guide with every rule in the benchmark?

IMHO we should continue generating it (even when this would mean we would do this not by using oscap tool like till now, but via this script). The motivation for being it available is that users might be interested in having a chance to look at all the uniq rules that are available for concrete product without the need to get such a list by combining the different profiles for that product.

@redhatrises

This comment has been minimized.

Show comment
Hide comment
@redhatrises

redhatrises Jun 12, 2015

Member

I should note at this point that building all the guides takes some time. The SSG build system right now is very stupid and builds everything all the time, regardless of whether the input files changed or not. Generating guides every time will slow the build process.

time ../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

...

real    0m8.802s
user    0m8.619s
sys     0m0.189s

I thought about being smart and generating several guides in parallel in the script but in the end I think it's a job of the build system to be smart, not the build scripts. But I will say here that it's possible to bring the times close to 3 seconds on quad core processors with very little work.

Sure, feel free to optimize the code if you are of the opinion it would be improvement. There are lot more issues with SSG build process than just this one (IIRC correctly there's a ticket that for Fedora some files are included twice when building datastream etc.). I am aware of these deficiencies, just don't have time right now to invest into fixing them. So the more of the issues you notice && fix, the better for SSG as a whole.

+1. This would be nice IMO.

IMHO we should continue generating it (even when this would mean we would do this not by using oscap tool like till now, but via this script). The motivation for being it available is that users might be interested in having a chance to look at all the uniq rules that are available for concrete product without the need to get such a list by combining the different profiles for that product.

+1. I can see using this for testing and other scenarios.

Member

redhatrises commented Jun 12, 2015

I should note at this point that building all the guides takes some time. The SSG build system right now is very stupid and builds everything all the time, regardless of whether the input files changed or not. Generating guides every time will slow the build process.

time ../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

...

real    0m8.802s
user    0m8.619s
sys     0m0.189s

I thought about being smart and generating several guides in parallel in the script but in the end I think it's a job of the build system to be smart, not the build scripts. But I will say here that it's possible to bring the times close to 3 seconds on quad core processors with very little work.

Sure, feel free to optimize the code if you are of the opinion it would be improvement. There are lot more issues with SSG build process than just this one (IIRC correctly there's a ticket that for Fedora some files are included twice when building datastream etc.). I am aware of these deficiencies, just don't have time right now to invest into fixing them. So the more of the issues you notice && fix, the better for SSG as a whole.

+1. This would be nice IMO.

IMHO we should continue generating it (even when this would mean we would do this not by using oscap tool like till now, but via this script). The motivation for being it available is that users might be interested in having a chance to look at all the uniq rules that are available for concrete product without the need to get such a list by combining the different profiles for that product.

+1. I can see using this for testing and other scenarios.

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jun 14, 2015

Contributor

@mpreisler
Having had chance to test this enhancement on RHEL-6, RHEL-7, and Fedora products now noticed the following two issues detailed below:

1), Guides generated with older RHEL-6's version of oscap doesn't seem to have full / valid content:

Usage example:

cd RHEL/6/output
../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

Generated 'ssg-rhel6-guide-CSCF-RHEL6-MLS.html' for profile ID 'xccdf_org.ssgproject.content_profile_CSCF-RHEL6-MLS'.
Generated 'ssg-rhel6-guide-usgcb-rhel6-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_usgcb-rhel6-server'.
Generated 'ssg-rhel6-guide-common.html' for profile ID 'xccdf_org.ssgproject.content_profile_common'.
Generated 'ssg-rhel6-guide-pci-dss.html' for profile ID 'xccdf_org.ssgproject.content_profile_pci-dss'.
Generated 'ssg-rhel6-guide-CS2.html' for profile ID 'xccdf_org.ssgproject.content_profile_CS2'.
Generated 'ssg-rhel6-guide-C2S.html' for profile ID 'xccdf_org.ssgproject.content_profile_C2S'.
Generated 'ssg-rhel6-guide-stig-rhel6-server-upstream.html' for profile ID 'xccdf_org.ssgproject.content_profile_stig-rhel6-server-upstream'.
Generated 'ssg-rhel6-guide-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_server'.
Generated 'ssg-rhel6-guide-rht-ccp.html' for profile ID 'xccdf_org.ssgproject.content_profile_rht-ccp'.

While the script is working also on Red Hat Enterprise Linux 6 (it produces the index file and HTML guides for particular profiles), the generated per-profile guides doesn't seem to have valid content. Have tried with both firefox and konqueror browsers and what's produced is blank HTML page with black background without mention of any rule:

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.7 Beta (Santiago)
# rpm -q openscap
openscap-1.0.10-3.el6.x86_64

The produced HTML document for CSCF-RHEL6-MLS and C2S are inlined below:

  • content for CSCF-RHEL6-MLS profile:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svg="http://www.w3.org/2000/svg">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="generator" content="" />
    <meta name="Content-Type" content="text/html;charset=utf-8" />
    <style type="text/css" media="all">
    html, body { background-color: black; font-family:sans-serif; margin:0; padding:0; }
    abbr { text-transform:none; border:none; font-variant:normal; }
    div.score-outer { height: .8em; width:100%; min-width:100px; background-color: red; }
    div.score-inner { height: 100%; background-color: green; }
    .score-max, .score-val, .score-percent { text-align:right; }
    .score-percent { font-weight: bold; }
    th, td { padding-left:.5em; padding-right:.5em; }
    .rule-selected, .result-pass strong, .result-fixed strong { color:green; }
    .rule-inactive, .unknown, .result-notselected strong, .result-notchecked strong, .result-notapplicable strong, .result-informational strong, .result-unknown strong { color:#555; }
    .rule-notselected, .result-error strong, .result-fail strong { color:red; }
    table { border-collapse: collapse; border: 1px black solid; width:100%; }
    table th, thead tr { background-color:black; color:white; }
    table td { border-right: 1px black solid; }
    table td.result, table td.link { text-align:center; }
    table td.num { text-align:right; }
    div#rule-results-summary { margin-bottom: 1em; }
    table tr.result-legend td { width: 10%; }
    div#content p { text-align:justify; }
    div.result-detail { border: 1px solid black; margin: 2em 0; padding: 0 1em; }
    div#content h2 { border-bottom:2px dashed; margin-top:1em; margin-bottom:0.5em; text-align:center; }
    div#content h2#summary { margin-top:0; }
    h1 { margin:1em 0; }
    div.raw table, div.raw table td { border:none; width:auto; padding:0; }
    div.raw table { margin-left: 2em; }
    div.raw table td { padding: .1em .7em; }
    table tr { border-bottom: 1px dotted #000; }
    dir.raw table tr { border-bottom: 0 !important; }
    pre.code { background: #ccc; padding:.2em; }
    ul.toc-struct li { list-style-type: none; }
    div.xccdf-rule { margin-left: 10%; }
    div#footer, p.remark, .link { font-size:.8em; }
    thead tr td { font-weight:bold; text-align:center; }
    .hidden { display:none; }
    td.score-bar { text-align:center; }
    td.score-bar span.media { width:100%; min-width:7em; height:.8em; display:block; margin:0; padding:0; }
    .oval-results { font-size:.8em; overflow:auto; }
    div#guide-top-table table { width: 100%; }
    td#common-info { min-width: 25.0em; border-right: 1px solid #000; }
    td#versions-revisions { width: 25.0em; }
  </style>
    <style type="text/css" media="screen">
    div#content, div#header, div#footer { margin-left:1em; margin-right:1em; }
    div#content { background-color: white; padding:2em; }
    div#footer, div#header { color:white; text-align:center; }
    a, a:visited { color:blue; text-decoration:underline; }
    div#content p.link { text-align:right; font-size:.8em; }
    div#footer a { color:white; }
    div.xccdf-group, div.xccdf-rule { border-left: 3px solid white; padding-left:.3em; }
    div.xccdf-group:target, div.xccdf-rule:target { border-left-color:#ccc; }
    .toc-struct li:target { background:#ddd; }
    abbr { border-bottom: 1px black dotted; }
    abbr.date { border-bottom:none; }
    pre.code { overflow:auto; }
    table tbody tr:hover { background: #ccc; }
    div.raw table tbody tr:hover { background: transparent !important; }
  </style>
    <style type="text/css" media="print">
    @page { margin:3cm; }
    html, body { background-color:white; font-family:serif; }
    .link { display:none; }
    a, a:visited { color:black; text-decoration:none; }
    div#header, div#footer { text-align:center; }
    div#header { padding-top:36%; }
    h1 { vertical-align:center; }
    h2 { page-break-before:always; }
    h3, h4, h5  { page-break-after:avoid; }
    pre.code { background: #ccc; }
    div#footer { margin-top:auto; }
    .toc-struct { page-break-after:always; }
  </style>
  </head>
  <body></body>
</html>
  • HTML code for C2S profile:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svg="http://www.w3.org/2000/svg">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="generator" content="" />
    <meta name="Content-Type" content="text/html;charset=utf-8" />
    <style type="text/css" media="all">
    html, body { background-color: black; font-family:sans-serif; margin:0; padding:0; }
    abbr { text-transform:none; border:none; font-variant:normal; }
    div.score-outer { height: .8em; width:100%; min-width:100px; background-color: red; }
    div.score-inner { height: 100%; background-color: green; }
    .score-max, .score-val, .score-percent { text-align:right; }
    .score-percent { font-weight: bold; }
    th, td { padding-left:.5em; padding-right:.5em; }
    .rule-selected, .result-pass strong, .result-fixed strong { color:green; }
    .rule-inactive, .unknown, .result-notselected strong, .result-notchecked strong, .result-notapplicable strong, .result-informational strong, .result-unknown strong { color:#555; }
    .rule-notselected, .result-error strong, .result-fail strong { color:red; }
    table { border-collapse: collapse; border: 1px black solid; width:100%; }
    table th, thead tr { background-color:black; color:white; }
    table td { border-right: 1px black solid; }
    table td.result, table td.link { text-align:center; }
    table td.num { text-align:right; }
    div#rule-results-summary { margin-bottom: 1em; }
    table tr.result-legend td { width: 10%; }
    div#content p { text-align:justify; }
    div.result-detail { border: 1px solid black; margin: 2em 0; padding: 0 1em; }
    div#content h2 { border-bottom:2px dashed; margin-top:1em; margin-bottom:0.5em; text-align:center; }
    div#content h2#summary { margin-top:0; }
    h1 { margin:1em 0; }
    div.raw table, div.raw table td { border:none; width:auto; padding:0; }
    div.raw table { margin-left: 2em; }
    div.raw table td { padding: .1em .7em; }
    table tr { border-bottom: 1px dotted #000; }
    dir.raw table tr { border-bottom: 0 !important; }
    pre.code { background: #ccc; padding:.2em; }
    ul.toc-struct li { list-style-type: none; }
    div.xccdf-rule { margin-left: 10%; }
    div#footer, p.remark, .link { font-size:.8em; }
    thead tr td { font-weight:bold; text-align:center; }
    .hidden { display:none; }
    td.score-bar { text-align:center; }
    td.score-bar span.media { width:100%; min-width:7em; height:.8em; display:block; margin:0; padding:0; }
    .oval-results { font-size:.8em; overflow:auto; }
    div#guide-top-table table { width: 100%; }
    td#common-info { min-width: 25.0em; border-right: 1px solid #000; }
    td#versions-revisions { width: 25.0em; }
  </style>
    <style type="text/css" media="screen">
    div#content, div#header, div#footer { margin-left:1em; margin-right:1em; }
    div#content { background-color: white; padding:2em; }
    div#footer, div#header { color:white; text-align:center; }
    a, a:visited { color:blue; text-decoration:underline; }
    div#content p.link { text-align:right; font-size:.8em; }
    div#footer a { color:white; }
    div.xccdf-group, div.xccdf-rule { border-left: 3px solid white; padding-left:.3em; }
    div.xccdf-group:target, div.xccdf-rule:target { border-left-color:#ccc; }
    .toc-struct li:target { background:#ddd; }
    abbr { border-bottom: 1px black dotted; }
    abbr.date { border-bottom:none; }
    pre.code { overflow:auto; }
    table tbody tr:hover { background: #ccc; }
    div.raw table tbody tr:hover { background: transparent !important; }
  </style>
    <style type="text/css" media="print">
    @page { margin:3cm; }
    html, body { background-color:white; font-family:serif; }
    .link { display:none; }
    a, a:visited { color:black; text-decoration:none; }
    div#header, div#footer { text-align:center; }
    div#header { padding-top:36%; }
    h1 { vertical-align:center; }
    h2 { page-break-before:always; }
    h3, h4, h5  { page-break-after:avoid; }
    pre.code { background: #ccc; }
    div#footer { margin-top:auto; }
    .toc-struct { page-break-after:always; }
  </style>
  </head>
  <body></body>
</html>

Opening these pages with browser displays blank HTML page with black background. I suspect this isn't problem of the build-all-guides script, but rather some bug in older openscap shipped in RHEL-6, but IMHO this should be workarounded somehow (till it's fixed).

On the other hand it's necessary to mention the script works fine with new openscap versions producing HTML report / guide the newer way (it works fine on RHEL-7 and Fedora systems). So this truly seems to be rather issue of underlying openscap version on RHEL-6 system, than issue of the script itself.

2), The second point (RFE) being a request if it would be possible to mention underlying profile name in the <title> of the produced guide. Right now it always generates guide having Guide for Secure Configuration of Red Hat Enterprise Linux 7 for the different profiles. From this title it might not be immediately obvious, the HTML documents differ (when looking just at the header of the guide). So maybe we could update the code to start generating something like e.g. RH-CCP Profile for Secure Configuration of Red Hat Enterprise Linux 7 in order to be immediately visible (when previewing the HTML document) the particular HTML guide files are different. I am not sure if this change should be performed in the openscap code, or rather in the scap-security-guide code itself. This description is mainly meant as a way how to start discussion about the need to change this behaviour (the underlying code change itself can be performed later where appropriate once agreed on).

Contributor

iankko commented Jun 14, 2015

@mpreisler
Having had chance to test this enhancement on RHEL-6, RHEL-7, and Fedora products now noticed the following two issues detailed below:

1), Guides generated with older RHEL-6's version of oscap doesn't seem to have full / valid content:

Usage example:

cd RHEL/6/output
../../../shared/utils/build-all-guides.py -i ssg-rhel6-ds.xml 

Generated 'ssg-rhel6-guide-CSCF-RHEL6-MLS.html' for profile ID 'xccdf_org.ssgproject.content_profile_CSCF-RHEL6-MLS'.
Generated 'ssg-rhel6-guide-usgcb-rhel6-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_usgcb-rhel6-server'.
Generated 'ssg-rhel6-guide-common.html' for profile ID 'xccdf_org.ssgproject.content_profile_common'.
Generated 'ssg-rhel6-guide-pci-dss.html' for profile ID 'xccdf_org.ssgproject.content_profile_pci-dss'.
Generated 'ssg-rhel6-guide-CS2.html' for profile ID 'xccdf_org.ssgproject.content_profile_CS2'.
Generated 'ssg-rhel6-guide-C2S.html' for profile ID 'xccdf_org.ssgproject.content_profile_C2S'.
Generated 'ssg-rhel6-guide-stig-rhel6-server-upstream.html' for profile ID 'xccdf_org.ssgproject.content_profile_stig-rhel6-server-upstream'.
Generated 'ssg-rhel6-guide-server.html' for profile ID 'xccdf_org.ssgproject.content_profile_server'.
Generated 'ssg-rhel6-guide-rht-ccp.html' for profile ID 'xccdf_org.ssgproject.content_profile_rht-ccp'.

While the script is working also on Red Hat Enterprise Linux 6 (it produces the index file and HTML guides for particular profiles), the generated per-profile guides doesn't seem to have valid content. Have tried with both firefox and konqueror browsers and what's produced is blank HTML page with black background without mention of any rule:

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.7 Beta (Santiago)
# rpm -q openscap
openscap-1.0.10-3.el6.x86_64

The produced HTML document for CSCF-RHEL6-MLS and C2S are inlined below:

  • content for CSCF-RHEL6-MLS profile:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svg="http://www.w3.org/2000/svg">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="generator" content="" />
    <meta name="Content-Type" content="text/html;charset=utf-8" />
    <style type="text/css" media="all">
    html, body { background-color: black; font-family:sans-serif; margin:0; padding:0; }
    abbr { text-transform:none; border:none; font-variant:normal; }
    div.score-outer { height: .8em; width:100%; min-width:100px; background-color: red; }
    div.score-inner { height: 100%; background-color: green; }
    .score-max, .score-val, .score-percent { text-align:right; }
    .score-percent { font-weight: bold; }
    th, td { padding-left:.5em; padding-right:.5em; }
    .rule-selected, .result-pass strong, .result-fixed strong { color:green; }
    .rule-inactive, .unknown, .result-notselected strong, .result-notchecked strong, .result-notapplicable strong, .result-informational strong, .result-unknown strong { color:#555; }
    .rule-notselected, .result-error strong, .result-fail strong { color:red; }
    table { border-collapse: collapse; border: 1px black solid; width:100%; }
    table th, thead tr { background-color:black; color:white; }
    table td { border-right: 1px black solid; }
    table td.result, table td.link { text-align:center; }
    table td.num { text-align:right; }
    div#rule-results-summary { margin-bottom: 1em; }
    table tr.result-legend td { width: 10%; }
    div#content p { text-align:justify; }
    div.result-detail { border: 1px solid black; margin: 2em 0; padding: 0 1em; }
    div#content h2 { border-bottom:2px dashed; margin-top:1em; margin-bottom:0.5em; text-align:center; }
    div#content h2#summary { margin-top:0; }
    h1 { margin:1em 0; }
    div.raw table, div.raw table td { border:none; width:auto; padding:0; }
    div.raw table { margin-left: 2em; }
    div.raw table td { padding: .1em .7em; }
    table tr { border-bottom: 1px dotted #000; }
    dir.raw table tr { border-bottom: 0 !important; }
    pre.code { background: #ccc; padding:.2em; }
    ul.toc-struct li { list-style-type: none; }
    div.xccdf-rule { margin-left: 10%; }
    div#footer, p.remark, .link { font-size:.8em; }
    thead tr td { font-weight:bold; text-align:center; }
    .hidden { display:none; }
    td.score-bar { text-align:center; }
    td.score-bar span.media { width:100%; min-width:7em; height:.8em; display:block; margin:0; padding:0; }
    .oval-results { font-size:.8em; overflow:auto; }
    div#guide-top-table table { width: 100%; }
    td#common-info { min-width: 25.0em; border-right: 1px solid #000; }
    td#versions-revisions { width: 25.0em; }
  </style>
    <style type="text/css" media="screen">
    div#content, div#header, div#footer { margin-left:1em; margin-right:1em; }
    div#content { background-color: white; padding:2em; }
    div#footer, div#header { color:white; text-align:center; }
    a, a:visited { color:blue; text-decoration:underline; }
    div#content p.link { text-align:right; font-size:.8em; }
    div#footer a { color:white; }
    div.xccdf-group, div.xccdf-rule { border-left: 3px solid white; padding-left:.3em; }
    div.xccdf-group:target, div.xccdf-rule:target { border-left-color:#ccc; }
    .toc-struct li:target { background:#ddd; }
    abbr { border-bottom: 1px black dotted; }
    abbr.date { border-bottom:none; }
    pre.code { overflow:auto; }
    table tbody tr:hover { background: #ccc; }
    div.raw table tbody tr:hover { background: transparent !important; }
  </style>
    <style type="text/css" media="print">
    @page { margin:3cm; }
    html, body { background-color:white; font-family:serif; }
    .link { display:none; }
    a, a:visited { color:black; text-decoration:none; }
    div#header, div#footer { text-align:center; }
    div#header { padding-top:36%; }
    h1 { vertical-align:center; }
    h2 { page-break-before:always; }
    h3, h4, h5  { page-break-after:avoid; }
    pre.code { background: #ccc; }
    div#footer { margin-top:auto; }
    .toc-struct { page-break-after:always; }
  </style>
  </head>
  <body></body>
</html>
  • HTML code for C2S profile:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svg="http://www.w3.org/2000/svg">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="generator" content="" />
    <meta name="Content-Type" content="text/html;charset=utf-8" />
    <style type="text/css" media="all">
    html, body { background-color: black; font-family:sans-serif; margin:0; padding:0; }
    abbr { text-transform:none; border:none; font-variant:normal; }
    div.score-outer { height: .8em; width:100%; min-width:100px; background-color: red; }
    div.score-inner { height: 100%; background-color: green; }
    .score-max, .score-val, .score-percent { text-align:right; }
    .score-percent { font-weight: bold; }
    th, td { padding-left:.5em; padding-right:.5em; }
    .rule-selected, .result-pass strong, .result-fixed strong { color:green; }
    .rule-inactive, .unknown, .result-notselected strong, .result-notchecked strong, .result-notapplicable strong, .result-informational strong, .result-unknown strong { color:#555; }
    .rule-notselected, .result-error strong, .result-fail strong { color:red; }
    table { border-collapse: collapse; border: 1px black solid; width:100%; }
    table th, thead tr { background-color:black; color:white; }
    table td { border-right: 1px black solid; }
    table td.result, table td.link { text-align:center; }
    table td.num { text-align:right; }
    div#rule-results-summary { margin-bottom: 1em; }
    table tr.result-legend td { width: 10%; }
    div#content p { text-align:justify; }
    div.result-detail { border: 1px solid black; margin: 2em 0; padding: 0 1em; }
    div#content h2 { border-bottom:2px dashed; margin-top:1em; margin-bottom:0.5em; text-align:center; }
    div#content h2#summary { margin-top:0; }
    h1 { margin:1em 0; }
    div.raw table, div.raw table td { border:none; width:auto; padding:0; }
    div.raw table { margin-left: 2em; }
    div.raw table td { padding: .1em .7em; }
    table tr { border-bottom: 1px dotted #000; }
    dir.raw table tr { border-bottom: 0 !important; }
    pre.code { background: #ccc; padding:.2em; }
    ul.toc-struct li { list-style-type: none; }
    div.xccdf-rule { margin-left: 10%; }
    div#footer, p.remark, .link { font-size:.8em; }
    thead tr td { font-weight:bold; text-align:center; }
    .hidden { display:none; }
    td.score-bar { text-align:center; }
    td.score-bar span.media { width:100%; min-width:7em; height:.8em; display:block; margin:0; padding:0; }
    .oval-results { font-size:.8em; overflow:auto; }
    div#guide-top-table table { width: 100%; }
    td#common-info { min-width: 25.0em; border-right: 1px solid #000; }
    td#versions-revisions { width: 25.0em; }
  </style>
    <style type="text/css" media="screen">
    div#content, div#header, div#footer { margin-left:1em; margin-right:1em; }
    div#content { background-color: white; padding:2em; }
    div#footer, div#header { color:white; text-align:center; }
    a, a:visited { color:blue; text-decoration:underline; }
    div#content p.link { text-align:right; font-size:.8em; }
    div#footer a { color:white; }
    div.xccdf-group, div.xccdf-rule { border-left: 3px solid white; padding-left:.3em; }
    div.xccdf-group:target, div.xccdf-rule:target { border-left-color:#ccc; }
    .toc-struct li:target { background:#ddd; }
    abbr { border-bottom: 1px black dotted; }
    abbr.date { border-bottom:none; }
    pre.code { overflow:auto; }
    table tbody tr:hover { background: #ccc; }
    div.raw table tbody tr:hover { background: transparent !important; }
  </style>
    <style type="text/css" media="print">
    @page { margin:3cm; }
    html, body { background-color:white; font-family:serif; }
    .link { display:none; }
    a, a:visited { color:black; text-decoration:none; }
    div#header, div#footer { text-align:center; }
    div#header { padding-top:36%; }
    h1 { vertical-align:center; }
    h2 { page-break-before:always; }
    h3, h4, h5  { page-break-after:avoid; }
    pre.code { background: #ccc; }
    div#footer { margin-top:auto; }
    .toc-struct { page-break-after:always; }
  </style>
  </head>
  <body></body>
</html>

Opening these pages with browser displays blank HTML page with black background. I suspect this isn't problem of the build-all-guides script, but rather some bug in older openscap shipped in RHEL-6, but IMHO this should be workarounded somehow (till it's fixed).

On the other hand it's necessary to mention the script works fine with new openscap versions producing HTML report / guide the newer way (it works fine on RHEL-7 and Fedora systems). So this truly seems to be rather issue of underlying openscap version on RHEL-6 system, than issue of the script itself.

2), The second point (RFE) being a request if it would be possible to mention underlying profile name in the <title> of the produced guide. Right now it always generates guide having Guide for Secure Configuration of Red Hat Enterprise Linux 7 for the different profiles. From this title it might not be immediately obvious, the HTML documents differ (when looking just at the header of the guide). So maybe we could update the code to start generating something like e.g. RH-CCP Profile for Secure Configuration of Red Hat Enterprise Linux 7 in order to be immediately visible (when previewing the HTML document) the particular HTML guide files are different. I am not sure if this change should be performed in the openscap code, or rather in the scap-security-guide code itself. This description is mainly meant as a way how to start discussion about the need to change this behaviour (the underlying code change itself can be performed later where appropriate once agreed on).

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 14, 2015

Member

@iankko good catch! Old openscap cannot build guides from datastreams so I think we should switch to generating guides from plain XCCDFs. That way it will work everywhere.

Member

mpreisler commented Jun 14, 2015

@iankko good catch! Old openscap cannot build guides from datastreams so I think we should switch to generating guides from plain XCCDFs. That way it will work everywhere.

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jun 15, 2015

Contributor

@mpreisler

@iankko good catch! Old openscap cannot build guides from datastreams so I think we should switch to generating guides from plain XCCDFs. That way it will work everywhere.

Thanks, Martin. Yes it would be better it to work (when built from) everywhere.

What about dynamically updating the guide title with profile name yet? (see the paragraph 2) of my previous comment) Would it be feasible somehow?

Contributor

iankko commented Jun 15, 2015

@mpreisler

@iankko good catch! Old openscap cannot build guides from datastreams so I think we should switch to generating guides from plain XCCDFs. That way it will work everywhere.

Thanks, Martin. Yes it would be better it to work (when built from) everywhere.

What about dynamically updating the guide title with profile name yet? (see the paragraph 2) of my previous comment) Would it be feasible somehow?

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 17, 2015

Member

What about dynamically updating the guide title with profile name yet? (see the paragraph 2) of my previous comment) Would it be feasible somehow?

I have fixed this upstream in OpenSCAP, we now display the profile name in bigger font right under the benchmark title in both report and guide. See OpenSCAP/openscap@6c11038

snapshot1

Member

mpreisler commented Jun 17, 2015

What about dynamically updating the guide title with profile name yet? (see the paragraph 2) of my previous comment) Would it be feasible somehow?

I have fixed this upstream in OpenSCAP, we now display the profile name in bigger font right under the benchmark title in both report and guide. See OpenSCAP/openscap@6c11038

snapshot1

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jun 18, 2015

Contributor

What about dynamically updating the guide title with profile name yet? (see the paragraph 2) of my previous comment) Would it be feasible somehow?

I have fixed this upstream in OpenSCAP, we now display the profile name in bigger font right under the benchmark title in both report and guide. See OpenSCAP/openscap@6c11038

Thank you for the prompt patch, Martin. Appreciated!

Contributor

iankko commented Jun 18, 2015

What about dynamically updating the guide title with profile name yet? (see the paragraph 2) of my previous comment) Would it be feasible somehow?

I have fixed this upstream in OpenSCAP, we now display the profile name in bigger font right under the benchmark title in both report and guide. See OpenSCAP/openscap@6c11038

Thank you for the prompt patch, Martin. Appreciated!

@iankko iankko added this to the 0.1.24 milestone Jun 22, 2015

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 24, 2015

Code Health
Repository health decreased by 0.22% when pulling 2142447 on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

Code Health
Repository health decreased by 0.22% when pulling 2142447 on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 24, 2015

Code Health
Repository health decreased by 0.22% when pulling f09c76c on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

Code Health
Repository health decreased by 0.22% when pulling f09c76c on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 24, 2015

Member

Tested b3701ab and it builds fine on RHEL6. Will now add support for the rest of products.

Member

mpreisler commented Jun 24, 2015

Tested b3701ab and it builds fine on RHEL6. Will now add support for the rest of products.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 24, 2015

Code Health
Repository health decreased by 0.22% when pulling b3701ab on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

Code Health
Repository health decreased by 0.22% when pulling b3701ab on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 24, 2015

Member

All the guides are now built with the same method. The build was tested on RHEL6 and Fedora 22. It is transparent to the OpenSCAP version, the guide looks different of course but the indexing and the code around it copes with the differences.

What remains before this is ready to merge:

  • build guides in parallel for speed
  • generate (default) profile
  • sort profiles by title, make (default) appear last
  • provide a bash snippet that can be copy pasted to scan with selected profile
  • installation paths as suggested by @iankko
  • build from datastreams if possible, use XCCDF 1.1 only if datastreams are not supported by used OpenSCAP
  • provide a list of links instead of a nice combobox for people who disable JavaScript
  • clean-up
Member

mpreisler commented Jun 24, 2015

All the guides are now built with the same method. The build was tested on RHEL6 and Fedora 22. It is transparent to the OpenSCAP version, the guide looks different of course but the indexing and the code around it copes with the differences.

What remains before this is ready to merge:

  • build guides in parallel for speed
  • generate (default) profile
  • sort profiles by title, make (default) appear last
  • provide a bash snippet that can be copy pasted to scan with selected profile
  • installation paths as suggested by @iankko
  • build from datastreams if possible, use XCCDF 1.1 only if datastreams are not supported by used OpenSCAP
  • provide a list of links instead of a nice combobox for people who disable JavaScript
  • clean-up
@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 24, 2015

Code Health
Repository health decreased by 0.22% when pulling ecbc56e on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

Code Health
Repository health decreased by 0.22% when pulling ecbc56e on mpreisler:build_all_guides into c74bfad on OpenSCAP:master.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 24, 2015

Code Health
Repository health decreased by 0.22% when pulling 3ed7bbf on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.22% when pulling 3ed7bbf on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jun 24, 2015

Contributor

Thanks for the changes, Martin. Has had a brief look. Running just scap-security-guide/shared/utils/build-all-guides.py (without any prior content preparation) returns the following traceback (both on RHEL6 and Fedora21 systems):

utils]# ./build-all-guides.py 
Traceback (most recent call last):
  File "./build-all-guides.py", line 250, in <module>
    main()
  File "./build-all-guides.py", line 169, in main
    parent_dir = os.path.dirname(os.path.abspath(options.input_content))
  File "/usr/lib64/python2.7/posixpath.py", line 367, in abspath
    if not isabs(path):
  File "/usr/lib64/python2.7/posixpath.py", line 61, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'
Contributor

iankko commented Jun 24, 2015

Thanks for the changes, Martin. Has had a brief look. Running just scap-security-guide/shared/utils/build-all-guides.py (without any prior content preparation) returns the following traceback (both on RHEL6 and Fedora21 systems):

utils]# ./build-all-guides.py 
Traceback (most recent call last):
  File "./build-all-guides.py", line 250, in <module>
    main()
  File "./build-all-guides.py", line 169, in main
    parent_dir = os.path.dirname(os.path.abspath(options.input_content))
  File "/usr/lib64/python2.7/posixpath.py", line 367, in abspath
    if not isabs(path):
  File "/usr/lib64/python2.7/posixpath.py", line 61, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'
@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 24, 2015

Member

Thanks for the changes, Martin. Has had a brief look. Running just scap-security-guide/shared/utils/build-all-guides.py (without any prior content preparation) returns the following traceback (both on RHEL6 and Fedora21 systems):

[snip]

That's sort of expected, the script needs to know which XCCDF or SDS it should operate on :-) I will fix the argument parser so that the switch is required. In the meantime make sure you include the --input argument.

Member

mpreisler commented Jun 24, 2015

Thanks for the changes, Martin. Has had a brief look. Running just scap-security-guide/shared/utils/build-all-guides.py (without any prior content preparation) returns the following traceback (both on RHEL6 and Fedora21 systems):

[snip]

That's sort of expected, the script needs to know which XCCDF or SDS it should operate on :-) I will fix the argument parser so that the switch is required. In the meantime make sure you include the --input argument.

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jun 24, 2015

Contributor

In the meantime make sure you include the --input argument.

Thanks, realized later.

Regardless of the issue my opinion is it looks very nice. Especially the guide profile selector is truly sophisticated. This will be real improvement wrt to current user experience. Thank you for your work on this!

Contributor

iankko commented Jun 24, 2015

In the meantime make sure you include the --input argument.

Thanks, realized later.

Regardless of the issue my opinion is it looks very nice. Especially the guide profile selector is truly sophisticated. This will be real improvement wrt to current user experience. Thank you for your work on this!

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 24, 2015

Code Health
Repository health decreased by 0.22% when pulling 09dc86d on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.22% when pulling 09dc86d on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 25, 2015

Code Health
Repository health decreased by 0.38% when pulling ed0c70a on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.38% when pulling ed0c70a on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 25, 2015

Code Health
Repository health decreased by 0.38% when pulling 842a7c0 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.38% when pulling 842a7c0 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 25, 2015

Code Health
Repository health decreased by 0.38% when pulling 2179f81 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.38% when pulling 2179f81 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 25, 2015

Code Health
Repository health decreased by 0.39% when pulling 7efbd1a on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.39% when pulling 7efbd1a on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 25, 2015

Member

Just discussed with @iankko that we will install all the guides to /usr/share/doc/scap-security-guide and put them in the doc subpackage. That way people who don't want the docs won't pay the storage price for all the guides.

Member

mpreisler commented Jun 25, 2015

Just discussed with @iankko that we will install all the guides to /usr/share/doc/scap-security-guide and put them in the doc subpackage. That way people who don't want the docs won't pay the storage price for all the guides.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jun 26, 2015

Code Health
Repository health decreased by 0.39% when pulling 9287526 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.39% when pulling 9287526 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jun 26, 2015

Member

Added a TODO item to always build from datastreams if possible, build from plain XCCDFs otherwise. Discussed with @iankko and @isimluk and we concluded that we really want people to move to datastreams. Using plain XCCDFs there would just reinforce their usage.

Member

mpreisler commented Jun 26, 2015

Added a TODO item to always build from datastreams if possible, build from plain XCCDFs otherwise. Discussed with @iankko and @isimluk and we concluded that we really want people to move to datastreams. Using plain XCCDFs there would just reinforce their usage.

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jul 2, 2015

Member

Everything in the TODO list is done now, this PR is ready for review.

Member

mpreisler commented Jul 2, 2015

Everything in the TODO list is done now, this PR is ready for review.

@iankko iankko self-assigned this Jul 2, 2015

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jul 2, 2015

Contributor

(Looks the "Do not merge yet' keyword has been removed from this already) => I will have a look at this.

Contributor

iankko commented Jul 2, 2015

(Looks the "Do not merge yet' keyword has been removed from this already) => I will have a look at this.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jul 3, 2015

Code Health
Repository health decreased by 0.39% when pulling 950f179 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

Code Health
Repository health decreased by 0.39% when pulling 950f179 on mpreisler:build_all_guides into ce7995c on OpenSCAP:master.

@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jul 3, 2015

Member

Me merging #592 a short while ago caused conflicts, therefore this can't be merged automatically.

I will resolve the conflicts shortly.

Member

mpreisler commented Jul 3, 2015

Me merging #592 a short while ago caused conflicts, therefore this can't be merged automatically.

I will resolve the conflicts shortly.

mpreisler added some commits Jun 11, 2015

mpreisler added some commits Jun 24, 2015

Changed Chromium build to use the new build-all-guides.py script
Also removed unnecessary file from Fedora build unlinked-notest.
Changed OpenStack build to use the new build-all-guides.py script
Please note that there were no profiles available, so right now the
build-all-guides.py code is commented out!
Fail explicitly when no INPUT is provided in build-all-guides.py
Also noted that OpenSCAP < 1.1 can't generate guides from source
datastreams.
Build 4 guides in parallel in build-all-guides.py
The number of jobs is configurable but defaults to 4. This greatly
speeds up the build, especially for RHEL6 where there are many profiles.

On my system with a dual-core CPU this commit achieves 100% speed-up
for the RHEL6 guides.
Build "(default)" profile as part of build-all-guides.py
And added default and index to profile-id blacklist.
Build HTML guides from source datastreams if possible
I know the detection is flawed and will break when we release OpenSCAP
2.0.0. The bet is that the build system of SCAP Security Guide will be
revamped before that happens :-)
@mpreisler

This comment has been minimized.

Show comment
Hide comment
@mpreisler

mpreisler Jul 3, 2015

Member

I have rebased this PR on top of f8e9a82

It can be merged without conflicts again.

Member

mpreisler commented Jul 3, 2015

I have rebased this PR on top of f8e9a82

It can be merged without conflicts again.

@landscape-bot

This comment has been minimized.

Show comment
Hide comment
@landscape-bot

landscape-bot Jul 3, 2015

Code Health
Repository health decreased by 0.38% when pulling 2f3b24c on mpreisler:build_all_guides into f8e9a82 on OpenSCAP:master.

Code Health
Repository health decreased by 0.38% when pulling 2f3b24c on mpreisler:build_all_guides into f8e9a82 on OpenSCAP:master.

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jul 3, 2015

Contributor

Couple of notes from testing below (has been tested on all three of RHEL-6.7 Beta system, RHEL-7.1 system, and Fedora 21 system):

  • First of all it needs to be said this is really impressive when compared to the current state. Thanks a lot for dedicating your time to work at this, Martin!

Couple of minor observations / suggestions below though:

  • We seem to tend to be building just RHEL-6 / CentOS6 / Scientific Linux 6 guides (even on RHEL-7 system). Unless I am missing something pretty obvious issuing make SSG_VERSION_IS_GIT_SNAPSHOT=no rpm command builds a final RPM containing just RHEL-6 / CentOS / Scientific Linux 6 index html files && corresponding HTML guides to them. This should be enhanced to include RHEL-7 profiles too,
  • Having had the scap-security-guide RPM built via the make SSG... target as above, and running cd /usr/share/doc/scap-security-guide && firefox ssg-rhel6-guide-index.html command the first HTML guide returns HTTP 404 / Not found for that file. Selecting another profile name subsequently shows that profile already. Returning to the 'C2S' profile back (C2S seem to be the first profile selected by default in the index) then displays the HTML guide again properly. The HTTP 404 / Not Found behaviour can be reproduced only for the first time the index file is launched in the browser (used firefox, therefore not sure this would be the same case for other browsers too),
  • For the command line snippet - IMHO it should be placed into separate / dedicated row under the profile name selection in the index file. The point is, sometimes the profile name is very long, which leads to the CLI snippet to be split into two rows. Therefore in order to improve the UX IMHO we should place it below the profile selection into it's dedicated row,
  • Running ./build-all-guides.py shows the -i option is required already => the former invalid syntax issue is fixed,
  • All the guides are placed into /usr/share/doc/scap-security-guide - by itself there isn't anything wrong with that. Just:
    • we should probably place them under product subfolder, e.g. guides for RHEL-6 to be placed under /usr/share/doc/scap-security-guide/guides/rhel6
    • alternative is to start shipping all these guides into separate scap-security-guide-doc subpackage.

All of those minor observations are NOT intended to be read as blocker for merging this PR (as already mentioned it's truly wonderful when compared when current state). They are rather intended just like suggestions for further enhancements.

Contributor

iankko commented Jul 3, 2015

Couple of notes from testing below (has been tested on all three of RHEL-6.7 Beta system, RHEL-7.1 system, and Fedora 21 system):

  • First of all it needs to be said this is really impressive when compared to the current state. Thanks a lot for dedicating your time to work at this, Martin!

Couple of minor observations / suggestions below though:

  • We seem to tend to be building just RHEL-6 / CentOS6 / Scientific Linux 6 guides (even on RHEL-7 system). Unless I am missing something pretty obvious issuing make SSG_VERSION_IS_GIT_SNAPSHOT=no rpm command builds a final RPM containing just RHEL-6 / CentOS / Scientific Linux 6 index html files && corresponding HTML guides to them. This should be enhanced to include RHEL-7 profiles too,
  • Having had the scap-security-guide RPM built via the make SSG... target as above, and running cd /usr/share/doc/scap-security-guide && firefox ssg-rhel6-guide-index.html command the first HTML guide returns HTTP 404 / Not found for that file. Selecting another profile name subsequently shows that profile already. Returning to the 'C2S' profile back (C2S seem to be the first profile selected by default in the index) then displays the HTML guide again properly. The HTTP 404 / Not Found behaviour can be reproduced only for the first time the index file is launched in the browser (used firefox, therefore not sure this would be the same case for other browsers too),
  • For the command line snippet - IMHO it should be placed into separate / dedicated row under the profile name selection in the index file. The point is, sometimes the profile name is very long, which leads to the CLI snippet to be split into two rows. Therefore in order to improve the UX IMHO we should place it below the profile selection into it's dedicated row,
  • Running ./build-all-guides.py shows the -i option is required already => the former invalid syntax issue is fixed,
  • All the guides are placed into /usr/share/doc/scap-security-guide - by itself there isn't anything wrong with that. Just:
    • we should probably place them under product subfolder, e.g. guides for RHEL-6 to be placed under /usr/share/doc/scap-security-guide/guides/rhel6
    • alternative is to start shipping all these guides into separate scap-security-guide-doc subpackage.

All of those minor observations are NOT intended to be read as blocker for merging this PR (as already mentioned it's truly wonderful when compared when current state). They are rather intended just like suggestions for further enhancements.

@iankko

This comment has been minimized.

Show comment
Hide comment
@iankko

iankko Jul 3, 2015

Contributor

Based on the above comment merging this. ACK.

Thanks a lot for your work, Martin!

Contributor

iankko commented Jul 3, 2015

Based on the above comment merging this. ACK.

Thanks a lot for your work, Martin!

iankko added a commit that referenced this pull request Jul 3, 2015

Merge pull request #578 from mpreisler/build_all_guides
Build HTML guide for every profile in input XCCDF or datastream

@iankko iankko merged commit d75b496 into OpenSCAP:master Jul 3, 2015

@mpreisler mpreisler deleted the mpreisler:build_all_guides branch Jul 3, 2015

@cliffbdf

This comment has been minimized.

Show comment
Hide comment
@cliffbdf

cliffbdf Jul 23, 2015

New to scap - tried running the profile for Centos7

(from https://github.com/OpenSCAP/scap-security-guide/releases/download/v0.1.24/scap-security-guide-0.1.24.zip)

and it fails with lots of errors similar to,
File 'scap/scap-security-guide-0.1.24/ssg-centos7-ds.xml' line 34925: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}object_component': No match found for key-sequence ['oval:ssg:obj:1231'] of keyref '{http://oval.mitre.org/XMLSchema/oval-definitions-5}objectKeyRef'.

Any ideas on what I am doing wrong? I ran with --cpe /usr/share/openscap/cpe/openscap-cpe-dict.xml

I also ran a "oscap xccdf validate on the ssg-centos7-ds.xml" file and it said that http://scap.nist.gov/schema/scap/source/1.2 is not a known XCCDF namespace, and indeed that URL does not exist.

New to scap - tried running the profile for Centos7

(from https://github.com/OpenSCAP/scap-security-guide/releases/download/v0.1.24/scap-security-guide-0.1.24.zip)

and it fails with lots of errors similar to,
File 'scap/scap-security-guide-0.1.24/ssg-centos7-ds.xml' line 34925: Element '{http://oval.mitre.org/XMLSchema/oval-definitions-5}object_component': No match found for key-sequence ['oval:ssg:obj:1231'] of keyref '{http://oval.mitre.org/XMLSchema/oval-definitions-5}objectKeyRef'.

Any ideas on what I am doing wrong? I ran with --cpe /usr/share/openscap/cpe/openscap-cpe-dict.xml

I also ran a "oscap xccdf validate on the ssg-centos7-ds.xml" file and it said that http://scap.nist.gov/schema/scap/source/1.2 is not a known XCCDF namespace, and indeed that URL does not exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment