Excel report by host, new filtering options, new config-file option

TheGroundZero · Nov 17, 2021 · bbe76d2 · bbe76d2
1 parent fe25b83
commit bbe76d2
Show file tree

Hide file tree

Showing 20 changed files with 1,639 additions and 366 deletions.
diff --git a/.github/workflows/pythonpublish.yml b/.github/workflows/pythonpublish.yml
@@ -16,12 +16,12 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install setuptools wheel twine
+        pip install build setuptools wheel twine
         pip install -r requirements.txt
     - name: Build and publish
       env:
         TWINE_USERNAME: ${{ secrets.PYPI_USER }}
         TWINE_PASSWORD: ${{ secrets.PYPI_PASS }}
       run: |
-        python setup.py sdist bdist_wheel
+        python -m build .
         twine upload dist/*
diff --git a/CHANGELOG b/CHANGELOG
@@ -0,0 +1,62 @@
+Changelog
+=========
+
+1.5.0  - New Features:
+
+       - Included option (--config-file) to define a .yaml file with all options but input and output filenames.
+         if this option is used all other options are ignored. any options not present in this file will be set to default
+
+       - Included option (--report-type) to define the type of report and created two new types of reports:
+         - a report summarizing the hosts with the highest number of vulnerabilities and sum of all its cvss severities and including a tab for each host listing each vulnerability
+         - a csv report ordered by host with all vulnerabilities (same fields as the by vulnerability type)
+           (I don't believe it's worth creating a .docx version of this report, so I'm not creating it)
+
+       - Included option (--network-exclude) to define a file with a list of ips or ipcidrs or range of ips (one by line) that will be excluded from the report
+
+       - Included option (--network-include) to define a file with a list of ips or ipcidrs or range of ips (one by line) that will be included in the report
+
+       - Included option (--regex-include) to define a file with a list of regex expressions to include in the report
+         the regex expressions will be matched against the name of the vulnerability
+
+       - Included option (--regex-exclude) to define a file with a list of regex expressions to exclude in the report
+         the regex expressions will be matched against the name of the vulnerability
+
+       - Included option (--cve-include) to define a file with a list of CVE numbers to include in the report
+
+       - Included option (--cve-exclude) to define a file with a list of CVE numbers to exclude from the report
+
+       -Fixes:
+
+       - Major code refactor to include the new reports and the new options
+
+       - Fix module packaging and shell script executions now run ok (import 'main' in top source __init__.py so the egg may be found) 
+
+       - Converted module packaging to python3.6+ packaging using setup.cfg e pyproject.toml
+
+       - Removed package top dir setup.py and requirements.txt files that are not used anymore
+
+       - Updated README.md to reflect those changes
+
+1.4.2  - Fixed "ValueError: Unknown format code 'f' for object of type 'str'"
+
+1.4.1  - Small bugfixes and code refactoring
+
+1.4.0  - Use Word template for report building
+
+1.3.1  - Add charts to Word document using matplotlib. Some code clean-up and small lay-out changes in Excel.
+
+1.3.0  - Fix retrieval of description and other useful info by parsing <tags> instead of <description>
+
+1.2.3  - Implement https://github.com/cr0hn/openvas_to_report/pull/12
+
+1.2.2  - Fix bug where port info was not correctly extracted
+
+1.2.1  - Fix bug where affected hosts were added on wrong row in Excel export
+
+1.2.0  - Functional export to Word document (.docx). Includes some formatting. TODO: graphs
+
+1.1.0a - Support for exporting to Word document (.docx). Limited formatting, needs more testing
+
+1.0.1a - Small updates, preparing for export to other formats
+
+1.0.0  - First official release, supports export to Excel with graphs, ToC and worksheet per vulnerability
diff --git a/README.md b/README.md
@@ -43,9 +43,11 @@ At this moment in time, the script only output .xlsx documents in one format, th
     git clone https://github.com/TheGroundZero/openvasreporting.git
     # Install required python packages
     cd openvasreporting
-    pip3 install -r requirements.txt
-    # Install module (not required when running from repo base folder)
-    #pip3 install .
+    pip3 install pip --upgrade
+    pip3 install build --upgrade
+    python -m build
+    # Install module
+    pip3 install dist/OpenVAS_Reporting-X.x.x-py3-xxxx-xxx.whl
 
 
 Alternatively, you can install the package through the Python package installer 'pip'.  
@@ -68,13 +70,35 @@ This currently has some issues (see #4)
 
 ### Parameters
 
-| Short param | Long param | Description     | Required | Default value                              |
-| :---------: | :--------- | :-------------- | :------: | :----------------------------------------- |
-| -i          | --input    | Input file(s)   | YES      | n/a                                        |
-| -o          | --output   | Output filename | No       | openvas_report                             |
-| -f          | --format   | Output format   | No       | xlsx                                       |
-| -l          | --level    | Minimal level   | No       | n                                          |
-| -t          | --template | Docx template   | No       | openvasreporting/src/openvas-template.docx |
+| Short param | Long param        | Description          | Required | Default value                              |
+| :---------: | :---------------: | :------------------: | :------: | :----------------------------------------- |
+| -i          | --input           | Input file(s)        | YES      | n/a                                        |
+| -o          | --output          | Output filename      | No       | openvas\_report                             |
+| -c          | --config-file     | .yml configuration   | No       | None                                       |
+| -f          | --format          | Output format        | No       | xlsx                                       |
+| -l          | --level           | Minimal level        | No       | n                                          |
+| -T          | --report-type     | Report by            | No       | vulnerability                              |
+|             |                   | vulnerability        |          |                                            |
+|             |                   | or by host           |          |                                            |
+| -t          | --template        | Docx template        | No       | openvasreporting/src/openvas-template.docx |
+| -n          | --network-include | file with networks   | No       | None                                       |
+|             |                   | to include           |          |                                            |
+| -N          | --network-exclude | file with networks   | No       | None                                       |
+|             |                   | to exclude           |          |                                            |
+| -r          | --regex-include   | file with regex to   | No       | None                                       |
+|             |                   | to include from name |          |                                            |
+| -R          | --regex-exclude   | file with regex to   | No       | None                                       |
+|             |                   | to exclude from name |          |                                            |
+| -e          | --cve-include     | file with CVEs to    | No       | None                                       |
+|             |                   | to include from name |          |                                            |
+| -E          | --cve-exclude     | file with CVEs to    | No       | None                                       |
+|             |                   | to exclude from name |          |                                            |
+
+## Filtering options
+
+### The -n/-N/-r/-R/-e/-E options will read a file with one option per line. Networks accepts CIDRs, IP Ranges or IPs. Regex accept any valid regex expression and will be case insensitive matched against the name of the vulnerability. CVEs are inserted in the CVE-YYYY-nnnnn format.
+
+### The -c option will read a .yml file with all configurations. If the -c option is used, any other options but input and output filenames are ignored. There is a sample of a configuration file in the docs/ folder
 
 ## Examples
 
@@ -107,7 +131,7 @@ Worksheets are sorted according to CVSS score and are colored according to the v
 
 Some of the ideas I still have for future functionality:
 
- - list vulnerabilities per host
- - filter by host (scope/exclude) as in OpenVAS2Report
+ - list vulnerabilities per host ==DONE==
+ - filter by host (scope/exclude) as in OpenVAS2Report ==DONE==
  - select threat levels individually (e.g. none and low; but not med, high and crit)
  - import other formats (not only XML), e.g. CSV as suggested in [this issue](https://github.com/TheGroundZero/openvasreporting_server/issues/3)
diff --git a/__init__.py b/__init__.py
@@ -29,5 +29,11 @@
 # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 #
 
+
+#
+# This file exists only so that I can debug using Wings. I don't know why (@dudacgf)
+#
+
 __author__ = 'TheGroundZero (https://github.com/TheGroundZero)'
+__maintainer__ = 'Eduardo Ferreira (@dudacgf)'
 __package__ = str("openvasreporting")
diff --git a/docs/changelog.rst b/docs/changelog.rst
@@ -1,6 +1,42 @@
 Changelog
 =========
 
+1.9.9  - New Features:
+
+       - Included option (--config-file) to define a .yaml file with all options but input and output filenames.
+         if this option is used all other options are ignored. any options not present in this file will be set to default
+
+       - Included option (--report-type) to define the type of report and created two new types of reports:
+         - a report summarizing the hosts with the highest number of vulnerabilities and sum of all its cvss severities and including a tab for each host listing each vulnerability
+         - a csv report ordered by host with all vulnerabilities (same fields as the by vulnerability type)
+           (I don't believe it's worth creating a .docx version of this report, so I'm not creating it)
+
+       - Included option (--network-exclude) to define a file with a list of ips or ipcidrs or range of ips (one by line) that will be excluded from the report
+
+       - Included option (--network-include) to define a file with a list of ips or ipcidrs or range of ips (one by line) that will be included in the report
+
+       - Included option (--regex-include) to define a file with a list of regex expressions to include in the report
+         the regex expressions will be matched against the name of the vulnerability
+
+       - Included option (--regex-exclude) to define a file with a list of regex expressions to exclude in the report
+         the regex expressions will be matched against the name of the vulnerability
+
+       - Included option (--cve-include) to define a file with a list of CVE numbers to include in the report
+
+       - Included option (--cve-exclude) to define a file with a list of CVE numbers to exclude from the report
+
+       -Fixes:
+
+       - Major code refactor to include the new reports and the new options
+
+       - Fix module packaging and shell script executions now run ok (import 'main' in top source __init__.py so the egg may be found) 
+
+       - Converted module packaging to python3.6+ packaging using setup.cfg e pyproject.toml
+
+       - Removed package top dir setup.py and requirements.txt files that are not used anymore
+
+       - Updated README.md to reflect those changes
+
 1.4.2  - Fixed "ValueError: Unknown format code 'f' for object of type 'str'"
 
 1.4.1  - Small bugfixes and code refactoring

diff --git a/docs/config-sample.yml b/docs/config-sample.yml
@@ -0,0 +1,31 @@
+level:
+  medium
+
+format:
+  xlsx
+
+reporttype:
+  host
+
+networks:
+  includes:
+    - 10.0.0.0/8
+    - 172.16.0.0/12
+    - 192.168.0.0/16
+  excludes:
+    - 10.1.16.234
+    - 10.1.16.236
+    - 10.1.16.237
+    - 10.1.16.238
+    - 10.1.16.239
+    - 192.168.16.120
+
+regex:
+  excludes:
+    - Defender
+    - Firefox
+
+# I use this section to filter out recent ms patches not put in production yet
+cve:
+  excludes:
+    - CVE-2021-1971
diff --git a/docs/usage/export-csv-host.rst b/docs/usage/export-csv-host.rst
@@ -0,0 +1,42 @@
+Export to Comma Separated Values
+--------------------------------
+
+When passing the --format csv parameter, the tool will export reports in Comma Separated Values (CSV) format.
+The CSV format is optimized for import in Excel.
+
+Examples
+^^^^^^^^
+
+Create CSV report from 1 OpenVAS XML report using default settings
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+   python3 openvasreporting.py -i openvasreport.xml -f csv -r h
+
+Create CSV report from multiple OpenVAS XML report using default settings
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+   python3 openvasreporting.py -i *.xml -f csv
+   # OR
+   python3 openvasreporting.py -i openvasreport.xml -i openvasreport1.xml -i openvasreport2.xml [-i ...] -f csv
+
+Create CSV report from 1 OpenVAS XML report, reporting only severity level high and up
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+   python3 openvasreporting.py -i openvasreport.xml -o openvas_report -f csv -l h
+
+
+Result
+^^^^^^
+
+The final report will look similar to this:
+
+.. todo::
+   [DOCS] Add examples of CSV report
+
+Vulnerabilities are sorted according to CVSS score (descending) and vulnerability name (ascending).
diff --git a/docs/usage/export-excel-host.rst b/docs/usage/export-excel-host.rst
@@ -0,0 +1,53 @@
+Export to Excel
+---------------
+
+By default (or when passing the --format xlsx parameter), the tool will export reports in Excel (xlsx) format.
+
+This report contains a summary sheet, table of contents, and a sheet per vulnerability containing vulnerability details
+and a list of affected hosts.
+
+Examples
+^^^^^^^^
+
+Create Excel report from 1 OpenVAS XML report using default settings
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+   python3 openvasreporting.py -i openvasreport.xml
+
+Create Excel report from multiple OpenVAS XML report using default settings
+"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+   python3 openvasreporting.py -i *.xml
+   # OR
+   python3 openvasreporting.py -i openvasreport.xml -i openvasreport1.xml -i openvasreport2.xml [-i ...]
+
+Create Excel report from 1 OpenVAS XML report, reporting only severity level high and up
+""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+   python3 openvasreporting.py -i openvasreport.xml -o openvas_report -f xlsx -l h
+
+
+Result
+^^^^^^
+
+The final report will look similar to this:
+
+.. image:: ../_static/img/screenshot-report.png
+   :alt: Report example screenshot - Summary
+   :width: 30%
+
+.. image:: ../_static/img/screenshot-report1.png
+   :alt: Report example screenshot - Table of Contents
+   :width: 30%
+
+.. image:: ../_static/img/screenshot-report2.png
+   :alt: Report example screenshot - Vulnerability description
+   :width: 30%
+
+Vulnerability detail worksheets are sorted according to CVSS score and are colored according to the threat level.
diff --git a/docs/usage/export.rst b/docs/usage/export.rst
@@ -30,6 +30,25 @@ Command line usage
    | Valid values are: c(ritical), h(igh), m(edium), l(low), n(one)
    | Defaults to: none
 
+\-r, -b, --report-type
+   | Optional 
+   | Selects if will list hosts by vulnerability (v) or vulnerabilities by host (h)
+   | Valid values are: v, h
+   | Defaults to: v
+
+\-e, --network-exclude
+   | Optional
+   | path to a file containing a list of ips, ipcidrs or ipaddrs (one per line) that 
+   | will be excluded from the report
+   | Defaults to: no excluded hosts
+
+\-s, --network-include
+   | Optional
+   | path to a file containing a list of ips, ipcidrs or ipaddrs (one per line) that 
+   | will be included in the report
+   | Defaults to: all hosts with appropriate level will be included
+
+
 \-t, --template
    | Optional, only used with '-f docx'
    | Template document for docx export. Document must contain formatting for styles used in export.
@@ -53,4 +72,6 @@ Command line usage
 
    export-excel
    export-word
-   export-csv
+   export-csv
+   export-excel-by-host
+   export-csv-by-host
diff --git a/openvasreporting/__init__.py b/openvasreporting/__init__.py
@@ -30,4 +30,10 @@
 #
 
 __author__ = 'TheGroundZero (https://github.com/TheGroundZero)'
-__package__ = str("openvasreporting")
+__maintainer__ = 'Eduardo Ferreira (@dudacgf)'
+__package__ = str("openvasreporting")
+
+## import main here so shell script execution finds the egg
+from .openvasreporting import main, convert
+from .libs.config import Config, Config_YAML
+
diff --git a/openvasreporting/__main__.py b/openvasreporting/__main__.py
@@ -3,9 +3,11 @@
 #
 # Project name: OpenVAS Reporting: A tool to convert OpenVAS XML reports into Excel files.
 # Project URL: https://github.com/TheGroundZero/openvasreporting
-from openvasreporting.openvasreporting import main
+
+from .openvasreporting import main
 
 __author__ = 'TheGroundZero (https://github.com/TheGroundZero)'
+__maintainer__ = 'Eduardo Ferreira (@dudacgf)'
 
 if __name__ == '__main__':
     if __package__ is None: