Merge pull request #48 from cisagov/wip_pe-mailer

Add P&E Mailer module pe-mailer

.pre-commit-config.yaml

@@ -109,6 +109,8 @@ repos:
     hooks:
       - id: mypy
         additional_dependencies:
+          - types-PyYAML
+          - types-requests
           - types-setuptools
   - repo: https://github.com/asottile/pyupgrade
     rev: v2.23.3

README.md

@@ -1,68 +1,102 @@
-# Posture and Exposure (P&E) Reports #
+# Posture & Exposure Reports (P&E Reports) #
 
 [![GitHub Build Status](https://github.com/cisagov/pe-reports/workflows/build/badge.svg)](https://github.com/cisagov/pe-reports/actions)
 [![Coverage Status](https://coveralls.io/repos/github/cisagov/pe-reports/badge.svg?branch=develop)](https://coveralls.io/github/cisagov/pe-reports?branch=develop)
 [![Total alerts](https://img.shields.io/lgtm/alerts/g/cisagov/pe-reports.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/cisagov/pe-reports/alerts/)
 [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/cisagov/pe-reports.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/cisagov/pe-reports/context:python)
 [![Known Vulnerabilities](https://snyk.io/test/github/cisagov/pe-reports/develop/badge.svg)](https://snyk.io/test/github/cisagov/pe-reports)
 
-This package is used to generate encrypted Posture and Exposure (P&E) PDF
-reports using raw_data.xlsx files.
+This package is used to generate and deliver CISA Posture & Exposure Reports
+(P&E Reports). Reports are delivered by email and include an encrypted PDF
+attachment with a series of embedded raw-data files of the collected materials.
+The reports are delivered in a two step process. First the `pe_reports` module
+collects the raw data and creates the encrypted PDFs. The `pe_mailer` then
+securely delivers the content.
 
-## Device Setup ##
+Topics of interest include *Exposed Credentials, Domain Masquerading, Malware,
+Inferred Vulnerabilities and the Dark Web*. The data collected for the reports
+is gathered on the 1st and 15th of each month.
 
-Install [LibreOffice](https://www.libreoffice.org/get-help/install-howto/macos/)
-for PowerPoint to PDF conversion.
+## Requirements ##
 
-Install python 3
+- [Python Environment](CONTRIBUTING.md#creating-the-python-virtual-environment)
 
-(Optional) [Setting up your Mac](https://github.com/cisagov/development-guide/blob/develop/dev_envs/mac-env-setup.md)
+- [cisagov MongoDB](https://github.com/cisagov/mongo-db-from-config)
+
+- [cisagov AWS SES](https://github.com/cisagov/cool-dns-cyber.dhs.gov)
 
 ## Installation ##
 
-Please see the
-[Creating the Python virtual environment](CONTRIBUTING.md#creating-the-python-virtual-environment)
-section of the [CONTRIBUTING](CONTRIBUTING.md) document for information about
-setting up a Python virtual environment.
+- `git clone https://github.com/cisagov/pe-reports.git`
+
+- `pip install -e .`
 
-Required configurations:
-*You must have access to the cyhy database
-Install [cisagov/mongo-db-from-config](https://github.com/cisagov/mongo-db-from-config)
-and follow the instructions to create the yaml file.
-The report generator will read `/secrets/database_creds.yml` by default if no
-yaml filepath is provided.
+## Create P&E Reports ##
 
-To generate a P&E report:
+- Configure [cisagov MongoDB connection](https://github.com/cisagov/mongo-db-from-config)
 
 ```console
-python3 /pe-reports/src/pe_reports YYYY-MM-DD DATA_DIRECTORY OUTPUT_DIRECTORY [OPTIONS]
+Usage:
+  pe-reports REPORT_DATE DATA_DIRECTORY OUTPUT_DIRECTORY [--db-creds-file=FILENAME] [--log-level=LEVEL]
+
+Arguments:
+  REPORT_DATE                   Date of the report, format YYYY-MM-DD.
+  DATA_DIRECTORY                The directory where the Excel data files are located.
+                                Organized by owner.
+  OUTPUT_DIRECTORY              The directory where the final PDF reports should be saved.
+  -c --db-creds-file=FILENAME   A YAML file containing the Cyber
+                                Hygiene database credentials.
+                                [default: /secrets/database_creds.yml]
+Options:
+  -h --help                     Show this message.
+  -v --version                  Show version information.
+  --log-level=LEVEL             If specified, then the log level will be set to
+                                the specified value.  Valid values are "debug", "info",
+                                "warning", "error", and "critical". [default: info]
 ```
 
-## Making Changes ##
-
-To change any general report format/standard visuals edit
-`/src/data/shell/pe_shell.pptx`
+## Deliver P&E Reports ##
 
-To make any style changes, edit `/src/pe_reports/stylesheet.py`
+- Configure [cisagov MongoDB connection](https://github.com/cisagov/mongo-db-from-config)
 
-To change metrics, edit `/src/pe_reports/report_metrics.py`
+- Load an AWS profile that assumes [this role](https://github.com/cisagov/cool-dns-cyber.dhs.gov/blob/develop/sessendemail_rolerole.tf#L33-L39)
 
-To change page setups/graphs, edit `/src/pe_reports/pages.py`
+```console
+Usage:
+  pe-mailer [--pe-report-dir=DIRECTORY] [--db-creds-file=FILENAME] [--log-level=LEVEL]
+
+Arguments:
+  -p --pe-report-dir=DIRECTORY  Directory containing the pe-reports output.
+  -c --db-creds-file=FILENAME   A YAML file containing the Cyber
+                                Hygiene database credentials.
+                                [default: /secrets/database_creds.yml]
+Options:
+  -h --help                     Show this message.
+  -v --version                  Show version information.
+  -s --summary-to=EMAILS        A comma-separated list of email addresses
+                                to which the summary statistics should be
+                                sent at the end of the run.  If not
+                                specified then no summary will be sent.
+  -t --test_emails=EMAILS       A comma-separated list of email addresses
+                                to which to test email send process. If not
+                                specified then no test will be sent.
+  -l --log-level=LEVEL          If specified, then the log level will be set to
+                                the specified value.  Valid values are "debug", "info",
+                                "warning", "error", and "critical". [default: info]
+```
 
 ## Contributing ##
 
-We welcome contributions!  Please see [`CONTRIBUTING.md`](CONTRIBUTING.md) for
-details.
+We welcome contributions!  Please see [`CONTRIBUTING.md`](CONTRIBUTING.md) for details.
 
 ## License ##
 
 This project is in the worldwide [public domain](LICENSE).
 
-This project is in the public domain within the United States, and
-copyright and related rights in the work worldwide are waived through
-the [CC0 1.0 Universal public domain
-dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+This project is in the public domain within the United States, and copyright
+and related rights in the work worldwide are waived through the
+[CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
 
-All contributions to this project will be released under the CC0
-dedication. By submitting a pull request, you are agreeing to comply
-with this waiver of copyright interest.
+All contributions to this project will be released under the CC0 dedication.
+By submitting a pull request, you are agreeing to comply with this waiver
+of copyright interest.

setup.py

@@ -84,19 +84,25 @@ def get_version(version_file):
     keywords="posture and exposure report",
     packages=find_packages(where="src"),
     package_dir={"": "src"},
-    package_data={"pe_reports": ["data/shell/*.pptx"]},
+    package_data={"pe_reports": ["data/shell/*.pptx"], "pe_mailer": ["data/*"]},
     py_modules=[splitext(basename(path))[0] for path in glob("src/*.py")],
     include_package_data=True,
     install_requires=[
+        "boto3",
+        "botocore",
+        "chevron",
         "docopt",
         "glob2",
+        "mongo-db-from-config @ http://github.com/cisagov/mongo-db-from-config/tarball/develop",
         "openpyxl",
         "pandas",
+        "pyyaml",
         "pymongo",
         "pymupdf",
         "python-pptx",
         "schema",
         "setuptools >= 24.2.0",
+        "types-PyYAML",
     ],
     extras_require={
         "test": [
@@ -113,6 +119,11 @@ def get_version(version_file):
             "pytest",
         ]
     },
-    # Conveniently allows one to run the CLI tool as `pe-reports`
-    entry_points={"console_scripts": ["pe-reports = pe_reports.report_generator:main"]},
+    # Conveniently allows one to run the CLI tool as `pe-reports` or 'pe-mailer'
+    entry_points={
+        "console_scripts": [
+            "pe-reports = pe_reports.report_generator:main",
+            "pe-mailer = pe_mailer.email_reports:main",
+        ]
+    },
 )

src/pe_mailer/__init__.py

@@ -0,0 +1,3 @@
+"""The pe_mailer library."""
+
+__all__ = ["email_reports", "message", "pe_message", "report_message", "stats_message"]

src/pe_mailer/__main__.py

@@ -0,0 +1,5 @@
+"""Code to run if this package is used as a Python module."""
+
+from .email_reports import main
+
+main()

src/pe_mailer/_version.py

@@ -0,0 +1,2 @@
+"""This file defines the version of this module."""
+__version__ = "1.0.0"