Merge e5092aa into 701dbb1

cisagov · Oct 1, 2021 · 698ea42 · 698ea42
2 parents 701dbb1 + e5092aa
commit 698ea42
Show file tree

Hide file tree

Showing 20 changed files with 1,737 additions and 65 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -109,6 +109,8 @@ repos:
     hooks:
       - id: mypy
         additional_dependencies:
+          - types-requests
+          - types-PyYAML
           - types-setuptools
   - repo: https://github.com/asottile/pyupgrade
     rev: v2.23.3

diff --git a/README.md b/README.md
@@ -1,68 +1,96 @@
-# Posture and Exposure (P&E) Reports #
+# Posture & Exposure Reports (P&E Reports) #
 
 [![GitHub Build Status](https://github.com/cisagov/pe-reports/workflows/build/badge.svg)](https://github.com/cisagov/pe-reports/actions)
 [![Coverage Status](https://coveralls.io/repos/github/cisagov/pe-reports/badge.svg?branch=develop)](https://coveralls.io/github/cisagov/pe-reports?branch=develop)
 [![Total alerts](https://img.shields.io/lgtm/alerts/g/cisagov/pe-reports.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/cisagov/pe-reports/alerts/)
 [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/cisagov/pe-reports.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/cisagov/pe-reports/context:python)
-[![Known Vulnerabilities](https://snyk.io/test/github/cisagov/pe-reports/develop/badge.svg)](https://snyk.io/test/github/cisagov/pe-reports)
 
-This package is used to generate encrypted Posture and Exposure (P&E) PDF
-reports using raw_data.xlsx files.
+This package is used to generate and deliver CISA Posture & Exposure Reports
+(P&E Reports). Reports are delivered by email and include an encrypted PDF
+attachment with a series of embedded raw-data files of the collected materials.
+The reports are delivered in a two step process. First the `pe_reports` module
+collects the raw-data and creates the encrypted PDFs. The `pe_mailer` then
+securely delivers the content.
 
-## Device Setup ##
+Topics of interest include *Exposed Credentials, Domain Masquerading, Malware,
+Inferred Vulnerabilities and the Dark Web*. The data collected for the reports
+is gathered on the 1st and 15th of each month.
 
-Install [LibreOffice](https://www.libreoffice.org/get-help/install-howto/macos/)
-for PowerPoint to PDF conversion.
+## Requirements ##
 
-Install python 3
-
-(Optional) [Setting up your Mac](https://github.com/cisagov/development-guide/blob/develop/dev_envs/mac-env-setup.md)
+- [Python Environment](CONTRIBUTING.md#creating-the-python-virtual-environment)
+- [cisagov MongoDB](https://github.com/cisagov/mongo-db-from-config)
+- [cisagov AWS SES](https://github.com/cisagov/cool-dns-cyber.dhs.gov)
 
 ## Installation ##
 
-Please see the
-[Creating the Python virtual environment](CONTRIBUTING.md#creating-the-python-virtual-environment)
-section of the [CONTRIBUTING](CONTRIBUTING.md) document for information about
-setting up a Python virtual environment.
-
-Required configurations:
-*You must have access to the cyhy database
-Install [cisagov/mongo-db-from-config](https://github.com/cisagov/mongo-db-from-config)
-and follow the instructions to create the yaml file.
-The report generator will read `/secrets/database_creds.yml` by default if no
-yaml filepath is provided.
-
-To generate a P&E report:
+- `git clone https://github.com/cisagov/pe-reports.git`
 
-```console
-python3 /pe-reports/src/pe_reports YYYY-MM-DD DATA_DIRECTORY OUTPUT_DIRECTORY [OPTIONS]
-```
+- `pip install -e .`
 
-## Making Changes ##
+## Create P&E Reports ##
 
-To change any general report format/standard visuals edit
-`/src/data/shell/pe_shell.pptx`
+- Connect to [cisagov MongoDB](https://github.com/cisagov/mongo-db-from-config)
 
-To make any style changes, edit `/src/pe_reports/stylesheet.py`
+```consol
+Usage:
+  pe-reports [--pe-report-dir=DIRECTORY] [--db-creds-file=FILENAME] [--log-level=LEVEL]
 
-To change metrics, edit `/src/pe_reports/report_metrics.py`
+Arguments:
+  -r --pe-report-dir=DIRECTORY  Directory containing the pe-reports output.
+  -c --db-creds-file=FILENAME   A YAML file containing the Cyber
+                                Hygiene database credentials.
+                                [default: /secrets/database_creds.yml]
+Options:
+  -h --help                     Show this message.
+  -v --version                  Show version information.
+  --log-level=LEVEL             If specified, then the log level will be set to
+                                the specified value.  Valid values are "debug", "info",
+                                "warning", "error", and "critical". [default: info]
+```
 
-To change page setups/graphs, edit `/src/pe_reports/pages.py`
+## Deliver P&E Reports ##
+
+- Connect to [cisagov MongoDB](https://github.com/cisagov/mongo-db-from-config)
+
+- Load [AWS Profile](https://github.com/cisagov/cool-dns-cyber.dhs.gov)
+
+```consol
+Usage:
+  pe-mailer [--pe-report-dir=DIRECTORY] [--db-creds-file=FILENAME] [--log-level=LEVEL]
+
+Arguments:
+  -r --pe-report-dir=DIRECTORY  Directory containing the pe-reports output.
+  -c --db-creds-file=FILENAME   A YAML file containing the Cyber
+                                Hygiene database credentials.
+                                [default: /secrets/database_creds.yml]
+Options:
+  -h --help                     Show this message.
+  -v --version                  Show version information.
+  -s --summary-to=EMAILS        A comma-separated list of email addresses
+                                to which the summary statistics should be
+                                sent at the end of the run.  If not
+                                specified then no summary will be sent.
+  -t --test_emails=EMAILS       A comma-separated list of email addresses
+                                to which to test email send process. If not
+                                specified then no test will be sent.
+  -l --log-level=LEVEL          If specified, then the log level will be set to
+                                the specified value.  Valid values are "debug", "info",
+                                "warning", "error", and "critical". [default: info]
+```
 
 ## Contributing ##
 
-We welcome contributions!  Please see [`CONTRIBUTING.md`](CONTRIBUTING.md) for
-details.
+We welcome contributions!  Please see [`CONTRIBUTING.md`](CONTRIBUTING.md) for details.
 
 ## License ##
 
 This project is in the worldwide [public domain](LICENSE).
 
-This project is in the public domain within the United States, and
-copyright and related rights in the work worldwide are waived through
-the [CC0 1.0 Universal public domain
-dedication](https://creativecommons.org/publicdomain/zero/1.0/).
+This project is in the public domain within the United States, and copyright
+and related rights in the work worldwide are waived through the
+[CC0 1.0 Universal public domain dedication](https://creativecommons.org/publicdomain/zero/1.0/).
 
-All contributions to this project will be released under the CC0
-dedication. By submitting a pull request, you are agreeing to comply
-with this waiver of copyright interest.
+All contributions to this project will be released under the CC0 dedication.
+By submitting a pull request, you are agreeing to comply with this waiver
+of copyright interest.
diff --git a/setup.py b/setup.py
@@ -77,23 +77,31 @@ def get_version(version_file):
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: 3.9",
+        # TODO: PyYAML is not compatable with 3.9
+        # Create Issue
+        # "Programming Language :: Python :: 3.9",
     ],
     python_requires=">=3.6",
     # What does your project relate to?
     keywords="posture and exposure report",
     packages=find_packages(where="src"),
     package_dir={"": "src"},
-    package_data={"pe_reports": ["data/shell/*.pptx"]},
+    package_data={"pe_reports": ["data/shell/*.pptx"], "pe_mailer": ["data/*"]},
     py_modules=[splitext(basename(path))[0] for path in glob("src/*.py")],
     include_package_data=True,
     install_requires=[
+        "boto3",
+        "botocore",
         "docopt",
         "glob2",
+        "mongo-db-from-config @ http://github.com/cisagov/mongo-db-from-config/tarball/develop",
         "openpyxl",
         "pandas",
+        "pyyaml",
+        "types-PyYAML",
         "pymongo",
         "pymupdf",
+        "pystache",
         "python-pptx",
         "schema",
         "setuptools >= 24.2.0",
@@ -113,6 +121,11 @@ def get_version(version_file):
             "pytest",
         ]
     },
-    # Conveniently allows one to run the CLI tool as `pe-reports`
-    entry_points={"console_scripts": ["pe-reports = pe_reports.report_generator:main"]},
+    # Conveniently allows one to run the CLI tool as `pe-reports` or 'pe-mailer'
+    entry_points={
+        "console_scripts": [
+            "pe-reports = pe_reports.report_generator:main",
+            "pe-mailer = pe_mailer.email_reports:main",
+        ]
+    },
 )
diff --git a/src/pe_mailer/__init__.py b/src/pe_mailer/__init__.py
@@ -0,0 +1,9 @@
+"""The pe_mailer library."""
+# We disable a Flake8 check for "Module imported but unused (F401)" here because
+# although this import is not directly used, it populates the value
+# package_name.__version__, which is used to get version information about this
+# Python package.
+
+from ._version import __version__  # noqa: F401
+
+__all__ = ["email_reports", "message", "pe_message", "report_message", "stats_message"]
diff --git a/src/pe_mailer/__main__.py b/src/pe_mailer/__main__.py
@@ -0,0 +1,5 @@
+"""Code to run if this package is used as a Python module."""
+
+from .email_reports import main
+
+main()
diff --git a/src/pe_mailer/_version.py b/src/pe_mailer/_version.py
@@ -0,0 +1,2 @@
+"""Working version of pe-reports."""
+__version__ = "1.0.0"