diff --git a/README.md b/README.md deleted file mode 100644 index ebdc8a5..0000000 --- a/README.md +++ /dev/null @@ -1,118 +0,0 @@ -# BinaryAlert: Serverless, Real-time & Retroactive Malware Detection -[![Build Status](https://travis-ci.org/airbnb/binaryalert.svg?branch=master)](https://travis-ci.org/airbnb/binaryalert) -[![Coverage Status](https://coveralls.io/repos/github/airbnb/binaryalert/badge.svg?branch=master)](https://coveralls.io/github/airbnb/binaryalert?branch=master) -[![Documentation Status](https://readthedocs.org/projects/binaryalert/badge/?version=latest)](http://www.binaryalert.io/?badge=latest) - - -![BinaryAlert Logo](docs/images/logo.png) - -BinaryAlert is an open-source serverless AWS pipeline where any file uploaded to an S3 bucket is -immediately scanned with a configurable set of [YARA](https://virustotal.github.io/yara/) rules. -An alert will fire as soon as any match is found, giving an incident response team the ability to -quickly contain the threat before it spreads. - - -## Features - * **Built with Amazon Web Services (AWS):** An AWS account is all you need to deploy BinaryAlert. - * **Broad YARA Support:** Add your own YARA rules and/or automatically clone them from third-party - repos. [`PE`](http://yara.readthedocs.io/en/latest/modules/pe.html), - [`math`](http://yara.readthedocs.io/en/latest/modules/math.html), and - [`hash`](http://yara.readthedocs.io/en/latest/modules/hash.html) YARA modules are supported. - * **Real-Time:** Files uploaded to BinaryAlert (S3 bucket) are immediately queued for analysis. - * **Serverless:** All computation is handled by [Lambda](https://aws.amazon.com/lambda/) - functions. No servers to manage means stronger security and automatic scaling! - * **Infrastructure-as-Code:** The entire infrastructure is described with - [Terraform](https://www.terraform.io) configuration files, enabling anyone to deploy BinaryAlert - in a matter of minutes with a single command. - * **Retroactive Analysis:** After updating the YARA ruleset, BinaryAlert will retroactively scan - the entire file corpus to find any new matches. - * **Easily Configurable:** BinaryAlert configuration is managed in a single Terraform variables - file. - * **Quality Code:** Written in Python3 with unit tests and linting to ensure a clean and reliable - codebase. - * **Low Cost:** The AWS bill is based only on how many files are analyzed. - - -## Quick Start - 1. Install dependencies - 1. Install Python3.6, pip3, [virtualenv](https://virtualenv.pypa.io/en/stable/), and - [Terraform v0.10.1+](https://www.terraform.io/downloads.html). - 2. Create a virtual environment: `virtualenv -p python3 venv` - 3. Activate the virtual env: `source venv/bin/activate` - 4. Install third-party libraries: `pip3 install -r requirements.txt` - 1. If the installation encounters problems finding - `openssl.h`, try `export CFLAGS='-I/usr/local/opt/openssl/include'` before the install. - 2. Configure settings - 1. Set your AWS credentials using any - [method supported by Terraform](https://www.terraform.io/docs/providers/aws/#authentication). - The two simplest options are to run `aws configure` (saves `~/.aws/credentials` file) or - ```bash - export AWS_DEFAULT_REGION="region-name" - export AWS_ACCESS_KEY_ID="access-key" - export AWS_SECRET_ACCESS_KEY="secret-key" - ``` - 2. Fill out the base configuration options in [`terraform.tfvars`](terraform/terraform.tfvars) - 3. Deploy: `python3 manage.py deploy` - 4. In order to receive YARA match alerts, you must manually subscribe to the generated SNS topics. - Go to the [SNS console](https://console.aws.amazon.com/sns/v2/home) and - [add a subscription](http://docs.aws.amazon.com/sns/latest/dg/SubscribeTopic.html) to the - `*_binaryalert_yara_match_alerts` topic (which receives YARA match alerts) and the - `*_binaryalert_metric_alarms` topic (which receives CloudWatch alerts if the service is down). - SNS supports a variety of subscription endpoints, including email and SMS. SNS subscriptions - must be confirmed by the destination, which is why this step can't be automated by Terraform. - -That's it! Now any file you upload to the BinaryAlert S3 bucket will automatically trigger YARA -analysis and you can rest easier knowing that your files are safe. - - -## CLI Tool: `manage.py` -For simplicity, BinaryAlert management commands are bundled together in [`manage.py`](manage.py). - -Usage: `python3 manage.py [--help] [command]` - - -## YARA Rules -YARA rules are stored in the [rules/](rules) folder. See [rules/README.md](rules/README.md) for more -information about adding and updating YARA rules. - - -## Architecture - ![BinaryAlert Architecture](docs/images/architecture.png) - - 1. The organization collects files and delivers them to their BinaryAlert S3 bucket. - Files of interest could include executable binaries, email attachments, documents, etc. - 2. Every file uploaded to the S3 bucket is immediately queued for analysis. - 3. A dispatching Lambda function runs every minute, grouping files into batches and invoking up to - dozens of analyzers in parallel. - 4. Each analyzer scans its files using a list of pre-compiled YARA rules. - 5. YARA matches are saved to DynamoDB and an alert is sent to an SNS topic. We use StreamAlert to - dispatch these alerts, but other organizations can instead consume the alerts via email or any - other supported SNS subscription. - 6. For retroactive analysis, a batching Lambda function enqueues the entire S3 bucket to be - re-analyzed. - 7. Configurable CloudWatch alarms will trigger if any BinaryAlert component is behaving - abnormally. This will notify a different SNS topic than the one used for YARA match alerts. - - -## Updating Pip Packages -The exact `pip3` package versions used are frozen in -[`requirements.txt`](requirements.txt). However, to make upgrading packages easier, -[`requirements_top_level.txt`](requirements_top_level.txt) contains only the top-level packages -required by BinaryAlert. To upgrade the package requirements, - -```bash -pip3 install -r requirements_top_level.txt --upgrade -pip3 freeze > requirements.txt -``` - - -## Directory Overview - * [`lambda_functions`](lambda_functions): Source code for each BinaryAlert Lambda function. - * [`rules`](rules): Collection of public and private YARA rules. - * [`terraform`](terraform): AWS infrastructure represented as Terraform configuration files. - * [`tests`](tests): Unit tests amd mocks. - -## Links -* [Announcement Post](https://medium.com/airbnb-engineering/binaryalert-real-time-serverless-malware-detection-ca44370c1b90) -* [Twitter](https://twitter.com/binaryalert_io) (unofficial) -* [Slack](https://binaryalert.herokuapp.com/) (unofficial) diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..03abf49 --- /dev/null +++ b/README.rst @@ -0,0 +1,36 @@ +BinaryAlert: Serverless, Real-Time & Retroactive Malware Detection +================================================================== +.. image:: https://travis-ci.org/airbnb/binaryalert.svg?branch=master + :target: https://travis-ci.org/airbnb/binaryalert + :alt: Build Status + +.. image:: https://coveralls.io/repos/github/airbnb/binaryalert/badge.svg?branch=master + :target: https://coveralls.io/github/airbnb/binaryalert?branch=master + :alt: Coverage Status + +.. image:: https://readthedocs.org/projects/binaryalert/badge/?version=latest + :target: http://www.binaryalert.io/?badge=latest + :alt: Documentation Status + +| + +.. image:: docs/images/logo.png + :align: center + :scale: 75% + :alt: BinaryAlert Logo + +BinaryAlert is an open-source serverless AWS pipeline where any file uploaded to an S3 bucket is +immediately scanned with a configurable set of `YARA `_ rules. +An alert will fire as soon as any match is found, giving an incident response team the ability to +quickly contain the threat before it spreads. + +Read the documentation at `binaryalert.io `_! + + +Links +----- + +- `Announcement Post `_ +- `Documentation `_ +- `Twitter `_ (unofficial) +- `Slack `_ (unofficial) diff --git a/lambda_functions/analyzer/README.md b/lambda_functions/analyzer/README.md deleted file mode 100644 index 5279f62..0000000 --- a/lambda_functions/analyzer/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# YARA Analyzer -This Lambda function is the core of BinaryAlert. Each invocation downloads one or more binaries from -S3, scans them against all available YARA rules, and forwards any matches to Dynamo and SNS. - - -## Updating YARA-Python -The [`yara-python`](https://github.com/VirusTotal/yara-python) library is natively compiled. -It must therefore be pre-built on an Amazon Linux AMI in order to run in Lambda. -This has already been done for you: [`yara_python_3.6.3.zip`](yara_python_3.6.3.zip) contains the -pre-built `yara_python` library (v3.6.3) for the Lambda environment and is automatically bundled -on every deploy. - -If, however, you need to update or re-create the zipfile, SSH to an EC2 instance running the -[AWS Lambda AMI](http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html) -and install `yara-python`: - -``` -$ sudo su -# yum update -# yum install gcc openssl-devel.x86_64 python35-devel.x86_64 python35-pip.noarch -# python3 - >>> import pip - >>> pip.main(['install', '--upgrade', 'pip']) - >>> exit() -# python3 - >>> import pip - >>> pip.main(['install', 'yara-python', '--target', '.']) - >>> exit() -# mv yara.cpython-35m-x86_64-linux-gnu.so yara.so -# cp /usr/lib64/libpython3.5m.so.1.0 . -# zip -r yara_python_VERSION.zip * -``` - -Some notes: -* Python3.6 is not currently available in the public Lambda AMI. You can either manually install -Python3.6 from source or (what's done here) include the required Python3.5 bytecode in the zipfile. -* The openssl development libraries are required to support the "hash" module. - -Then replace [`yara_python_3.6.3.zip`](yara_python_3.6.3.zip) in the repo with the newly generated -package from the EC2 instance and update the filename in [`manage.py`](../../manage.py). diff --git a/lambda_functions/analyzer/README.rst b/lambda_functions/analyzer/README.rst new file mode 100644 index 0000000..a18140d --- /dev/null +++ b/lambda_functions/analyzer/README.rst @@ -0,0 +1,41 @@ +YARA Analyzer +============= +This Lambda function is the core of BinaryAlert. Each invocation downloads one or more binaries from +S3, scans them against all available YARA rules, and forwards any matches to Dynamo and SNS. + + +Updating YARA-Python +-------------------- +The `yara-python `_ library is natively compiled. +It must therefore be pre-built on an Amazon Linux AMI in order to run in Lambda. +This has already been done for you: ``yara_python_3.6.3.zip`` contains the +pre-built ``yara_python`` library (v3.6.3) for the Lambda environment and is automatically bundled +on every deploy. + +If, however, you need to update or re-create the zipfile, SSH to an EC2 instance running the +`AWS Lambda AMI `_ +and install ``yara-python``: + +.. code-block:: bash + + $ sudo su + # yum update + # yum install gcc openssl-devel.x86_64 python35-devel.x86_64 python35-pip.noarch + # python3 + >>> import pip + >>> pip.main(['install', '--upgrade', 'pip']) + >>> exit() + # python3 + >>> import pip + >>> pip.main(['install', 'yara-python', '--target', '.']) + >>> exit() + # mv yara.cpython-35m-x86_64-linux-gnu.so yara.so + # cp /usr/lib64/libpython3.5m.so.1.0 . + # zip -r yara_python_VERSION.zip * + +Then replace ``yara_python_3.6.3.zip`` in the repo. + +Some notes: + +- Python3.6 is not currently available in the public Lambda AMI. You can either manually install Python3.6 from source or (what's done here) include the required Python3.5 bytecode in the zipfile. +- The openssl development libraries are required to support the "hash" module. diff --git a/lambda_functions/downloader/README.md b/lambda_functions/downloader/README.md deleted file mode 100644 index 68fdec8..0000000 --- a/lambda_functions/downloader/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# CarbonBlack Binary Downloader -This optional Lambda function copies a binary from CarbonBlack Enterprise Response into the BinaryAlert S3 bucket for analysis. -It can invoked every time CarbonBlack logs a `binarystore.file.added` event over the server message bus. - - -## One-Time Copy All CarbonBlack Binaries -`python3 copy_all.py` will enumerate all CarbonBlack binaries and copy each one into S3 (keyed by a -random UUID). It uses multiple processes, but can still take several hours. - -Note that CarbonBlack only stores the first 25MB of binary data. The CarbonBlack MD5 will cover the entire -original file, but only the first 25MB of data will be copied to S3. - -## Cbapi Pip Dependency -The `cbapi` library works best when pre-built on the Lambda AMI. Follow the same instructions given -in the [analyzer README](../analyzer/README.md) to upgrade `cbapi_1.3.2.zip` when needed. diff --git a/lambda_functions/downloader/README.rst b/lambda_functions/downloader/README.rst new file mode 100644 index 0000000..ba3a1c1 --- /dev/null +++ b/lambda_functions/downloader/README.rst @@ -0,0 +1,11 @@ +CarbonBlack Binary Downloader +============================= +This optional Lambda function copies a binary from CarbonBlack Enterprise Response into the BinaryAlert S3 bucket for analysis. +It can invoked every time CarbonBlack logs a ``binarystore.file.added`` event over the server message bus. + +For more information, see the `documentation `_. + +Cbapi Pip Dependency +-------------------- +The ``cbapi`` library works best when pre-built on the Lambda AMI. Follow the same instructions given +in the `analyzer README <../analyzer/README.rst>`_ to upgrade ``cbapi_1.3.2.zip`` when needed. diff --git a/manage.py b/manage.py index c233703..e1c07a6 100755 --- a/manage.py +++ b/manage.py @@ -478,8 +478,8 @@ def live_test(self) -> None: if dynamo_record_found: print('\nLive test succeeded! Verify the alert was sent to your SNS subscription(s).') else: - # TODO: Link to troubleshooting documentation - raise TestFailureError('\nLive test failed!') + raise TestFailureError( + '\nLive test failed! See https://binaryalert.io/troubleshooting-faq.html') @staticmethod def unit_test() -> None: diff --git a/rules/README.md b/rules/README.md deleted file mode 100644 index a9aa449..0000000 --- a/rules/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# BinaryAlert YARA Rules -This folder contains the YARA rules that are bundled into BinaryAlert. - - -## Adding/Updating YARA Rules -To add or update YARA rules: - 1. Clone YARA rules from other open-source repos: `python3 ../manage.py update_rules`. - At the time of writing, third-party YARA rules are pulled from subsets of - [Neo23x0/signature-base](https://github.com/Neo23x0/signature-base) and - [YARA-Rules/rules](https://github.com/YARA-Rules/rules). - 2. Add your own custom `.yar` or `.yara` rules files anywhere in this directory tree. - - -### Supported YARA Modules -YARA has a variety of [different modules](http://yara.readthedocs.io/en/latest/modules.html); only -the `Math` and `PE` modules are officially supported at this time. - - -## Compiling YARA Rules -YARA rules can be compiled into a single binary file: `python3 compile_rules.py`. This happens -automatically during unit tests and for Lambda deployment. -See the Troubleshooting section below for help resolving any issues that may arise. - - -### Troubleshooting Rule Compilation -YARA rules can fail to compile for a variety of reasons. In most cases, we just remove the offending -rules file. A few examples are listed below: - - -#### Failures During Local Testing - -``` -yara.SyntaxError: ...: internal fatal error -``` - -This is a [known issue](https://github.com/Yara-Rules/rules/issues/176) due to YARA being unable to -handle very long regexes. - -``` -yara.SyntaxError: ...: undefined identifier "extension" -``` -This rule is expecting an -[external variable](http://yara.readthedocs.io/en/latest/writingrules.html#external-variables) -named "extension." Look at the YARA rule to understand what it's expecting the variable to -represent. Add a new external variable definition during rule compilation and rule matching. - - -#### Failures In Lambda -If, when testing in the Lambda console, you see an error similar to `internal error: 34: Error`, -it's likely that there is a new rule using an unsupported module. The rule may pass local testing -but fail in the Lambda env because the sets of YARA modules are different between -the two environments.