Megatron - A System for Abuse- and Incident Handling
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.settings Added UTF-8 encoding for two files. Oct 8, 2013
conf Changed mail addresses handling for ticket system Jan 27, 2015
doc Version 1.1.1, for info see release-note.txt Mar 29, 2017
launch GitHub import: Megatron repository moved to GitHub Jul 8, 2013
lib-src GitHub import: Megatron repository moved to GitHub Jul 8, 2013
lib GitHub import: Megatron repository moved to GitHub Jul 8, 2013
script Initial revision: Utility script for generating organization reports. May 27, 2014
sql Bug fixes and changes of features after beta testing. Oct 10, 2014
src-test/se/sitic/megatron TicketHandler functionality added Nov 17, 2014
src Version 1.1.1, for info see release-note.txt Mar 29, 2017
test-data New: --whois prints machine info to stdout from a list of IPs, hostna… Jun 4, 2014
.classpath GitHub import: Megatron repository moved to GitHub Jul 8, 2013
.gitignore Ignore /local-files Oct 1, 2013
.project GitHub import: Megatron repository moved to GitHub Jul 8, 2013
LICENSE GitHub import: Megatron repository moved to GitHub Jul 8, 2013
NOTICE GitHub import: Megatron repository moved to GitHub Jul 8, 2013
README.md Update README.md Jan 22, 2015
Version.template Version 1.1.1, for info see release-note.txt Mar 29, 2017
build.properties Version 1.1.1, for info see release-note.txt Mar 29, 2017
build.xml GitHub import: Megatron repository moved to GitHub Jul 8, 2013
build_common.xml GitHub import: Megatron repository moved to GitHub Jul 8, 2013
megatron-dev.sh Organization database/object model changes, after comments from Tor -… Sep 29, 2014
megatron.bat GitHub import: Megatron repository moved to GitHub Jul 8, 2013
megatron.sh Improved: Class can be specified in megatron.sh using --class. Sep 29, 2014

README.md

Megatron - A System for Abuse- and Incident Handling

Megatron is a tool implemented by CERT-SE which collects and analyses log files with bad machines, e.g. from Shadowserver. Apart from abuse mail handling, Megatron can be used to collect statistics, convert log files, and do log file analysis during incident handling.

Major features:

  • Flexible parsing: many input sources are supported without any coding. Regular expressions are used to extract tokens which are bound to variables. A variable corresponds to a field in the database and can be used in for example filters and templates.
  • Organization matching: Megatron tries to match every input record to an organization using its IP-blocks, ASNs, and domain names. Each organization have a priority which makes it possible to filter out important records.
  • Database: input records are stored in a relational database (MySQL is default), making data mining possible.
  • Filtering: a wide variety of filters can be used to filter input records.
  • Data decoration: data may be added to the record before it is saved to the database. Types of lookups: IP to ASN, IP to country code, IP to hostname, hostname to IP, URL to hostname, hostname to country code, and IP to geolocation (longitude, latitude, and city). Three of MaxMind's databases are supported (ASN, City, and Country).
  • Performance: large data volumes can be handled. All lookups except DNS queries are local, for instance ASN lookups uses database queries by importing a BGP routing table to the database.

For all features, see readme-general.txt

Example of Usage

Convert a file with IP addresses to a pipe-separated file with IP, AS, country code, and hostname:

$ megatron.sh --job-type ip-flowing --export --no-db test-data/multiple-ips-per-line3.log

As above but output file is tab-separated and contains geolocation data:

$ megatron.sh --job-type ip-flowing-verbose --export --no-db test-data/multiple-ips-per-line3.log

Prints information about two IP addresses (hostnames or URLs works as well):

$ megatron.sh --whois 130.238.7.133 217.21.237.35
IP              | AS     | CC | Hostname                                      | AS Name                                       | Organization
130.238.7.133   | 1653   | SE | live.webb.uu.se                               | SUNET Swedish University Network              | Uppsala universitet
217.21.237.35   | 29672  | SE | stockholm.se                                  | S:t Erik Kommunikation AB                     | Stockholms stad

Prints information about URLs in specified file:

$ megatron.sh --whois infection-urls.txt

Process file and save result in the database:

$ megatron.sh --job-type shadowserver-drone test-data/2009-06-08-drone-report-se.log

Preview of mail to be sent:

$ megatron.sh --job shadowserver-drone_2013-06-22_160142 --id 4242 --mail-dry-run

Send emails for the job:

$ megatron.sh --job shadowserver-drone_2009-06-22_160142 --id 4242 --mail

Megatron is designed to automate day-to-day abuse handling. Running Megatron from the command line is only necessary in special cases.

Download

Binaries, config files, and source code are included in this tarball:

MD5: 816a6b9644cc929822e861221b07ea37
SHA1: 77c8c1bedd6318574b34d85daaaae8a6d94e6b1e

Installation

Megatron is cross-platform as it is implemented in Java and Python. It has been tested on Windows 7 and Ubuntu 12.04. Megatron requires a MySQL database. Do not worry, the installation is easy and described in detail in this document: readme-install.txt

Documentation

To get a grasp of Megatron, we recommend skimming the following:

Other resources are this presentation and the projects Wiki.

License

Megatron is distributed under the terms of the Apache Software Foundation license version 2.0, which is included in the file LICENSE in the root of the project.

Support and Warranty

CERT-SE does not offer any support, and Megatron is provided "as is" without warranty of any kind.

However, we are a bunch of nerds at CERT-SE and we would love to hear from you and discuss technical stuff. Send feedback to cert@cert.se. Our PGP keys can be found on CERT-SE's contact page.

Mailing List

Megatron Hacking is the mailing list for Megatron. Join by sending an empty email to:

megatron-hacking+subscribe@googlegroups.com

Or you can go to the following URL:

https://groups.google.com/group/megatron-hacking/subscribe

The mailing list is for discussions about Megatron (installation problems, bug fixes, feature requests, etc.). Keep in mind that the list is public.