Skip to content
Entrada - A big data tool for network analytics
Branch: master
Clone or download
Maarten Wullink
Maarten Wullink added updated dashboards
Latest commit b73f7f2 Jul 16, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
conf cleanup Jul 10, 2019
dashboard added updated dashboards Jul 16, 2019
docker-compose cleanup Jul 15, 2019
lib/com refactoring/cleanup Jun 13, 2019
samples added samples and fix tabs Jul 15, 2019
src added updated dashboards Jul 16, 2019
.gitignore added more resolver checks and refactoring Jun 7, 2019
CONTRIBUTING.md cleanup Jul 4, 2019
Dockerfile text Jul 15, 2019
LICENSE added license Jul 15, 2019
README.md text Jul 15, 2019
pom.xml cleanup Jul 15, 2019

README.md

ENTRADA

ENTRADA - A big data tool for network analytics.

ENTRADA processes data (PCAP-files) from an input location (local, HDFS or S3) and converts and enriches the PCAP-data to Apache Parquet format, finally sending the results to one of following endpoints:

  • HDFS + Impala (hadoop)
  • S3 + Athena (aws)
  • Local disk (local)

See the database schema for more information about all the database columns.

The data is enriched by adding the following details to each row.

  • Geolocation (Country)
  • Autonomous system (ASN) details
  • Detection of public resolvers (Google, OpenDNS, Quad9 and Cloudflare)
  • TCP round-trip time (RTT)

Apache Impala, AWS Athena or Apache Spark can be used to analyse the generated Parquet data.

ENTRADA handles the required workflow actions such as:

  • Loading and archiving PCAP files
  • Converting and enriching data
  • Creating database schema and tables
  • Creating a S3 bucket
  • Configuring S3 security policy and encryption
  • Creating filesystem directories
  • Moving data files around
  • Uploading data to HDFS or S3
  • Compacting Parquet files on HDFS or S3

For more information see the ENTRADA wiki.

How to use

ENTRADA is deployed using Docker Compose, download one of the example Docker Compose scripts and save it as docker-compose.yml and then edit the script to configure the environment variables to fit your requirements.
Start the container using the docker-compose command:

   docker-compose up

For more more details about deployment and available onfiguration options see the ENTRADA wiki.

License

This project is distributed under the GPLv3, see LICENSE.

Attribution

When building a product or service using ENTRADA, we kindly request that you include the following attribution text in all advertising and documentation.

This product includes ENTRADA created by <a href="https://www.sidnlabs.nl">SIDN Labs</a>, available from
<a href="http://entrada.sidnlabs.nl">http://entrada.sidnlabs.nl</a>.
You can’t perform that action at this time.