Skip to content
Permalink
Browse files

GitBook: [master] 12 pages and one asset modified

  • Loading branch information...
dreadl0ck authored and gitbook-bot committed May 5, 2019
1 parent ed60793 commit f0a60b97328e72c2422b3f789fd45e0a78136bdd
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -4,7 +4,9 @@ description: A brief overview

# Overview

![](.gitbook/assets/netcap-logov2.jpg)


![](.gitbook/assets/screenshot-2019-05-05-at-13.41.40.png)

The _Netcap_ \(NETwork CAPture\) framework efficiently converts a stream of network packets into highly accessible type-safe structured data that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime.

@@ -29,8 +31,8 @@ _Netcap_ uses Google's Protocol Buffers to encode its output, which allows acces

Currently there are 8 applications:

* net.capture \(capture audit records\)
* net.dump \(work with audit records\)
* net.capture \(capture audit records live or from dumpfiles\)
* net.dump \(dump with audit records in various formats\)
* net.label \(tool for creating labeled CSV datasets from netcap data\)
* net.collect \(collection server for distributed collection\)
* net.agent \(sensor agent for distributed collection\)
@@ -44,6 +46,26 @@ Currently there are 8 applications:
* monitoring medical / industrial devices
* research on anomaly-based detection mechanisms
* Forensic data analysis
*
## Demos

A simple demonstration of generating audit records from a PCAP dump file, querying and displaying the collected information in various ways

{% embed url="https://asciinema.org/a/217939" %}



And live operation decoding traffic from my wireless network interface, while I am surfing the web

{% embed url="https://asciinema.org/a/217941" %}



Watch a quick demo of the deep neural network for classification of malicious behavior, on a small PCAP dump file with traffic from the LOKI Bot. First, the PCAP file is parsed with [netcap](https://github.com/dreadl0ck/netcap-tf-dnn/blob/master/github.com/dreadl0ck/netcap), in order to get audit records that will be labeled afterwards with the [netlabel](https://github.com/dreadl0ck/netcap#netlabel-command-line-tool) tool. The labeled CSV data for the TCP audit record type is then used for training \(75%\) and evaluation \(25%\) of the classification accuracy provided by the deep neural network.

{% embed url="https://asciinema.org/a/217944" %}



## License

@@ -13,7 +13,7 @@
* [Distributed Collection](distributed-collection.md)
* [Workers](workers.md)
* [Filtering and Export](filtering-and-export.md)
* [Cheatsheets](cheatsheets.md)
* [Downloads](cheatsheets.md)
* [Internals](internals.md)
* [Metrics](metrics.md)
* [Python Integration](python-integration.md)
@@ -2,7 +2,23 @@
description: A collection of cheatsheets and useful resources
---

# Cheatsheets
# Downloads

## Publications

### Thesis

{% file src=".gitbook/assets/mied18.pdf" %}

### Thesis Presentation

{% file src=".gitbook/assets/mied18\_os.pdf" %}

### SecurIT Cup 2018 Presentation

{% file src=".gitbook/assets/securitcup\_slides\_philipp\_mieden.pdf" %}

## Cheatsheets

### List of all supported protocols and fields

@@ -1,3 +1,7 @@
---
description: Process Netcap audit records and extract the data you are interested in
---

# Filtering and Export

Netcap offers a simple interface to filter for specific fields and select only those of interest. Filtering and exporting specific fields can be performed with all available audit record types, over a uniform command-line interface. By default, output is generated as CSV with the field names added as first line. It is also possible to use a custom separator string. Fields are exported in the order they are named in the select statement. Sub structures of audit records \(for example IPv4Options from an IPv4 packet\), are converted to a human readable string representation. More examples for using this feature on the command-line can be found in the usage section.
@@ -14,12 +14,54 @@ By using a simple reverse proxy for HTTP traffic, the operating system handles t

### Usage

Spin up a single proxy instance from the commandline:

`$ net.proxy -local 127.0.0.1:4000 -remote http://google.com`

Specifiy a custom config file for proxying multiple services:

```text
$ net.proxy -config example_config.yml
```

The default config path is **net.proxy-config.yml**, so if this file exists in the folder where you execute the proxy, you do not need to specify it on the commandline.

### Configuration

For proxying several services, you need to provide a config file, here is a simple example:

```yaml
# Proxies map holds all reverse proxies
proxies:
service1:
local: 127.0.0.1:443
remote: http://127.0.0.1:8080
tls: true
service2:
local: 127.0.0.1:9999
remote: http://192.168.1.20
service3:
local: 127.0.0.1:7000
remote: https://google.com
# CertFile for TLS secured connections
certFile: "certs/cert.crt"
# KeyFile for TLS secured connections
keyFile: "certs/cert.key"
# Logdir is used as destination for the logfile
logdir: "logs"
```

### Help

```text
Usage of net.proxy:
-version bool
print netcap package version and exit
-config string
set config file path (default "net.proxy-config.yml")
-debug
@@ -1,3 +1,7 @@
---
description: Setup instructions
---

# Installation

Installation via go get:
@@ -1,3 +1,7 @@
---
description: Capture full packet payloads
---

# Payload Capture

It is now possible to capture payload data for the following protocols: TCP, UDP, ModbusTCP, USB
@@ -8,3 +12,5 @@ This can be enabled with the **-payload** flag:
$ net.cap -r traffic.pcap -payload
```



@@ -1,3 +1,7 @@
---
description: For those who can't wait to get their hands dirty.
---

# Quickstart

### Basic Commands
@@ -1,3 +1,7 @@
---
description: The Netcap audit record format
---

# Specification

_Netcap_ files have the file extension **.ncap** or **.ncap.gz** if compressed with gzip and contain serialized protocol buffers of one type. Naming of each file happens according to the naming in the [gopacket](https://godoc.org/github.com/google/gopacket) library: a short uppercase letter representation for common protocols, and a camel case version full word version for less common protocols. Audit records are modeled as protocol buffers. Each file contains a header that specifies which type of audit records is inside the file, what version of _Netcap_ was used to generate it, what input source was used and what time it was created. Each audit record should be tagged with the timestamp the packet was seen, in the format _seconds.microseconds_. Output is written to a file that represents each data structure from the protocol buffers definition, i.e. _TCP.ncap_, _UDP.ncap_. For this purpose, the audit records are written as length delimited records into the file.
@@ -1,3 +1,7 @@
---
description: Label audit records for supervised machine learning
---

# Audit Record Labeling

### Introduction
@@ -1,3 +1,7 @@
---
description: Capture traffic sent via Universal Serial Bus (USB) protocol
---

# USB Capture

USB live capture is now possible, currently the following Audit Records exist: USB and USBRequestBlockSetup.
@@ -1,3 +1,7 @@
---
description: This is where the magic happens
---

# Workers

To make use of multi-core processors, processing of packets should happen in an asynchronous way. Since Netcap should be usable on a stream of packets, fetching of packets has to happen sequentially, but decoding them can be parallelized. The packets read from the input data source \(PCAP file or network interface\) are assigned to a configurable number of workers routines via round-robin. Each of those worker routines operates independently, and has all selected encoders loaded. It decodes all desired layers of the packet, and writes the encoded data into a buffer that will be flushed to disk after reaching its capacity.

0 comments on commit f0a60b9

Please sign in to comment.
You can’t perform that action at this time.