tcpdp is TCP dump tool with custom dumper and structured logger written in Go.
Clone or download
Latest commit 54281a6 Dec 16, 2018

README.md

tcpdp Build Status GitHub release codecov

tcpdp is TCP dump tool with custom dumper and structured logger written in Go.

tcpdp has 3 modes:

  • TCP Proxy server mode
  • Probe mode ( using libpcap )
  • Read pcap file mode

Usage

tcpdp proxy : TCP proxy server mode

$ tcpdp proxy -l localhost:12345 -r localhost:1234 -d hex # hex.Dump()
$ tcpdp proxy -l localhost:55432 -r db.internal.example.com:5432 -d pg # Dump query of PostgreSQL
$ tcpdp proxy -l localhost:33306 -r db.example.com:3306 -d mysql # Dump query of MySQL

With server-starter

https://github.com/lestrrat-go/server-starter

$ start_server --port 33306 -- tcpdp proxy -s -r db.example.com:3306 -d mysql

With config file

$ tcpdp proxy -c config.toml

tcpdp probe : Probe mode (like tcpdump)

$ tcpdp probe -i lo0 -t localhost:3306 -d mysql # is almost the same setting as 'tcpdump -i lo0 host 127.0.0.1 and tcp port 3306'
$ tcpdp probe -i eth0 -t 3306 -d hex # is almost the same setting as 'tcpdump -i eth0 tcp port 3306'

tcpdp read : Read pcap file mode

$ tcpdump -i eth0 host 127.0.0.1 and tcp port 3306 -w mysql.pcap
$ tcpdp read mysql.pcap -d mysql -t 3306 -f ltsv

tcpdp config Create config

$ tcpdp config > myconfig.toml

Show current config

$ tcpdp config

config format

[tcpdp]
pidfile = "/var/run/tcpdp.pid"
dumper = "mysql"

[probe]
target = "db.example.com:3306"
interface = "en0"
bufferSize = "2MB"
immediateMode = false
snapshotLength = "auto"
internalBufferLength = 10000

[proxy]
useServerStarter = false
listenAddr = "localhost:3306"
remoteAddr = "db.example.com:3306"

[log]
dir = "/var/log/tcpdp"
enable = true
stdout = true
format = "ltsv"
rotateEnable = true
rotationTime = "daily"
rotationCount = 7
# You can execute arbitrary commands after rotate
# $1 = prev filename
# $2 = current filename
rotationHook = "/path/to/after_rotate.sh"
fileName = "tcpdp.log"

[dumpLog]
dir = "/var/log/dump"
enable = true
stdout = false
format = "json"
rotateEnable = true
rotationTime = "hourly"
rotationCount = 24
fileName = "dump.log"

Installation

$ go get github.com/k1LoW/tcpdp

Architecture

tcpdp proxy connection diagram

      client_addr
           ^
           |        tcpdp
+----------|---------------+
|          v               |
|  proxy_listen_addr       |
|         + ^              |
|         | |   +--------+ |
|         |<----+ dumper | |
|         | |<--+        | |
|         | |   +--------+ |
|         v +              |
|  proxy_client_addr       |
|          ^               |
+----------|---------------+
           |
           v
      remote_addr

tcpdp probe connection diagram

                    server
+--------------------------+
|                          |
|                      +---+---+
|       <--------------| eth0  |----------->
|            interface +---+---+
|            /target     ^ |
|                        | |
|         tcpdp          | |
|        +--------+      | |
|        | dumper +------+ |
|        +--------+        |
+--------------------------+

tcpdp read diagram

                  tcpdp
+--------+ STDIN +--------+ STDOUT
| *.pcap +------>+ dumper +-------->
+--------+       +--------+

tcpdp.log ( tcpdp proxy or tcpdp probe )

key description mode
ts timestamp proxy / probe / read
level log level proxy / probe
msg log message proxy / probe
error error info proxy / probe
caller error caller proxy / probe
conn_id TCP connection ID by tcpdp proxy / probe
target probe target proxy / probe
dumper dumper type proxy / probe
use_server_starter use server_starter proxy
conn_seq_num TCP comunication sequence number by tcpdp proxy
client_addr client address tcpdp.log, hex, mysql, pg
remote_addr remote address proxy
proxy_listen_addr listen address proxy
direction client to remote: -> / remote to client: <- proxy
interface probe target interface probe
mtu interface MTU (Maximum Transmission Unit) probe
mss TCP connection MSS (Max Segment Size) probe
probe_target_addr probe target address probe
buffer_size libpcap buffer_size probe
immediate_mode libpcap immediate_mode probe
snapshot_length libpcap snapshot length probe
internal_buffer_length tcpdp internal packet buffer length probe

Dumper

mysql

MySQL query dumper

NOTICE: MySQL query dumper require --target option when tcpdp proxy tcpdp probe

key description mode
ts timestamp proxy / probe / read
conn_id TCP connection ID by tcpdp proxy / probe / read
conn_seq_num TCP comunication sequence number by tcpdp proxy
client_addr client address proxy
proxy_listen_addr listen address proxy
proxy_client_addr proxy client address proxy
remote_addr remote address proxy
direction client to remote: -> / remote to client: <- proxy
interface probe target interface probe
src_addr src address probe / read
dst_addr dst address probe / read
probe_target_addr probe target address probe
query SQL query proxy / probe / read
stmt_id statement id proxy / probe / read
stmt_prepare_query prepared statement query proxy / probe / read
stmt_execute_values prepared statement execute values proxy / probe / read
character_set character set proxy / probe / read
username username proxy / probe / read
database database proxy / probe / read
seq_num sequence number by MySQL proxy / probe / read
command_id command_id for MySQL proxy / probe / read

pg

PostgreSQL query dumper

NOTICE: PostgreSQL query dumper require --target option tcpdp proxy tcpdp probe

key description mode
ts timestamp proxy / probe / read
conn_id TCP connection ID by tcpdp proxy / probe / read
conn_seq_num TCP comunication sequence number by tcpdp proxy
client_addr client address proxy
proxy_listen_addr listen address proxy
proxy_client_addr proxy client address proxy
remote_addr remote address proxy
direction client to remote: -> / remote to client: <- proxy
interface probe target interface probe
src_addr src address probe / read
dst_addr dst address probe / read
probe_target_addr probe target address probe
query SQL query proxy / probe / read
portal_name portal Name proxy / probe / read
stmt_name prepared statement name proxy / probe / read
parse_query prepared statement query proxy / probe / read
bind_values prepared statement bind(execute) values proxy / probe / read
username username proxy / probe / read
database database proxy / probe / read
message_type message type for PostgreSQL proxy / probe / read

hex

key description mode
ts timestamp proxy / probe / read
conn_id TCP connection ID by tcpdp proxy / probe / read
conn_seq_num TCP comunication sequence number by tcpdp proxy
client_addr client address proxy
proxy_listen_addr listen address proxy
proxy_client_addr proxy client address proxy
remote_addr remote address proxy
direction client to remote: -> / remote to client: <- proxy
interface probe target interface probe
src_addr src address probe / read
dst_addr dst address probe / read
probe_target_addr probe target address probe
bytes bytes string by hex.Dump proxy / probe / read
ascii ascii string by hex.Dump proxy / probe / read

References