Skip to content

Latest commit

 

History

76 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dnstap-filter

A tool to filter records from dnstap sources and write matched records to various outputs.

Installation

go install github.com/kimitoboku/dnstap-filter/cmd/dnstap-filter@latest

Build dependency

The device: input scheme uses libpcap via cgo. Install the development headers before building:

# Debian / Ubuntu
sudo apt-get install libpcap-dev

# macOS (Homebrew)
brew install libpcap

Usage

dnstap-filter --in <input-spec> [--out <output-spec>]... [--filter <expression>]

Input (--in)

Specifies the source of dnstap data. The format is scheme:address.

SchemeExampleDescription
file:<path>file:input.dnstapRead from a dnstap frame stream file
unix:<path>unix:/var/run/named/dnstap.sockListen on a Unix domain socket (server mode)
tcp:<host:port>tcp:0.0.0.0:6000Listen on a TCP port (server mode)
pcap:<path>pcap:input.pcapRead DNS packets from a pcap file
device:<iface>device:en0Live capture from a network interface (requires root)
device:alldevice:allLive capture from all available interfaces (requires root)

A bare path without a scheme (e.g. input.dnstap) is treated as file: for backward compatibility.

Note: device: input requires libpcap-dev (Debian/Ubuntu) or libpcap (macOS via Homebrew) to be installed at build time. Root privileges or appropriate capabilities are required at runtime for live packet capture.

Note: unix: and tcp: inputs run as servers that accept connections from DNS servers (BIND, Unbound, etc.). The process runs until killed (Ctrl+C).

Output (--out)

Specifies the destination for filtered dnstap data. The format is scheme:address.

SchemeExampleDescription
(omitted)Print <time> <Q\vert{}R> <name> <type> [<rcode>] to stdout (default)
stdout:<fields>stdout:time,qr,name,type,rcodeCustomizable stdout output (comma-separated fields)
file:<path>file:output.dnstapWrite dnstap frame stream file (supports SIGHUP log rotation)
unix:<path>unix:/var/run/collector.sockConnect to a Unix domain socket (client mode)
tcp:<host:port>tcp:127.0.0.1:6001Connect to a TCP collector (client mode)
yaml:<path>yaml:/tmp/out.yamlWrite human-readable YAML format
yaml:-yaml:-Write human-readable YAML format to stdout
jsonl:<path>jsonl:/tmp/out.jsonlWrite structured JSONL format (one JSON object per line)
jsonl:-jsonl:-Write structured JSONL format to stdout
dns:<host:port>dns:8.8.8.8:53Replay DNS queries to target server (UDP, fire-and-forget)
stats:<path>stats:report.htmlStatistics report (format by extension: .html, .json, .xml)

A bare path without a scheme (e.g. output.dnstap) is treated as file: for backward compatibility.

--out can be specified multiple times to fan-out filtered data to multiple destinations simultaneously. For example, you can monitor on stdout while also saving to a file:

dnstap-filter --in file:input.dnstap --out stdout:time,name,type --out file:output.dnstap --filter "suffix=example.com."

stdout: fields

The stdout: scheme accepts a comma-separated list of field names to control which columns are printed. If no fields are specified (stdout:), the default set time,qr,name,type,rcode is used.

FieldDescriptionExample
timeTimestamp (query time preferred, then response time)2024-01-01 12:00:00
qrQuery/Response indicatorQ or R
msgtypeFull dnstap message type nameCLIENT_QUERY
nameDNS query namewww.example.com.
typeDNS query typeA, AAAA, MX
rcodeResponse code (omitted for queries)NOERROR, NXDOMAIN
ipClient IP address (QueryAddress)192.168.1.1
# Default fields
dnstap-filter --in file:input.dnstap --out stdout:
# Output: 2024-01-01 12:00:00 Q www.example.com. A
# Output: 2024-01-01 12:00:01 R www.example.com. A NOERROR

# Custom fields: show client IP and full message type
dnstap-filter --in file:input.dnstap --out stdout:time,msgtype,ip,name,type,rcode

# Minimal: name and type only
dnstap-filter --in file:input.dnstap --out stdout:name,type

jsonl: output

The jsonl: scheme outputs each dnstap message as a single-line compact JSON object (JSONL / JSON Lines format). Each line contains dnstap metadata and a parsed DNS message with header flags, question, answer, authority, and additional sections.

{"type":"MESSAGE","message_type":"CLIENT_RESPONSE","timestamp":"2024-01-01T00:00:00Z","socket_family":"INET","socket_protocol":"UDP","query_address":"192.168.1.100","query_port":12345,"dns":{"id":1234,"qr":true,"opcode":"QUERY","rcode":"NOERROR","flags":{"aa":false,"tc":false,"rd":true,"ra":true,"ad":false,"cd":false},"question":[{"name":"www.example.com.","type":"A","class":"IN"}],"answer":[{"name":"www.example.com.","type":"A","class":"IN","ttl":300,"data":"93.184.216.34"}],"authority":[],"additional":[]}}

Top-level fields:

FieldDescription
typeDnstap type (always MESSAGE)
message_typeMessage type (CLIENT_QUERY, CLIENT_RESPONSE, etc.)
timestampTimestamp in RFC3339Nano format (UTC)
socket_familyINET or INET6 (omitted if unavailable)
socket_protocolUDP or TCP (omitted if unavailable)
query_addressClient IP address (omitted if unavailable)
query_portClient port (omitted if 0)
response_addressServer IP address (omitted if unavailable)
response_portServer port (omitted if 0)

The dns object contains the parsed DNS message:

FieldDescription
dns.idDNS message ID
dns.qrtrue for response, false for query
dns.opcodeOpcode (QUERY, IQUERY, STATUS, etc.)
dns.rcodeResponse code (NOERROR, NXDOMAIN, etc.)
dns.flagsHeader flags (aa, tc, rd, ra, ad, cd)
dns.questionQuestion section (array of {name, type, class})
dns.answerAnswer section (array of {name, type, class, ttl, data})
dns.authorityAuthority section
dns.additionalAdditional section

dns: output (DNS replay)

The dns: scheme replays DNS query messages from dnstap data to a target DNS server via UDP. Only query-type messages (CLIENT_QUERY, RESOLVER_QUERY, etc.) are sent; response messages are silently skipped. The output is fire-and-forget — DNS responses from the target server are not read.

If the port is omitted (e.g. dns:8.8.8.8), port 53 is used by default.

Combined with the --speed flag, this can be used to replay captured DNS traffic at the original pace or at a custom speed.

# Replay queries to 8.8.8.8
dnstap-filter --in file:input.dnstap --out dns:8.8.8.8:53

# Replay at original timestamp speed
dnstap-filter --in file:input.dnstap --out dns:8.8.8.8:53 --speed 1

# Replay only A queries to a local server
dnstap-filter --in file:input.dnstap --out dns:127.0.0.1:5353 --filter "qtype=A"

# Monitor on stdout while replaying
dnstap-filter --in file:input.dnstap --out stdout: --out dns:192.168.1.1:53

stats: output (statistics report)

The stats: scheme collects statistics from filtered dnstap messages and writes a report on exit. Statistics are aggregated in 60-second time windows.

The output format is determined by the file extension:

ExtensionFormatDescription
.htmlHTMLHuman-readable report with tables and time-series charts
.jsonJSONMachine-readable structured data
.xmlXMLDSC (DNS Statistics Collector) compatible format
.mdMarkdownMarkdown document with tables
-JSONJSON output to stdout

Collected statistics per window:

  • Total frame count — number of matched messages
  • Top-N queried domains — most frequently queried domain names (default N=20)
  • Query type distribution — count of each DNS query type (A, AAAA, MX, etc.)
  • Response code distribution — count of each response code (NOERROR, NXDOMAIN, etc.)
  • Top-N client IPs — most active client IP addresses (default N=20)

The report includes per-window breakdowns and an all-time summary.

Three optional flags control how statistics are aggregated:

  • --stats-top-n=N — number of top entries to keep in domain and client IP rankings (default: 20)
  • --stats-domain-labels=N — aggregate query names by the last N DNS labels; e.g. N=2 maps www.example.com.example.com. (default: 0 = full qname)
  • --stats-subnet-prefix=N — mask client IPs to a prefix length before counting; e.g. N=24 groups all IPs in the same /24 (default: 0 = no masking)
  • --stats-window=D — time window interval for aggregation (default: 60s); accepts Go duration syntax: 30s, 5m, 1h
# Generate an HTML statistics report (with time-series charts)
dnstap-filter --in file:input.dnstap --out stats:report.html

# Generate a JSON statistics report
dnstap-filter --in file:input.dnstap --out stats:report.json

# Generate a DSC-compatible XML report
dnstap-filter --in file:input.dnstap --out stats:report.xml

# Generate a Markdown report
dnstap-filter --in file:input.dnstap --out stats:report.md

# Print JSON stats to stdout
dnstap-filter --in file:input.dnstap --out stats:-

# Keep only top 50 entries in rankings
dnstap-filter --in file:input.dnstap --out stats:report.html --stats-top-n 50

# Aggregate domains by last 2 labels (e.g. www.example.com. -> example.com.)
dnstap-filter --in file:input.dnstap --out stats:report.html --stats-domain-labels 2

# Group client IPs by /24 subnet
dnstap-filter --in file:input.dnstap --out stats:report.html --stats-subnet-prefix 24

# Combine multiple aggregation options
dnstap-filter --in file:input.dnstap --out stats:report.html \
  --stats-top-n 30 --stats-domain-labels 2 --stats-subnet-prefix 24

# Combine with regular output: monitor on stdout while collecting stats
dnstap-filter --in file:input.dnstap --out stdout --out stats:report.html --filter "suffix=example.com."

# Filter and collect stats simultaneously
dnstap-filter --in unix:/var/run/named/dnstap.sock --out file:output.dnstap --out stats:stats.html --filter "rcode=NXDOMAIN"

Note: unix: and tcp: outputs are clients that connect to a collector. If the collector is not available, they retry every 10 seconds.

Flags

Required:

  • --in: input spec (required unless --print-filter-tree is used)

Optional:

  • --out: output spec (repeatable; default: print <time> <Q|R> <name> <type> [<rcode>] to stdout)
  • --filter: filter expression
  • --print-filter-tree: print the parsed filter tree and exit
  • --cout, -c: process only the first N records from the beginning of input
  • --speed: output pacing based on dnstap timestamps (default 0 = max speed)
    • 0 = max speed (no delay, current behavior)
    • 1 = realtime (replay at original timestamp intervals)
    • 2 = 2x speed (half the delay), 0.5 = half speed (double the delay)
  • --stats-top-n: top-N entry limit for domain and client IP rankings in stats: output (default: 20)
  • --stats-domain-labels: aggregate query names by last N DNS labels in stats: output (default: 0 = full qname)
  • --stats-subnet-prefix: mask client IPs to prefix length in stats: output (default: 0 = no masking)
  • --stats-window: time window interval for stats aggregation (default: 60s; accepts Go duration syntax: 30s, 5m, 1h)

Filter expression

Predicates can be combined with logical operators and, or, not, and (...) for grouping. Operator precedence: not binds tightest, then and, then or; parentheses override precedence. Omitting --filter matches all messages.

Predicates

ip=<addr> — IP address match

Matches messages where the query source address (QueryAddress) or the response destination address (ResponseAddress) equals <addr>.

Use src.ip or dst.ip to restrict matching to a single direction:

PredicateMatches
ip=<addr>QueryAddress or ResponseAddress
src.ip=<addr>QueryAddress only
dst.ip=<addr>ResponseAddress only
--filter "ip=192.168.1.10"
--filter "src.ip=192.168.1.10"
--filter "dst.ip=10.0.0.1"

subnet=<CIDR> — IP subnet match

Matches messages where the query source address or response destination address falls within the given CIDR range.

Use src.subnet or dst.subnet to restrict matching to a single direction:

PredicateMatches
subnet=<CIDR>QueryAddress or ResponseAddress
src.subnet=<CIDR>QueryAddress only
dst.subnet=<CIDR>ResponseAddress only
--filter "subnet=192.168.0.0/24"
--filter "src.subnet=10.0.0.0/8"
--filter "dst.subnet=172.16.0.0/12"

port=<number> — Port number match

Matches messages where the query source port (QueryPort) or response destination port (ResponsePort) equals <number>.

Use src.port or dst.port to restrict matching to a single direction:

PredicateMatches
port=<number>QueryPort or ResponsePort
src.port=<number>QueryPort only
dst.port=<number>ResponsePort only
--filter "port=53"
--filter "src.port=12345"
--filter "dst.port=53"

fqdn=<name> — Query name exact match

Matches messages whose DNS question name equals <name> exactly (including the trailing dot).

--filter "fqdn=www.example.com."

suffix=<suffix> — Query name suffix match

Matches messages whose DNS question name ends with <suffix>. Useful for matching all names under a domain.

--filter "suffix=example.com."

qtype=<type> — DNS query type

Matches messages whose DNS question type equals <type>. Type names are case-insensitive. Applies to both queries and responses.

--filter "qtype=AAAA"
--filter "qtype=MX"
--filter "qtype=TXT"

rcode=<rcode> — DNS response code

Matches DNS response messages with the given response code. Rcode names are case-insensitive.

--filter "rcode=NXDOMAIN"
--filter "rcode=SERVFAIL"
--filter "rcode=NOERROR"

rdata=<value> — Response answer record data

Matches DNS response messages based on the data in the Answer section. The match mode is determined automatically from <value>:

Value formatMatch typeTarget record types
Valid IP address (e.g. 1.1.1.1, 2001:db8::1)Exact IP matchA, AAAA
CIDR notation (e.g. 10.0.0.0/8)Subnet matchA, AAAA
Other stringSubstring matchTXT
# Exact IP match — find responses resolving to a specific address
--filter "rdata=93.184.216.34"
--filter "rdata=2606:2800:220:1:248:1893:25c8:1946"

# Subnet match — find responses resolving to an address in a range
--filter "rdata=10.0.0.0/8"
--filter "rdata=192.168.0.0/16"

# TXT substring match — find SPF, DKIM, or other TXT records
--filter "rdata=v=spf1"
--filter "rdata=v=DKIM1"

rdata.size=<n> — Response answer record count

Matches DNS response messages where the number of records in the Answer section equals <n>. <n> must be a non-negative integer.

# Responses with exactly 1 answer record
--filter "rdata.size=1"

# Responses with no answer records (empty answer section)
--filter "rdata.size=0"

# Responses with exactly 3 answer records (e.g. round-robin A records)
--filter "rdata.size=3"

msgtype=<type> — dnstap message type

Matches messages of a specific dnstap message type. Type names are case-insensitive.

Available types:

TypeDescription
CLIENT_QUERYQuery received from a DNS client
CLIENT_RESPONSEResponse sent to a DNS client
RESOLVER_QUERYQuery sent by the resolver to an upstream server
RESOLVER_RESPONSEResponse received by the resolver from upstream
AUTH_QUERYQuery received by an authoritative server
AUTH_RESPONSEResponse sent by an authoritative server
FORWARDER_QUERYQuery sent by a forwarder
FORWARDER_RESPONSEResponse received by a forwarder
--filter "msgtype=CLIENT_QUERY"
--filter "msgtype=CLIENT_RESPONSE"

Negation (not)

The not operator negates any predicate or grouped expression.

# Exclude AAAA queries
--filter "not qtype=AAAA"

# Internal subnet, excluding NXDOMAIN responses
--filter "subnet=10.0.0.0/8 and not rcode=NXDOMAIN"

# Exclude a group of query types
--filter "not (qtype=A or qtype=AAAA)"

Examples

# Default output: print time, Q/R, name, type and rcode to stdout
dnstap-filter --in file:input.dnstap --filter "ip=1.1.1.1"
# Output: 2024-01-15 12:34:56 Q example.com. A
# Output: 2024-01-15 12:34:56 R example.com. A NOERROR

# Filter file to file (backward-compatible bare paths)
dnstap-filter --in input.dnstap --out output.dnstap --filter "ip=1.1.1.1"

# Explicit file scheme
dnstap-filter --in file:input.dnstap --out file:output.dnstap --filter "ip=1.1.1.1"

# Listen on Unix socket (e.g. from BIND/Unbound) and write to file
dnstap-filter --in unix:/var/run/named/dnstap.sock --out file:output.dnstap --filter "suffix=example.com."

# Listen on TCP and write to file
dnstap-filter --in tcp:0.0.0.0:6000 --out file:output.dnstap --filter "rcode=NXDOMAIN"

# Inspect filtered records as YAML on stdout
dnstap-filter --in file:input.dnstap --out yaml:- --filter "ip=1.1.1.1"

# Write filtered records as YAML to a file
dnstap-filter --in file:input.dnstap --out yaml:/tmp/filtered.yaml --filter "suffix=.evil.com."

# Inspect filtered records as JSONL on stdout
dnstap-filter --in file:input.dnstap --out jsonl:- --filter "ip=1.1.1.1"

# Write filtered records as JSONL to a file
dnstap-filter --in file:input.dnstap --out jsonl:/tmp/filtered.jsonl --filter "suffix=.evil.com."

# JSONL output piped to jq for pretty printing
dnstap-filter --in file:input.dnstap --out jsonl:- --filter "rcode=NXDOMAIN" | jq .

# Forward from Unix socket to a TCP collector
dnstap-filter --in unix:/var/run/named/dnstap.sock --out tcp:collector.example.com:6000 --filter "rcode=NXDOMAIN"

# Filter by client subnet
dnstap-filter --in file:input.dnstap --filter "subnet=192.168.0.0/24"

# Filter AAAA queries only
dnstap-filter --in file:input.dnstap --filter "qtype=AAAA"

# Filter responses containing a specific IP in the answer
dnstap-filter --in file:input.dnstap --filter "rdata=93.184.216.34"

# Filter responses with answers in a subnet
dnstap-filter --in file:input.dnstap --filter "rdata=10.0.0.0/8"

# Filter responses containing specific TXT content
dnstap-filter --in file:input.dnstap --filter "rdata=v=spf1"

# Filter responses with exactly 3 answer records
dnstap-filter --in file:input.dnstap --filter "rdata.size=3"

# Filter responses with no answer records
dnstap-filter --in file:input.dnstap --filter "rdata.size=0"

# Filter only client queries (not responses)
dnstap-filter --in file:input.dnstap --filter "msgtype=CLIENT_QUERY"

# AND condition
dnstap-filter --in file:input.dnstap --filter "subnet=192.168.0.0/24 and suffix=example.com."

# OR condition
dnstap-filter --in file:input.dnstap --filter "rcode=NXDOMAIN or rcode=SERVFAIL"

# Use parentheses to control precedence
dnstap-filter --in file:input.dnstap --filter "subnet=192.168.0.0/24 and (qtype=AAAA or rcode=NXDOMAIN)"

# Complex: client subnet, AAAA queries or NXDOMAINs for a domain
dnstap-filter --in file:input.dnstap \
  --filter "subnet=10.0.0.0/8 and suffix=example.com. and (qtype=AAAA or rcode=NXDOMAIN)"

# Exclude AAAA queries
dnstap-filter --in file:input.dnstap --filter "not qtype=AAAA"

# Internal subnet, excluding NXDOMAIN
dnstap-filter --in file:input.dnstap --filter "subnet=10.0.0.0/8 and not rcode=NXDOMAIN"

# Exclude a group of query types
dnstap-filter --in file:input.dnstap --filter "not (qtype=A or qtype=AAAA)"

# Live capture from a specific network interface
sudo dnstap-filter --in device:en0 --filter "suffix=example.com."

# Live capture from all interfaces
sudo dnstap-filter --in device:all --filter "qtype=AAAA"

# Live capture with YAML output
sudo dnstap-filter --in device:en0 --out yaml:- --filter "rcode=NXDOMAIN"

# Fan-out: monitor on stdout while saving to file
dnstap-filter --in file:input.dnstap --out stdout:time,name,type --out file:output.dnstap --filter "suffix=example.com."

# Fan-out: stdout and YAML to a file simultaneously
dnstap-filter --in unix:/var/run/named/dnstap.sock --out stdout:time,qr,name --out yaml:/tmp/debug.yaml --filter "rcode=NXDOMAIN"

# Fan-out: write to multiple dnstap files
dnstap-filter --in file:input.dnstap --out file:copy1.dnstap --out file:copy2.dnstap

# Print filter tree only
dnstap-filter --filter "ip=1.1.1.1 and (suffix=example.com. or rcode=NXDOMAIN)" --print-filter-tree

# Process only first 100 records
dnstap-filter --in file:input.dnstap --filter "suffix=example.com." --cout 100

# Replay at original timestamp speed (realtime)
dnstap-filter --in file:input.dnstap --speed 1

# Replay at 2x speed
dnstap-filter --in file:input.dnstap --speed 2

# Replay at half speed (slow motion)
dnstap-filter --in file:input.dnstap --speed 0.5

# Replay DNS queries to a target server
dnstap-filter --in file:input.dnstap --out dns:8.8.8.8:53

# Replay at original speed to a local DNS server
dnstap-filter --in file:input.dnstap --out dns:127.0.0.1:5353 --speed 1

# Replay only specific queries with filtering
dnstap-filter --in file:input.dnstap --out dns:192.168.1.1:53 --filter "suffix=example.com. and qtype=A"

# Monitor replayed queries on stdout simultaneously
dnstap-filter --in file:input.dnstap --out stdout:time,name,type --out dns:8.8.8.8:53 --speed 1

# Generate an HTML statistics report (includes time-series charts)
dnstap-filter --in file:input.dnstap --out stats:report.html --filter "suffix=example.com."

# Generate a DSC-compatible XML statistics report
dnstap-filter --in file:input.dnstap --out stats:report.xml

# Generate a Markdown statistics report
dnstap-filter --in file:input.dnstap --out stats:report.md

# Collect stats while monitoring on stdout
dnstap-filter --in file:input.dnstap --out stdout:time,name,type --out stats:stats.json --filter "rcode=NXDOMAIN"

# Aggregate by second-level domain and /24 subnet
dnstap-filter --in file:input.dnstap --out stats:report.html \
  --stats-domain-labels 2 --stats-subnet-prefix 24

Notes

  • The program exits with an error at startup if the filter expression is invalid.
  • and / or / not operators are case-insensitive.
  • yaml: output is human-readable and cannot be used as input to another dnstap-filter instance.
  • jsonl: output produces one compact JSON object per line (JSONL format), suitable for processing with tools like jq.
  • File output (file:) supports SIGHUP for log rotation (closes and reopens the output file).
  • The legacy positional-argument CLI (input output ip) has been removed.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages