-
Notifications
You must be signed in to change notification settings - Fork 75
Using Mercury
David McGrew edited this page Jul 22, 2020
·
4 revisions
These examples were created with mercury version 2.2.0. They illustrate the different ways that the program can be used, and how to invoke it in those use cases. The ellipsis (...) indicates that the additional output is not shown. The file top-https.pcap used in the examples is in the subdirectory test/data/. Some of the output lines are quite long; these can be viewed in their entirety by scrolling to the right. Some examples use the jq program to pretty-print the JSON output.
- Analyzing packet capture files
- Analyzing packet capture files, for selected protocols
- Analyzing packet capture files, for selected protocols, with more metadata
- Analyzing packet capture files with detailed DNS JSON output
- Analyzing packet capture files with detailed certificates JSON output
- Analyzing packet capture files with HTTP metadata
- Capturing traffic and reporting fingerprints and metadata as JSON to standard output
- Capturing traffic, reporting JSON to a file, using multiple threads
Read the PCAP file top-https.pcap and write JSON to standard output. The option -r (or --read) indicates the file to be read.
$ mercury -r top-https.pcap
{"dns":{"base64":"90CBgAABAAEADQANBmdvb2dsZQNjb20AABwAAcAMABwAAQAAAAUAECYH+LBABAgQAAAAAAAAIA7AEwACAAEAAAAFABQBYwxndGxkLXNlcnZlcnMDbmV0AMATAAIAAQAAAAUABAFpwEbAEwACAAEAAAAFAAQBZsBGwBMAAgABAAAABQAEAWHARsATAAIAAQAAAAUABAFswEbAEwACAAEAAAAFAAQBa8BGwBMAAgABAAAABQAEAW3ARsATAAIAAQAAAAUABAFlwEbAEwACAAEAAAAFAAQBYsBGwBMAAgABAAAABQAEAWjARsATAAIAAQAAAAUABAFnwEbAEwACAAEAAAAFAAQBasBGwBMAAgABAAAABQAEAWTARsCEAAEAAQAAAAUABMAFBh7A1AABAAEAAAAFAATAIQ4ewEQAAQABAAAABQAEwBpcHsEUAAEAAQAAAAUABMAfUB7AxAABAAEAAAAFAATADF4ewHQAAQABAAAABQAEwCMzHsD0AAEAAQAAAAUABMAqXR7A5AABAAEAAAAFAATANnAewGQAAQABAAAABQAEwCusHsEEAAEAAQAAAAUABMAwTx7ApAABAAEAAAAFAATANLIewJQAAQABAAAABQAEwCmiHsC0AAEAAQAAAAUABMA3Ux4="},"src_ip":"192.168.113.2","dst_ip":"192.168.113.237","protocol":17,"src_port":53,"dst_port":37225,"event_start":1565200314.223537}
{"dns":{"base64":"2PuBgAABAAEADQAOBmdvb2dsZQNjb20AAAEAAcAMAAEAAQAAAAUABKzZD07AEwACAAEAAAAFABQBaQxndGxkLXNlcnZlcnMDbmV0AMATAAIAAQAAAAUABAFhwDrAEwACAAEAAAAFAAQBbcA6wBMAAgABAAAABQAEAWvAOsATAAIAAQAAAAUABAFqwDrAEwACAAEAAAAFAAQBY8A6wBMAAgABAAAABQAEAWLAOsATAAIAAQAAAAUABAFnwDrAEwACAAEAAAAFAAQBbMA6wBMAAgABAAAABQAEAWXAOsATAAIAAQAAAAUABAFowDrAEwACAAEAAAAFAAQBZMA6wBMAAgABAAAABQAEAWbAOsBYAAEAAQAAAAUABMAFBh7AqAABAAEAAAAFAATAIQ4ewJgAAQABAAAABQAEwBpcHsD4AAEAAQAAAAUABMAfUB7A2AABAAEAAAAFAATADF4ewQgAAQABAAAABQAEwCMzHsC4AAEAAQAAAAUABMAqXR7A6AABAAEAAAAFAATANnAewDgAAQABAAAABQAEwCusHsCIAAEAAQAAAAUABMAwTx7AeAABAAEAAAAFAATANLIewMgAAQABAAAABQAEwCmiHsBoAAEAAQAAAAUABMA3Ux7AWAAcAAEAAAAFABAgAQUDqD4AAAAAAAAAAgAw"},"src_ip":"192.168.113.2","dst_ip":"192.168.113.237","protocol":17,"src_port":53,"dst_port":40385,"event_start":1565200314.223559}
{"fingerprints":{"tcp":"(faf0)(020405b4)(04)(08)(01)(030307)"},"src_ip":"192.168.113.237","dst_ip":"172.217.15.78","protocol":6,"src_port":38790,"dst_port":443,"event_start":1565200314.224204}
{"fingerprints":{"tls":"(0303)(130213031301c02cc030009fcca9cca8ccaac02bc02f009ec024c028006bc023c0270067c00ac0140039c009c0130033009d009c003d003c0035002f00ff)((0000)(000b000403000102)(000a000c000a001d0017001e00190018)(0023)(0016)(0017)(000d0030002e040305030603080708080809080a080b080408050806040105010601030302030301020103020202040205020602)(002b0009080304030303020301)(002d00020101)(0033))"},"tls":{"client":{"server_name":"google.com"}},"src_ip":"192.168.113.237","dst_ip":"172.217.15.78","protocol":6,"src_port":38790,"dst_port":443,"event_start":1565200314.266206}
{"fingerprints":{"tls_server":"(0303)(1302)((0033)(002b00020304))"},"src_ip":"172.217.15.78","dst_ip":"192.168.113.237","protocol":6,"src_port":443,"dst_port":38790,"event_start":1565200314.317713}
...As above, but selecting the TLS protocol, so that only data for that protocol is reported. The -s (or --select) option indicates the protocol (or protocols) selected.
$ mercury -r top-https.pcap --select=tls
{"fingerprints":{"tls":"(0303)(130213031301c02cc030009fcca9cca8ccaac02bc02f009ec024c028006bc023c0270067c00ac0140039c009c0130033009d009c003d003c0035002f00ff)((0000)(000b000403000102)(000a000c000a001d0017001e00190018)(0023)(0016)(0017)(000d0030002e040305030603080708080809080a080b080408050806040105010601030302030301020103020202040205020602)(002b0009080304030303020301)(002d00020101)(0033))"},"tls":{"client":{"server_name":"google.com"}},"src_ip":"192.168.113.237","dst_ip":"172.217.15.78","protocol":6,"src_port":38790,"dst_port":443,"event_start":1565200314.266206}
{"fingerprints":{"tls_server":"(0303)(1302)((0033)(002b00020304))"},"src_ip":"172.217.15.78","dst_ip":"192.168.113.237","protocol":6,"src_port":443,"dst_port":38790,"event_start":1565200314.317713}
{"fingerprints":{"tls":"(0303)(130213031301c02cc030009fcca9cca8ccaac02bc02f009ec024c028006bc023c0270067c00ac0140039c009c0130033009d009c003d003c0035002f00ff)((0000)(000b000403000102)(000a000c000a001d0017001e00190018)(0023)(0016)(0017)(000d0030002e040305030603080708080809080a080b080408050806040105010601030302030301020103020202040205020602)(002b0009080304030303020301)(002d00020101)(0033))"},"tls":{"client":{"server_name":"www.google.com"}},"src_ip":"192.168.113.237","dst_ip":"172.217.7.228","protocol":6,"src_port":55912,"dst_port":443,"event_start":1565200314.496049}
{"fingerprints":{"tls_server":"(0303)(1302)((0033)(002b00020304))"},"src_ip":"172.217.7.228","dst_ip":"192.168.113.237","protocol":6,"src_port":443,"dst_port":55912,"event_start":1565200314.548679}
...To select multiple protocols, use a comma-separated list. In this example, --select=tcp,tls selects TCP and TLS. For the TCP protocol, fingerprints constructed from TCP SYN packets are reported.
$ mercury -r ../test/data/top-https.pcap --select=tcp,tls
{"fingerprints":{"tcp":"(faf0)(020405b4)(04)(08)(01)(030307)"},"src_ip":"192.168.113.237","dst_ip":"172.217.15.78","protocol":6,"src_port":38790,"dst_port":443,"event_start":1565200314.224204}
{"fingerprints":{"tls":"(0303)(130213031301c02cc030009fcca9cca8ccaac02bc02f009ec024c028006bc023c0270067c00ac0140039c009c0130033009d009c003d003c0035002f00ff)((0000)(000b000403000102)(000a000c000a001d0017001e00190018)(0023)(0016)(0017)(000d0030002e040305030603080708080809080a080b080408050806040105010601030302030301020103020202040205020602)(002b0009080304030303020301)(002d00020101)(0033))"},"tls":{"client":{"server_name":"google.com"}},"src_ip":"192.168.113.237","dst_ip":"172.217.15.78","protocol":6,"src_port":38790,"dst_port":443,"event_start":1565200314.266206}
{"fingerprints":{"tls_server":"(0303)(1302)((0033)(002b00020304))"},"src_ip":"172.217.15.78","dst_ip":"192.168.113.237","protocol":6,"src_port":443,"dst_port":38790,"event_start":1565200314.317713}As above, but reporting more data about each TLS session. The --metadata option causes more data features to be written into JSON. This example uses jq, just to format the output for clarity's sake.
$ mercury -r top-https.pcap --select=tls --metadata | jq .
{
"fingerprints": {
"tls": "(0303)(130213031301c02cc030009fcca9cca8ccaac02bc02f009ec024c028006bc023c0270067c00ac0140039c009c0130033009d009c003d003c0035002f00ff)((0000)(000b000403000102)(000a000c000a001d0017001e00190018)(0023)(0016)(0017)(000d0030002e040305030603080708080809080a080b080408050806040105010601030302030301020103020202040205020602)(002b0009080304030303020301)(002d00020101)(0033))"
},
"tls": {
"client": {
"version": "0303",
"random": "b3afd945f15e47a81a8c87eb4a8f47b04cd087a2e4b67f3bbefae8ab4c472f0c",
"session_id": "e5d2be5888a873a7753789438a91eafe22564432b617a5ebcadc7f026d37238f",
"cipher_suites": "130213031301c02cc030009fcca9cca8ccaac02bc02f009ec024c028006bc023c0270067c00ac0140039c009c0130033009d009c003d003c0035002f00ff",
"compression_methods": "00",
"server_name": "google.com"
}
},
"src_ip": "192.168.113.237",
"dst_ip": "172.217.15.78",
"protocol": 6,
"src_port": 38790,
"dst_port": 443,
"event_start": 1565200314.266206
}
...To report DNS responses as detailed JSON objects, instead of base64 strings, use the --dns-json option. In this example we also use the --select=dns option to focus the output on that protocol, though that option operates independently of--dns-json. This example uses jq, just to format the output for clarity's sake.
$ mercury -r top-https.pcap --select=dns --dns-json | jq .
{
"dns": {
"response": {
"qname": "google.com",
"opcode": 0,
"rcode": 0,
"flags": 33152,
"answer": [
{
"rrtype": "AAAA",
"rdata": "2607:f8b0:4004:810::200e",
"ttl": 5
}
],
"authority": [
{
"rrtype": "NS",
"rrname": "c.gtld-servers.net",
"ttl": 5
},
{
"rrtype": "NS",
"rrname": "i.gtld-servers.net",
"ttl": 5
},
{
"rrtype": "NS",
"rrname": "f.gtld-servers.net",
"ttl": 5
},
{
"rrtype": "NS",
"rrname": "a.gtld-servers.net",
"ttl": 5
},
{
"rrtype": "NS",
"rrname": "l.gtld-servers.net",
"ttl": 5
},
...To report PKIX certificates as detailed JSON objects, instead of base64 strings, use the --certs-json option. In this example we also use the --select=tls option to focus the output on that protocol, though that option operates independently of --certs-json. The output is filtered through jq to select the tls.server.certs field, which is a JSON array of certificate objects. Certificate objects have a complex structure, with many optional fields, as shown in this example.
$ mercury -r top-https.pcap --select=tls --certs-json | jq .tls.server.certs
[
{
"cert": {
"version": "02",
"serial_number": "07a27140cfd9fc949427514f8555426f",
"signature_identifier": {
"algorithm": "sha256WithRSAEncryption"
},
"issuer": [
{
"country_name": "US"
},
{
"organization_name": "DigiCert Inc"
},
{
"organizational_unit_name": "www.digicert.com"
},
{
"common_name": "DigiCert SHA2 High Assurance Server CA"
}
],
"validity": [
{
"not_before": "2019-03-07 00:00:00Z"
},
{
"not_after": "2020-03-07 12:00:00Z"
}
],
"subject": [
{
"country_name": "US"
},
{
"state_or_province_name": "California"
},
{
"locality_name": "San Francisco"
},
{
"organization_name": "Twitter, Inc."
},
{
"organizational_unit_name": "atla"
},
{
"common_name": "twitter.com"
}
],
"subject_public_key_info": {
"algorithm_identifier": {
"algorithm": "rsaEncryption"
},
"subject_public_key": {
"modulus": "00e657da4725b5fadc3d3c9f00016d200813b9e880a9e53f93a337380aeb39344918bb8b0acbe3ddaf8ed98e1cc41bcfca1b0081b33e9cb957b5fd33887e520e32732ceea654ae93ef5c593a323ccf4d475646f0a8e9c55463c3f365f2817e16d686a33ade1dd70329399a1ce81fcb87ecbb402154bccfb174c0f4f39272ad666f686c37a1042ae036eb0c16a85826d2cdd6dbb91935c6981cb4ddb1779ac5fe7e4c838524181c9347f3447c1f65b958a8f9b6d3a38b4f88a45bc0eda7ce818658c692f13f9412d4e97a5dd85cfa54b0fd9f91c3c5ce986de9e62b3a2eea86d6ae816f297acde3c8f871c69f77b6f347d8eafb49a060e9c33a9848888cdd84cfcb",
"exponent": "010001",
"bits_in_modulus": 2048,
"bits_in_exponent": 17
}
},
"extensions": [
{
"authority_key_identifier": {
"key_identifier": "5168ff90af0207753cccd9656462a212b859723b"
},
"critical": false
},
{
"subject_key_identifier": "84368f7f4b6945e738c1688210ed156c9e910c69",
"critical": false
},
{
"subject_alt_name": [
{
"dns_name": "twitter.com"
},
{
"dns_name": "www.twitter.com"
}
],
"critical": false
},
{
"key_usage": [
"digital_signature",
"key_encipherment"
],
"critical": true
},
{
"ext_key_usage": [
"id-kp-serverAuth",
"id-kp-clientAuth"
],
"critical": false
},
{
"crl_distribution_points": [
{
"crl_distribution_point": [
{
"distribution_point_name": {
"full_name": {
"uri": "http://crl3.digicert.com/sha2-ha-server-g6.crl"
}
}
}
]
},
{
"crl_distribution_point": [
{
"distribution_point_name": {
"full_name": {
"uri": "http://crl4.digicert.com/sha2-ha-server-g6.crl"
}
}
}
]
}
],
"critical": false
},
{
"certificate_policies": [
{
"policy_information": [
{
"policy_identifier": "2.16.840.1.114412.1.1",
"policy_qualifier_info": {
"qualifier_id": "id-qt-cps",
"qualifier": "https://www.digicert.com/CPS"
}
}
]
},
{
"policy_information": [
{
"policy_identifier": "2.23.140.1.2.2"
}
]
}
],
"critical": false
},
{
"authority_info_access": [
{
"access_method": "id-ad-ocsp",
"access_location": {
"uri": "http://ocsp.digicert.com"
}
},
{
"access_method": "id-ad-caIssuers",
"access_location": {
"uri": "http://cacerts.digicert.com/DigiCertSHA2HighAssuranceServerCA.crt"
}
}
],
"critical": false
},
{
"basic_constraints": {
"ca": false,
"path_len_constraint": 0
},
"critical": true
},
{
"signed_certificate_timestamp_list": "00f1007700bbd9dfbc1f8a71b593942397aa927b473857950aab52e81a909664368e1ed18500000169595561be0000040300483046022100993add9d305ff8dfa08415eec10bf7d8ae8cb6e057d690446a1da2f67d23a64a022100a9ab633de84b7d78fb932ddedbe502d299b543000a8348914750f16ceaa62afc0076008775bfe7597cf88c43995fbdf36eff568d475636ff4ab560c1b4eaff5ea0830f000001695955630300000403004730450220106d3044cfaa12a4e729073d499d75b88561911a4576a70b2c33d21390003cd502210093c11fa60bca4f2274bd27b408a0f92b77f483ac211c8a028b12fb0a8ec32382",
"critical": false
}
],
"violations": [
"invalid"
]
}
}
]
...To report detailed metadata for HTTP, use the --metadata option. In this example we also use the --select=http option to focus the output on that protocol, though that option operates independently of --metadata. This example pipes the output through jq, just to format the output for clarity's sake.
$ mercury -r top-https.pcap --select=http --metadata | jq .
{
"fingerprints": {
"http": "(474554)(485454502f312e31)(486f7374)(4163636570743a202a2f2a)(436f6e6e656374696f6e3a20636c6f7365)"
},
"complete": "yes",
"http": {
"request": {
"method": "GET",
"uri": "/",
"protocol": "HTTP/1.1",
"host": "connectivity-check.ubuntu.com"
}
},
"src_ip": "192.168.113.237",
"dst_ip": "35.224.99.156",
"protocol": 6,
"src_port": 53560,
"dst_port": 80,
"event_start": 1565200503.658237
}
{
"fingerprints": {
"http_server": "(485454502f312e31)(323034)(4e6f20436f6e74656e74)(44617465)(5365727665723a204170616368652f322e342e313820285562756e747529)(436f6e6e656374696f6e3a20636c6f7365)"
},
"complete": "yes",
"src_ip": "35.224.99.156",
"dst_ip": "192.168.113.237",
"protocol": 6,
"src_port": 80,
"dst_port": 53560,
"event_start": 1565200503.759359
}Read network traffic from the interface and write JSON to standard output. The option -c (or --capture) indicates the network interface to be used, which in this case is wlp0s20f3. In this example, the additional options --select=dns --dns-json are used, which cause mercury to only output DNS data, and to report it as a detailed JSON object. The non-JSON output are notices reported to the standard error (stderr). Note that to capture traffic from a network interface, mercury must be run as root. To reduce security exposure, it drops root privileges once it has acquired the system resources that it needs. This example shows the DNS queries that result from loading the webpage https://aaai.org.
$ sudo mercury --capture wlp0s20f3 --select=dns --dns-json
mem: 333750720 frac: 0.010000
Notice: requested memory 331350016 will be less than desired memory 333750720
Requesting PACKET_RX_RING with 331350016 bytes (79 blocks of size 4194304) for thread 0
dropped root privileges
Thread 0 with thread id 139922282174208 started...
{"dns":{"response":{"qname":"aaai.org","opcode":0,"rcode":0,"flags":33152,"answer":[{"rrtype":"A","rdata":"144.208.67.177","ttl":348}],"authority":[],"additional":[{"type":41,"class":1280,"rdlength":0,"ttl":0}]}},"src_ip":"161.44.124.122","dst_ip":"10.117.145.148","protocol":17,"src_port":53,"dst_port":49003,"event_start":1595441617.877828}
{"dns":{"response":{"qname":"aaai.org","opcode":0,"rcode":0,"flags":33152,"answer":[],"authority":[{"rrtype":"SOA","rrname":"ns1.hover.com","ttl":900}],"additional":[{"type":41,"class":1280,"rdlength":0,"ttl":0}]}},"src_ip":"161.44.124.122","dst_ip":"10.117.145.148","protocol":17,"src_port":53,"dst_port":49145,"event_start":1595441617.881148}
{"dns":{"response":{"qname":"www.aaai.org","opcode":0,"rcode":0,"flags":33152,"answer":[{"rrtype":"CNAME","rrname":"aaai.org","ttl":795},{"rrtype":"A","rdata":"144.208.67.177","ttl":346}],"authority":[],"additional":[{"type":41,"class":1280,"rdlength":0,"ttl":0}]}},"src_ip":"161.44.124.122","dst_ip":"10.117.145.148","protocol":17,"src_port":53,"dst_port":50898,"event_start":1595441619.352738}
{"dns":{"response":{"qname":"www.aaai.org","opcode":0,"rcode":0,"flags":33152,"answer":[{"rrtype":"CNAME","rrname":"aaai.org","ttl":588}],"authority":[{"rrtype":"SOA","rrname":"ns1.hover.com","ttl":898}],"additional":[{"type":41,"class":1280,"rdlength":0,"ttl":0}]}},"src_ip":"161.44.124.122","dst_ip":"10.117.145.148","protocol":17,"src_port":53,"dst_port":34164,"event_start":1595441619.355369}
{"dns":{"response":{"qname":"aaai.org","opcode":0,"rcode":0,"flags":33152,"answer":[],"authority":[{"rrtype":"SOA","rrname":"ns1.hover.com","ttl":898}],"additional":[{"type":41,"class":1280,"rdlength":0,"ttl":0}]}},"src_ip":"161.44.124.122","dst_ip":"10.117.145.148","protocol":17,"src_port":53,"dst_port":59718,"event_start":1595441619.437380}
^C
Gracefully shutting down: Interrupt
Thread 0 with thread id 139922282174208 exiting...
--
343 packets captured
181799 bytes captured
343 packets seen by socket
0 packets dropped
0 socket queue freezesRead network traffic from the interface and write JSON to a file, using as many worker threads as there are cores (processors) available, to ensure that packet processing can keep up with a high throughput network. The option -c (or --capture) indicates the network interface to be used, which in this case is wlp0s20f3. The option -t (or --threads) is set to "cpu", which automatically detects the number of cores; it can alternatively be set to a positive number. Note that to capture traffic from a network interface, mercury must be run as root. To reduce security exposure, it drops root privileges once it has acquired the system resources that it needs.
The output are notices reported to the standard error (stderr); note that each worker thread has its own ring buffer that is shared with the kernel.
$ sudo ./mercury --capture wlp0s20f3 -f output.json -t cpu
mem: 333750720 frac: 0.010000
Notice: requested memory 333447168 will be less than desired memory 333750720
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 0
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 1
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 2
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 3
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 4
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 5
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 6
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 7
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 8
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 9
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 10
Requesting PACKET_RX_RING with 27787264 bytes (106 blocks of size 262144) for thread 11
dropped root privileges
Thread 10 with thread id 140707167328000 started...
Thread 2 with thread id 140707234469632 started...
Thread 9 with thread id 140707175720704 started...
Thread 6 with thread id 140707200898816 started...
Thread 4 with thread id 140707217684224 started...
Thread 8 with thread id 140707184113408 started...
Thread 5 with thread id 140707209291520 started...
Thread 7 with thread id 140707192506112 started...
Thread 0 with thread id 140707251255040 started...
Thread 11 with thread id 140707158935296 started...
Thread 3 with thread id 140707226076928 started...
Thread 1 with thread id 140707242862336 started...
^C