Forward DNS

Jon Hart edited this page Apr 21, 2017 · 4 revisions

Project Sonar produces a Forward DNS dataset every week or so. This data is created by extracting domain names from a number of sources and then sending an ANY query for each domain. The sources used to build the list of domains include:

  • Reverse DNS (PTR) Records
  • Common Name and SubjectAltName fields from SSL Certificates
  • HTML elements and Location headers seen in HTTP responses
  • Zone files from COM, INFO, ORG, NET, BIZ, INFO and other TLDs
  • Zone files from gTLDs

The data format is a gzip-compressed JSON file, where each line of the file is a JSON document with attributes for the record name, type, value and time of resolution.

An example below using curl, pigz and jq to fetch, decompress and pretty print the first 10 records from a study in April 2017:

$  curl --silent  https://scans.io/data/rapid7/sonar.fdns_v2/20170417-fdns.json.gz | pigz -dc | head -n 10 | jq 
{
  "timestamp": "1492468299",
  "name": "reseauocoz.cluster007.ovh.net",
  "type": "cname",
  "value": "cluster007.ovh.net"
}
{
  "timestamp": "1492435280",
  "name": "isutility.web9.hubspot.com",
  "type": "cname",
  "value": "a1049.b.akamai.net"
}
{
  "timestamp": "1492435280",
  "name": "isutility.web9.hubspot.com",
  "type": "cname",
  "value": "webhost9.hubspot.net"
}
{
  "timestamp": "1492436557",
  "name": "sendv54sxu8f12g.ihance.net",
  "type": "a",
  "value": "52.52.156.202"
}
{
  "timestamp": "1492436557",
  "name": "sendv54sxu8f12g.ihance.net",
  "type": "a",
  "value": "54.215.210.155"
}
{
  "timestamp": "1492466485",
  "name": "shops.myshopify.com",
  "type": "cname",
  "value": "shops.myshopify.com"
}
{
  "timestamp": "1492467470",
  "name": "www.triblocal.com.s3-website-us-east-1.amazonaws.com",
  "type": "cname",
  "value": "s3-website-us-east-1.amazonaws.com"
}
{
  "timestamp": "1492442430",
  "name": "*.2925.com.dycdn.com",
  "type": "a",
  "value": "121.201.116.57"
}
{
  "timestamp": "1492401188",
  "name": "*.2bask.com",
  "type": "a",
  "value": "176.31.246.156"
}
{
  "timestamp": "1492418685",
  "name": "*.51kdyun.com",
  "type": "a",
  "value": "50.117.89.173"
}
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.