Forward DNS

The data referenced in this document can be found in the Forward DNS section of the Rapid7 Open Data website.

Summary

Project Sonar produces a Forward DNS dataset every week or so. This data is created by extracting domain names from a number of sources and then sending an ANY query for each domain. The sources used to build the list of domains include:

• Reverse DNS (PTR) Records
• Common Name and SubjectAltName fields from SSL Certificates
• HTML elements and Location headers seen in HTTP responses
• Zone files from COM, INFO, ORG, NET, BIZ, INFO and other TLDs
• Zone files from gTLDs

Data format / schema

The dataset schema document can be found on the Open Data website here

The data format is a gzip-compressed JSON file, where each line of the file is a JSON document with attributes for the record name, type, value and time of resolution.

An example below using curl, pigz and jq to fetch, decompress and pretty print the first 10 records from a study in April 2017:

$  curl --silent  https://scans.io/data/rapid7/sonar.fdns_v2/20170417-fdns.json.gz | pigz -dc | head -n 10 | jq 
{
  "timestamp": "1492468299",
  "name": "reseauocoz.cluster007.ovh.net",
  "type": "cname",
  "value": "cluster007.ovh.net"
}
{
  "timestamp": "1492435280",
  "name": "isutility.web9.hubspot.com",
  "type": "cname",
  "value": "a1049.b.akamai.net"
}
{
  "timestamp": "1492435280",
  "name": "isutility.web9.hubspot.com",
  "type": "cname",
  "value": "webhost9.hubspot.net"
}
{
  "timestamp": "1492436557",
  "name": "sendv54sxu8f12g.ihance.net",
  "type": "a",
  "value": "52.52.156.202"
}
{
  "timestamp": "1492436557",
  "name": "sendv54sxu8f12g.ihance.net",
  "type": "a",
  "value": "54.215.210.155"
}
{
  "timestamp": "1492466485",
  "name": "shops.myshopify.com",
  "type": "cname",
  "value": "shops.myshopify.com"
}
{
  "timestamp": "1492467470",
  "name": "www.triblocal.com.s3-website-us-east-1.amazonaws.com",
  "type": "cname",
  "value": "s3-website-us-east-1.amazonaws.com"
}
{
  "timestamp": "1492442430",
  "name": "*.2925.com.dycdn.com",
  "type": "a",
  "value": "121.201.116.57"
}
{
  "timestamp": "1492401188",
  "name": "*.2bask.com",
  "type": "a",
  "value": "176.31.246.156"
}
{
  "timestamp": "1492418685",
  "name": "*.51kdyun.com",
  "type": "a",
  "value": "50.117.89.173"
}