Skip to content

HTTP

Jump to bottom Edit New page
Tom Sellers edited this page Sep 3, 2021 · 10 revisions

The data referenced in this document can be found in the HTTP GET Responses and HTTPS GET Responses sections of the Rapid7 Open Data website.

Summary

Project Sonar produces a HTTP/HTTPS data on a regular basis. This data is gathered by first performing a TCP SYN scan across the entire IPv4 address space for certain ports known to host HTTP services. A collection script is then run against every system that returned a positive response. The collection script sends a HTTP GET / HTTP/1.1 request with the Host header set to the IP address of the system.

Data format / schema

The data format is gzip-compressed JSON, with one record per line, and the complete HTTP response, including the HTTP headers stored as a base64-encoded string in the "data" key. The example below grabs the first 10 results from the 2014-12-09 scan.

Note that many servers require a known hostname to be sent in the Host header and return a 400 response if only the IP address is sent, as this scan does. We have experimented with a VHOST-enabled scanner (leveraging the FDNS dataset), but this results in over 1.5Tb of compressed data per scan. The current GET scan results in 64Gb or more data per run.

The current collection script will save up to 150K of data per host.

 curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | \ 
 zcat | head -n 10

{"vhost": "23.41.188.80", "host": "23.41.188.80", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTQNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2JkMDFmZWE1JiM0NjsxNDE4MjA3MzAwJiM0NjsxNzViODJiNAo8L0JPRFk+PC9IVE1MPgo=", "port": 80, "ip": "23.41.188.80"}
{"vhost": "141.101.123.250", "host": "141.101.123.250", "data": "SFRUUC8xLjEgNDAzIEZvcmJpZGRlbg0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMCBHTVQNCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PVVURi04DQpUcmFuc2Zlci1FbmNvZGluZzogY2h1bmtlZA0KQ29ubmVjdGlvbjogY2xvc2UNClNldC1Db29raWU6IF9fY2ZkdWlkPWQzMzYzZDlhZjc4ZjkxNjBjNDg4NWVlZjgzYmQyNTg3ZTE0MTgyMDczMDA7IGV4cGlyZXM9VGh1LCAxMC1EZWMtMTUgMTA6Mjg6MjAgR01UOyBwYXRoPS87IEh0dHBPbmx5DQpDYWNoZS1Db250cm9sOiBtYXgtYWdlPTE1DQpFeHBpcmVzOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjM1IEdNVA0KU2VydmVyOiBjbG91ZGZsYXJlLW5naW54DQpDRi1SQVk6IDE5NjhjMTRlYTgwOTBjM2ItU0VBDQoNCmUwNQ0KPCFET0NUWVBFIGh0bWw+CjwhLS1baWYgbHQgSUUgN10+IDxodG1sIGNsYXNzPSJuby1qcyBpZTYgb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgSUUgN10+ICAgIDxodG1sIGNsYXNzPSJuby1qcyBpZTcgb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgSUUgOF0+ICAgIDxodG1sIGNsYXNzPSJuby1qcyBpZTggb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgZ3QgSUUgOF0+PCEtLT4gPGh0bWwgY2xhc3M9Im5vLWpzIiBsYW5nPSJlbi1VUyI+IDwhLS08IVtlbmRpZl0tLT4KPGhlYWQ+Cjx0aXRsZT5EaXJlY3QgSVAgYWNjZXNzIG5vdCBhbGxvd2VkIHwgMTQxLjEwMS4xMjMuMjUwIHwgQ2xvdWRGbGFyZTwvdGl0bGU+PC90aXRsZT4KPG1ldGEgY2hhcnNldD0iVVRGLTgiIC8+CjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiAvPgo8bWV0YSBodHRwLWVxdWl2PSJYLVVBLUNvbXBhdGlibGUiIGNvbnRlbnQ9IklFPUVkZ2UsY2hyb21lPTEiIC8+CjxtZXRhIG5hbWU9InJvYm90cyIgY29udGVudD0ibm9pbmRleCwgbm9mb2xsb3ciIC8+CjxtZXRhIG5hbWU9InZpZXdwb3J0IiBjb250ZW50PSJ3aWR0aD1kZXZpY2Utd2lkdGgsaW5pdGlhbC1zY2FsZT0xLG1heGltdW0tc2NhbGU9MSIgLz4KPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBpZD0iY2Zfc3R5bGVzLWNzcyIgaHJlZj0iL2Nkbi1jZ2kvc3R5bGVzL2NmLmVycm9ycy5jc3MiIHR5cGU9InRleHQvY3NzIiBtZWRpYT0ic2NyZWVuLHByb2plY3Rpb24iIC8+CjwhLS1baWYgbHQgSUUgOV0+PGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBpZD0nY2Zfc3R5bGVzLWllLWNzcycgaHJlZj0iL2Nkbi1jZ2kvc3R5bGVzL2NmLmVycm9ycy5pZS5jc3MiIHR5cGU9InRleHQvY3NzIiBtZWRpYT0ic2NyZWVuLHByb2plY3Rpb24iIC8+PCFbZW5kaWZdLS0+CjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+Ym9keXttYXJnaW46MDtwYWRkaW5nOjB9PC9zdHlsZT4KPCEtLVtpZiBsdGUgSUUgOV0+PHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy9qcXVlcnkubWluLmpzIj48L3NjcmlwdD48IVtlbmRpZl0tLT4KPCEtLVtpZiBndGUgSUUgMTBdPjwhLS0+PHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy96ZXB0by5taW4uanMiPjwvc2NyaXB0PjwhLS08IVtlbmRpZl0tLT4KPHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy9jZi5jb21tb24uanMiPjwvc2NyaXB0PgoKPC9oZWFkPgo8Ym9keT4KICA8ZGl2IGlkPSJjZi13cmFwcGVyIj4KICAgIDxkaXYgY2xhc3M9ImNmLWFsZXJ0IGNmLWFsZXJ0LWVycm9yIGNmLWNvb2tpZS1lcnJvciIgaWQ9ImNvb2tpZS1hbGVydCIgZGF0YS10cmFuc2xhdGU9ImVuYWJsZV9jb29raWVzIj5QbGVhc2UgZW5hYmxlIGNvb2tpZXMuPC9kaXY+CiAgICA8ZGl2IGlkPSJjZi1lcnJvci1kZXRhaWxzIiBjbGFzcz0iY2YtZXJyb3ItZGV0YWlscy13cmFwcGVyIj4KICAgICAgPGRpdiBjbGFzcz0iY2Ytd3JhcHBlciBjZi1oZWFkZXIgY2YtZXJyb3Itb3ZlcnZpZXciPgogICAgICAgIDxoMT4KICAgICAgICAgIDxzcGFuIGNsYXNzPSJjZi1lcnJvci10eXBlIiBkYXRhLXRyYW5zbGF0ZT0iZXJyb3IiPkVycm9yPC9zcGFuPgogICAgICAgICAgPHNwYW4gY2xhc3M9ImNmLWVycm9yLWNvZGUiPjEwMDM8L3NwYW4+CiAgICAgICAgICA8c21hbGwgY2xhc3M9ImhlYWRpbmctcmF5LWlkIj5SYXkgSUQ6IDE5NjhjMTRlYTgwOTBjM2I8L3NtYWxsPgogICAgICAgIDwvaDE+CiAgICAgICAgPGgyIGNsYXNzPSJjZi1zdWJoZWFkbGluZSIgZGF0YS10cmFuc2xhdGU9ImVycm9yX2Rlc2MiPkRpcmVjdCBJUCBhY2Nlc3Mgbm90IGFsbG93ZWQ8L2gyPgogICAgICA8L2Rpdj48IS0tIC8uaGVhZGVyIC0tPgoKICAgICAgPHNlY3Rpb24+PC9zZWN0aW9uPjwhLS0gc3BhY2VyIC0tPgoKICAgICAgPGRpdiBjbGFzcz0iY2Ytc2VjdGlvbiBjZi13cmFwcGVyIj4KICAgICAgICA8ZGl2IGNsYXNzPSJjZi1jb2x1bW5zIHR3byI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJjZi1jb2x1bW4iPgogICAgICAgICAgICA8aDIgZGF0YS10cmFuc2xhdGU9IndoYXRfaGFwcGVuZWQiPldoYXQgaGFwcGVuZWQ/PC9oMj4KICAgICAgICAgICAgPHA+WW91J3ZlIHJlcXVlc3RlZCBhbiBJUCBhZGRyZXNzIHRoYXQgaXMgcGFydCBvZiB0aGUgPGEgZGF0YS1vcmlnLXByb3RvPSJodHRwcyIgZGF0YS1vcmlnLXJlZj0id3d3LmNsb3VkZmxhcmUuY29tIiB0YXJnZXQ9Il9ibGFuayI+Q2xvdWRGbGFyZTwvYT4gbmV0d29yay48L3A+CiAgICAgICAgICA8L2Rpdj4KCiAgICAgICAgICAKICAgICAgICAgIDxkaXYgY2xhc3M9ImNmLWNvbHVtbiI+CiAgICAgICAgICAgIDxoMiBkYXRhLXRyYW5zbGF0ZT0id2hhdF9jYW5faV9kbyI+V2hhdCBjYW4gSSBkbz88L2gyPgogICAgICAgICAgICA8cD5JZiB5b3UgYXJlIGludGVyZXN0ZWQgaW4gbGVhcm5pbmcgbW9yZSBhYm91dCBDbG91ZEZsYXJlLCBwbGVhc2UgPGEgZGF0YS1vcmlnLXByb3RvPSJodHRwcyIgZGF0YS1vcmlnLXJlZj0id3d3LmNsb3VkZmxhcmUuY29tIiB0YXJnZXQ9Il9ibGFuayI+dmlzaXQgb3VyIHdlYnNpdGU8L2E+LjwvcD4KICAgICAgICAgIDwvZGl2PgogICAgICAgICAgCiAgICAgICAgPC9kaXY+CiAgICAgIDwvZGl2PjwhLS0gLy5zZWN0aW9uIC0tPgoKICAgICAgPGRpdiBjbGFzcz0iY2YtZXJyb3ItZm9vdGVyIGNmLXdyYXBwZXIiPgogIDxwPgogICAgPHNwYW4gY2xhc3M9ImNmLWZvb3Rlci1pdGVtIj5DbG91ZEZsYXJlIFJheSBJRDogPHN0cm9uZz4xOTY4YzE0ZWE4MDkwYzNiPC9zdHJvbmc+PC9zcGFuPgogICAgPHNwYW4gY2xhc3M9ImNmLWZvb3Rlci1zZXBhcmF0b3IiPiZidWxsOzwvc3Bhbj4KICAgIDxzcGFuIGNsYXNzPSJjZi1mb290ZXItaXRlbSI+PHNwYW4gZGF0YS10cmFuc2xhdGU9InlvdXJfaXAiPllvdXIgSVA8L3NwYW4+OiA1NC42OC45MS43NDwvc3Bhbj4KICAgIDxzcGFuIGNsYXNzPSJjZi1mb290ZXItc2VwYXJhdG9yIj4mYnVsbDs8L3NwYW4+CiAgICA8c3BhbiBjbGFzcz0iY2YtZm9vdGVyLWl0ZW0iPjxzcGFuIGRhdGEtdHJhbnNsYXRlPSJwZXJmb3JtYW5jZV9zZWN1cml0eV9ieSI+UGVyZm9ybWFuY2UgJmFtcDsgc2VjdXJpdHkgYnk8L3NwYW4+IDxhIGRhdGEtb3JpZy1wcm90bz0iaHR0cHMiIGRhdGEtb3JpZy1yZWY9Ind3dy5jbG91ZGZsYXJlLmNvbS81eHgtZXJyb3ItbGFuZGluZyIgaWQ9ImJyYW5kX2xpbmsiIHRhcmdldD0iX2JsYW5rIj5DbG91ZEZsYXJlPC9hPjwvc3Bhbj4KICAgIAogIDwvcD4KPC9kaXY+PCEtLSAvLmVycm9yLWZvb3RlciAtLT4KCgogICAgPC9kaXY+PCEtLSAvI2NmLWVycm9yLWRldGFpbHMgLS0+CiAgPC9kaXY+PCEtLSAvI2NmLXdyYXBwZXIgLS0+CgogIDxzY3JpcHQgdHlwZT0idGV4dC9qYXZhc2NyaXB0Ij4KICB3aW5kb3cuX2NmX3RyYW5zbGF0aW9uID0ge307CiAgCiAgCjwvc2NyaXB0PgoKPC9ib2R5Pgo8L2h0bWw+Cg0KMA0KDQo=", "port": 80, "ip": "141.101.123.250"}
{"vhost": "23.193.132.247", "host": "23.193.132.247", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTQNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2MwMmMwZTZiJiM0NjsxNDE4MjA3MzAwJiM0NjsxY2JmMjMwZQo8L0JPRFk+PC9IVE1MPgo=", "port": 80, "ip": "23.193.132.247"}
{"vhost": "205.164.8.208", "host": "205.164.8.208", "data": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZA0KQ29udGVudC10eXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAwDQpDb25uZWN0aW9uOiBjbG9zZQ0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMCBHTVQNClNlcnZlcjogVU5PDQoNCg==", "port": 80, "ip": "205.164.8.208"}
{"vhost": "23.5.208.123", "host": "23.5.208.123", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2OzIyZGNmMTgwJiM0NjsxNDE4MjA3MzAwJiM0NjszNGM0YzYwCjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.5.208.123"}
{"vhost": "172.252.77.210", "host": "172.252.77.210", "data": "SFRUUC8xLjEgNDAzIEZvcmJpZGRlbg0KU2VydmVyOiBuZ2lueA0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyNjowMCBHTVQNCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sDQpDb250ZW50LUxlbmd0aDogNTY0DQpDb25uZWN0aW9uOiBjbG9zZQ0KDQo8aHRtbD4NCjxoZWFkPjx0aXRsZT40MDMgRm9yYmlkZGVuPC90aXRsZT48L2hlYWQ+DQo8Ym9keSBiZ2NvbG9yPSJ3aGl0ZSI+DQo8Y2VudGVyPjxoMT40MDMgRm9yYmlkZGVuPC9oMT48L2NlbnRlcj4NCjxocj48Y2VudGVyPm5naW54PC9jZW50ZXI+DQo8L2JvZHk+DQo8L2h0bWw+DQo8IS0tIGEgcGFkZGluZyB0byBkaXNhYmxlIE1TSUUgYW5kIENocm9tZSBmcmllbmRseSBlcnJvciBwYWdlIC0tPg0KPCEtLSBhIHBhZGRpbmcgdG8gZGlzYWJsZSBNU0lFIGFuZCBDaHJvbWUgZnJpZW5kbHkgZXJyb3IgcGFnZSAtLT4NCjwhLS0gYSBwYWRkaW5nIHRvIGRpc2FibGUgTVNJRSBhbmQgQ2hyb21lIGZyaWVuZGx5IGVycm9yIHBhZ2UgLS0+DQo8IS0tIGEgcGFkZGluZyB0byBkaXNhYmxlIE1TSUUgYW5kIENocm9tZSBmcmllbmRseSBlcnJvciBwYWdlIC0tPg0KPCEtLSBhIHBhZGRpbmcgdG8gZGlzYWJsZSBNU0lFIGFuZCBDaHJvbWUgZnJpZW5kbHkgZXJyb3IgcGFnZSAtLT4NCjwhLS0gYSBwYWRkaW5nIHRvIGRpc2FibGUgTVNJRSBhbmQgQ2hyb21lIGZyaWVuZGx5IGVycm9yIHBhZ2UgLS0+DQo=", "port": 80, "ip": "172.252.77.210"}
{"vhost": "198.15.186.4", "host": "198.15.186.4", "data": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9dXMtYXNjaWkNClNlcnZlcjogTWljcm9zb2Z0LUhUVFBBUEkvMi4wDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDAyOjI0OjI0IEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCkNvbnRlbnQtTGVuZ3RoOiAzMTUNCg0KPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiJodHRwOi8vd3d3LnczLm9yZy9UUi9odG1sNC9zdHJpY3QuZHRkIj4NCjxIVE1MPjxIRUFEPjxUSVRMRT5Ob3QgRm91bmQ8L1RJVExFPg0KPE1FVEEgSFRUUC1FUVVJVj0iQ29udGVudC1UeXBlIiBDb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXMtYXNjaWkiPjwvSEVBRD4NCjxCT0RZPjxoMj5Ob3QgRm91bmQ8L2gyPg0KPGhyPjxwPkhUVFAgRXJyb3IgNDA0LiBUaGUgcmVxdWVzdGVkIHJlc291cmNlIGlzIG5vdCBmb3VuZC48L3A+DQo8L0JPRFk+PC9IVE1MPg0K", "port": 80, "ip": "198.15.186.4"}
{"vhost": "104.148.103.201", "host": "104.148.103.201", "data": "SFRUUC8xLjEgNDAwIEJhZCBSZXF1ZXN0DQpDb250ZW50LVR5cGU6IHRleHQvaHRtbA0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMSBHTVQNCkNvbm5lY3Rpb246IGNsb3NlDQpDb250ZW50LUxlbmd0aDogMzkNCg0KPGgxPkJhZCBSZXF1ZXN0IChJbnZhbGlkIEhvc3RuYW1lKTwvaDE+", "port": 80, "ip": "104.148.103.201"}
{"vhost": "23.73.210.133", "host": "23.73.210.133", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2OzMxNWRmNmQ4JiM0NjsxNDE4MjA3MzAwJiM0Njs2YWY5NzE5CjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.73.210.133"}
{"vhost": "23.6.1.2", "host": "23.6.1.2", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2M1ODgwMmNjJiM0NjsxNDE4MjA3MzAwJiM0NjsxMmM5ZGEwCjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.6.1.2"}

A sample request is shown below. Each line is terminated by a CRLF and an extra CRLF is appended to the request.

GET / HTTP/1.1
Host: ${IP_ADDRESS}
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)
Accept-Encoding: none
Connection: close

The DAP utility can be used to process the JSON data as a stream:

$ curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | \
  zcat | head -n 1 | \
  bin/dap json + transform data=base64decode + decode_http_reply data + json

{
  "data.http_title": "Invalid URL",
  "data.http_body": "<HTML><HEAD>\n<TITLE>Invalid URL</TITLE>\n</HEAD><BODY>\n<H1>Invalid URL</H1>\nThe requested URL \"&#47;\", is invalid.<p>\nReference&#32;&#35;9&#46;bd01fea5&#46;1418207300&#46;175b82b4\n</BODY></HTML>\n",
  "data.http_date": "20141210T04:28:20",
  "data.http_server": "AkamaiGHost",
  "vhost": "23.41.188.80",
  "host": "23.41.188.80",
  "data": "HTTP/1.0 400 Bad Request\r\nServer: AkamaiGHost\r\nMime-Version: 1.0\r\nContent-Type: text/html\r\nContent-Length: 194\r\nExpires: Wed, 10 Dec 2014 10:28:20 GMT\r\nDate: Wed, 10 Dec 2014 10:28:20 GMT\r\nConnection: close\r\n\r\n<HTML><HEAD>\n<TITLE>Invalid URL</TITLE>\n</HEAD><BODY>\n<H1>Invalid URL</H1>\nThe requested URL \"&#47;\", is invalid.<p>\nReference&#32;&#35;9&#46;bd01fea5&#46;1418207300&#46;175b82b4\n</BODY></HTML>\n",
  "port": 80,
  "ip": "23.41.188.80",
  "data.http_code": 400,
  "data.http_message": "Bad Request",
  "data.http_raw_headers": {
    "connection": "close",
    "expires": "Wed, 10 Dec 2014 10:28:20 GMT",
    "content-type": "text/html",
    "mime-version": "1.0"
  }
}
Add a custom footer

Introduction

Dataset Information

Help

Clone this wiki locally