rhodgman-r7 edited this page Sep 12, 2016 · 7 revisions

Project Sonar produces a HTTP dataset every week. This data is gathered by first performing a TCP SYN scan across the entire IPv4 address space for port 80. A collection script is then run against every system that returned a positive response. The collection script sends a HTTP GET / HTTP/1.1 request with the Host header set to the IP address of the system.

The data format is gzip-compressed JSON, with one record per line, and the complete HTTP response, including the HTTP headers stored as a base64-encoded string in the "data" key. The example below grabs the first 10 results from the 2014-12-09 scan.

Note that many servers require a known hostname to be sent in the Host header and return a 400 response if only the IP address is sent, as this scan does. We have experimented with a VHOST-enabled scanner (leveraging the FDNS dataset), but this results in over 1.5Tb of compressed data per scan. The current GET scan results in 64Gb or more data per run.

The current collection script will save up to 150K of data per host.

 curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | \ 
 zcat | head -n 10

{"vhost": "23.41.188.80", "host": "23.41.188.80", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTQNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2JkMDFmZWE1JiM0NjsxNDE4MjA3MzAwJiM0NjsxNzViODJiNAo8L0JPRFk+PC9IVE1MPgo=", "port": 80, "ip": "23.41.188.80"}
{"vhost": "141.101.123.250", "host": "141.101.123.250", "data": "SFRUUC8xLjEgNDAzIEZvcmJpZGRlbg0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMCBHTVQNCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sOyBjaGFyc2V0PVVURi04DQpUcmFuc2Zlci1FbmNvZGluZzogY2h1bmtlZA0KQ29ubmVjdGlvbjogY2xvc2UNClNldC1Db29raWU6IF9fY2ZkdWlkPWQzMzYzZDlhZjc4ZjkxNjBjNDg4NWVlZjgzYmQyNTg3ZTE0MTgyMDczMDA7IGV4cGlyZXM9VGh1LCAxMC1EZWMtMTUgMTA6Mjg6MjAgR01UOyBwYXRoPS87IEh0dHBPbmx5DQpDYWNoZS1Db250cm9sOiBtYXgtYWdlPTE1DQpFeHBpcmVzOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjM1IEdNVA0KU2VydmVyOiBjbG91ZGZsYXJlLW5naW54DQpDRi1SQVk6IDE5NjhjMTRlYTgwOTBjM2ItU0VBDQoNCmUwNQ0KPCFET0NUWVBFIGh0bWw+CjwhLS1baWYgbHQgSUUgN10+IDxodG1sIGNsYXNzPSJuby1qcyBpZTYgb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgSUUgN10+ICAgIDxodG1sIGNsYXNzPSJuby1qcyBpZTcgb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgSUUgOF0+ICAgIDxodG1sIGNsYXNzPSJuby1qcyBpZTggb2xkaWUiIGxhbmc9ImVuLVVTIj4gPCFbZW5kaWZdLS0+CjwhLS1baWYgZ3QgSUUgOF0+PCEtLT4gPGh0bWwgY2xhc3M9Im5vLWpzIiBsYW5nPSJlbi1VUyI+IDwhLS08IVtlbmRpZl0tLT4KPGhlYWQ+Cjx0aXRsZT5EaXJlY3QgSVAgYWNjZXNzIG5vdCBhbGxvd2VkIHwgMTQxLjEwMS4xMjMuMjUwIHwgQ2xvdWRGbGFyZTwvdGl0bGU+PC90aXRsZT4KPG1ldGEgY2hhcnNldD0iVVRGLTgiIC8+CjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiAvPgo8bWV0YSBodHRwLWVxdWl2PSJYLVVBLUNvbXBhdGlibGUiIGNvbnRlbnQ9IklFPUVkZ2UsY2hyb21lPTEiIC8+CjxtZXRhIG5hbWU9InJvYm90cyIgY29udGVudD0ibm9pbmRleCwgbm9mb2xsb3ciIC8+CjxtZXRhIG5hbWU9InZpZXdwb3J0IiBjb250ZW50PSJ3aWR0aD1kZXZpY2Utd2lkdGgsaW5pdGlhbC1zY2FsZT0xLG1heGltdW0tc2NhbGU9MSIgLz4KPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBpZD0iY2Zfc3R5bGVzLWNzcyIgaHJlZj0iL2Nkbi1jZ2kvc3R5bGVzL2NmLmVycm9ycy5jc3MiIHR5cGU9InRleHQvY3NzIiBtZWRpYT0ic2NyZWVuLHByb2plY3Rpb24iIC8+CjwhLS1baWYgbHQgSUUgOV0+PGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBpZD0nY2Zfc3R5bGVzLWllLWNzcycgaHJlZj0iL2Nkbi1jZ2kvc3R5bGVzL2NmLmVycm9ycy5pZS5jc3MiIHR5cGU9InRleHQvY3NzIiBtZWRpYT0ic2NyZWVuLHByb2plY3Rpb24iIC8+PCFbZW5kaWZdLS0+CjxzdHlsZSB0eXBlPSJ0ZXh0L2NzcyI+Ym9keXttYXJnaW46MDtwYWRkaW5nOjB9PC9zdHlsZT4KPCEtLVtpZiBsdGUgSUUgOV0+PHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy9qcXVlcnkubWluLmpzIj48L3NjcmlwdD48IVtlbmRpZl0tLT4KPCEtLVtpZiBndGUgSUUgMTBdPjwhLS0+PHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy96ZXB0by5taW4uanMiPjwvc2NyaXB0PjwhLS08IVtlbmRpZl0tLT4KPHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcHQiIHNyYz0iL2Nkbi1jZ2kvc2NyaXB0cy9jZi5jb21tb24uanMiPjwvc2NyaXB0PgoKPC9oZWFkPgo8Ym9keT4KICA8ZGl2IGlkPSJjZi13cmFwcGVyIj4KICAgIDxkaXYgY2xhc3M9ImNmLWFsZXJ0IGNmLWFsZXJ0LWVycm9yIGNmLWNvb2tpZS1lcnJvciIgaWQ9ImNvb2tpZS1hbGVydCIgZGF0YS10cmFuc2xhdGU9ImVuYWJsZV9jb29raWVzIj5QbGVhc2UgZW5hYmxlIGNvb2tpZXMuPC9kaXY+CiAgICA8ZGl2IGlkPSJjZi1lcnJvci1kZXRhaWxzIiBjbGFzcz0iY2YtZXJyb3ItZGV0YWlscy13cmFwcGVyIj4KICAgICAgPGRpdiBjbGFzcz0iY2Ytd3JhcHBlciBjZi1oZWFkZXIgY2YtZXJyb3Itb3ZlcnZpZXciPgogICAgICAgIDxoMT4KICAgICAgICAgIDxzcGFuIGNsYXNzPSJjZi1lcnJvci10eXBlIiBkYXRhLXRyYW5zbGF0ZT0iZXJyb3IiPkVycm9yPC9zcGFuPgogICAgICAgICAgPHNwYW4gY2xhc3M9ImNmLWVycm9yLWNvZGUiPjEwMDM8L3NwYW4+CiAgICAgICAgICA8c21hbGwgY2xhc3M9ImhlYWRpbmctcmF5LWlkIj5SYXkgSUQ6IDE5NjhjMTRlYTgwOTBjM2I8L3NtYWxsPgogICAgICAgIDwvaDE+CiAgICAgICAgPGgyIGNsYXNzPSJjZi1zdWJoZWFkbGluZSIgZGF0YS10cmFuc2xhdGU9ImVycm9yX2Rlc2MiPkRpcmVjdCBJUCBhY2Nlc3Mgbm90IGFsbG93ZWQ8L2gyPgogICAgICA8L2Rpdj48IS0tIC8uaGVhZGVyIC0tPgoKICAgICAgPHNlY3Rpb24+PC9zZWN0aW9uPjwhLS0gc3BhY2VyIC0tPgoKICAgICAgPGRpdiBjbGFzcz0iY2Ytc2VjdGlvbiBjZi13cmFwcGVyIj4KICAgICAgICA8ZGl2IGNsYXNzPSJjZi1jb2x1bW5zIHR3byI+CiAgICAgICAgICA8ZGl2IGNsYXNzPSJjZi1jb2x1bW4iPgogICAgICAgICAgICA8aDIgZGF0YS10cmFuc2xhdGU9IndoYXRfaGFwcGVuZWQiPldoYXQgaGFwcGVuZWQ/PC9oMj4KICAgICAgICAgICAgPHA+WW91J3ZlIHJlcXVlc3RlZCBhbiBJUCBhZGRyZXNzIHRoYXQgaXMgcGFydCBvZiB0aGUgPGEgZGF0YS1vcmlnLXByb3RvPSJodHRwcyIgZGF0YS1vcmlnLXJlZj0id3d3LmNsb3VkZmxhcmUuY29tIiB0YXJnZXQ9Il9ibGFuayI+Q2xvdWRGbGFyZTwvYT4gbmV0d29yay48L3A+CiAgICAgICAgICA8L2Rpdj4KCiAgICAgICAgICAKICAgICAgICAgIDxkaXYgY2xhc3M9ImNmLWNvbHVtbiI+CiAgICAgICAgICAgIDxoMiBkYXRhLXRyYW5zbGF0ZT0id2hhdF9jYW5faV9kbyI+V2hhdCBjYW4gSSBkbz88L2gyPgogICAgICAgICAgICA8cD5JZiB5b3UgYXJlIGludGVyZXN0ZWQgaW4gbGVhcm5pbmcgbW9yZSBhYm91dCBDbG91ZEZsYXJlLCBwbGVhc2UgPGEgZGF0YS1vcmlnLXByb3RvPSJodHRwcyIgZGF0YS1vcmlnLXJlZj0id3d3LmNsb3VkZmxhcmUuY29tIiB0YXJnZXQ9Il9ibGFuayI+dmlzaXQgb3VyIHdlYnNpdGU8L2E+LjwvcD4KICAgICAgICAgIDwvZGl2PgogICAgICAgICAgCiAgICAgICAgPC9kaXY+CiAgICAgIDwvZGl2PjwhLS0gLy5zZWN0aW9uIC0tPgoKICAgICAgPGRpdiBjbGFzcz0iY2YtZXJyb3ItZm9vdGVyIGNmLXdyYXBwZXIiPgogIDxwPgogICAgPHNwYW4gY2xhc3M9ImNmLWZvb3Rlci1pdGVtIj5DbG91ZEZsYXJlIFJheSBJRDogPHN0cm9uZz4xOTY4YzE0ZWE4MDkwYzNiPC9zdHJvbmc+PC9zcGFuPgogICAgPHNwYW4gY2xhc3M9ImNmLWZvb3Rlci1zZXBhcmF0b3IiPiZidWxsOzwvc3Bhbj4KICAgIDxzcGFuIGNsYXNzPSJjZi1mb290ZXItaXRlbSI+PHNwYW4gZGF0YS10cmFuc2xhdGU9InlvdXJfaXAiPllvdXIgSVA8L3NwYW4+OiA1NC42OC45MS43NDwvc3Bhbj4KICAgIDxzcGFuIGNsYXNzPSJjZi1mb290ZXItc2VwYXJhdG9yIj4mYnVsbDs8L3NwYW4+CiAgICA8c3BhbiBjbGFzcz0iY2YtZm9vdGVyLWl0ZW0iPjxzcGFuIGRhdGEtdHJhbnNsYXRlPSJwZXJmb3JtYW5jZV9zZWN1cml0eV9ieSI+UGVyZm9ybWFuY2UgJmFtcDsgc2VjdXJpdHkgYnk8L3NwYW4+IDxhIGRhdGEtb3JpZy1wcm90bz0iaHR0cHMiIGRhdGEtb3JpZy1yZWY9Ind3dy5jbG91ZGZsYXJlLmNvbS81eHgtZXJyb3ItbGFuZGluZyIgaWQ9ImJyYW5kX2xpbmsiIHRhcmdldD0iX2JsYW5rIj5DbG91ZEZsYXJlPC9hPjwvc3Bhbj4KICAgIAogIDwvcD4KPC9kaXY+PCEtLSAvLmVycm9yLWZvb3RlciAtLT4KCgogICAgPC9kaXY+PCEtLSAvI2NmLWVycm9yLWRldGFpbHMgLS0+CiAgPC9kaXY+PCEtLSAvI2NmLXdyYXBwZXIgLS0+CgogIDxzY3JpcHQgdHlwZT0idGV4dC9qYXZhc2NyaXB0Ij4KICB3aW5kb3cuX2NmX3RyYW5zbGF0aW9uID0ge307CiAgCiAgCjwvc2NyaXB0PgoKPC9ib2R5Pgo8L2h0bWw+Cg0KMA0KDQo=", "port": 80, "ip": "141.101.123.250"}
{"vhost": "23.193.132.247", "host": "23.193.132.247", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTQNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2MwMmMwZTZiJiM0NjsxNDE4MjA3MzAwJiM0NjsxY2JmMjMwZQo8L0JPRFk+PC9IVE1MPgo=", "port": 80, "ip": "23.193.132.247"}
{"vhost": "205.164.8.208", "host": "205.164.8.208", "data": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZA0KQ29udGVudC10eXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAwDQpDb25uZWN0aW9uOiBjbG9zZQ0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMCBHTVQNClNlcnZlcjogVU5PDQoNCg==", "port": 80, "ip": "205.164.8.208"}
{"vhost": "23.5.208.123", "host": "23.5.208.123", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2OzIyZGNmMTgwJiM0NjsxNDE4MjA3MzAwJiM0NjszNGM0YzYwCjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.5.208.123"}
{"vhost": "172.252.77.210", "host": "172.252.77.210", "data": "SFRUUC8xLjEgNDAzIEZvcmJpZGRlbg0KU2VydmVyOiBuZ2lueA0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyNjowMCBHTVQNCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sDQpDb250ZW50LUxlbmd0aDogNTY0DQpDb25uZWN0aW9uOiBjbG9zZQ0KDQo8aHRtbD4NCjxoZWFkPjx0aXRsZT40MDMgRm9yYmlkZGVuPC90aXRsZT48L2hlYWQ+DQo8Ym9keSBiZ2NvbG9yPSJ3aGl0ZSI+DQo8Y2VudGVyPjxoMT40MDMgRm9yYmlkZGVuPC9oMT48L2NlbnRlcj4NCjxocj48Y2VudGVyPm5naW54PC9jZW50ZXI+DQo8L2JvZHk+DQo8L2h0bWw+DQo8IS0tIGEgcGFkZGluZyB0byBkaXNhYmxlIE1TSUUgYW5kIENocm9tZSBmcmllbmRseSBlcnJvciBwYWdlIC0tPg0KPCEtLSBhIHBhZGRpbmcgdG8gZGlzYWJsZSBNU0lFIGFuZCBDaHJvbWUgZnJpZW5kbHkgZXJyb3IgcGFnZSAtLT4NCjwhLS0gYSBwYWRkaW5nIHRvIGRpc2FibGUgTVNJRSBhbmQgQ2hyb21lIGZyaWVuZGx5IGVycm9yIHBhZ2UgLS0+DQo8IS0tIGEgcGFkZGluZyB0byBkaXNhYmxlIE1TSUUgYW5kIENocm9tZSBmcmllbmRseSBlcnJvciBwYWdlIC0tPg0KPCEtLSBhIHBhZGRpbmcgdG8gZGlzYWJsZSBNU0lFIGFuZCBDaHJvbWUgZnJpZW5kbHkgZXJyb3IgcGFnZSAtLT4NCjwhLS0gYSBwYWRkaW5nIHRvIGRpc2FibGUgTVNJRSBhbmQgQ2hyb21lIGZyaWVuZGx5IGVycm9yIHBhZ2UgLS0+DQo=", "port": 80, "ip": "172.252.77.210"}
{"vhost": "198.15.186.4", "host": "198.15.186.4", "data": "SFRUUC8xLjEgNDA0IE5vdCBGb3VuZA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7IGNoYXJzZXQ9dXMtYXNjaWkNClNlcnZlcjogTWljcm9zb2Z0LUhUVFBBUEkvMi4wDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDAyOjI0OjI0IEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCkNvbnRlbnQtTGVuZ3RoOiAzMTUNCg0KPCFET0NUWVBFIEhUTUwgUFVCTElDICItLy9XM0MvL0RURCBIVE1MIDQuMDEvL0VOIiJodHRwOi8vd3d3LnczLm9yZy9UUi9odG1sNC9zdHJpY3QuZHRkIj4NCjxIVE1MPjxIRUFEPjxUSVRMRT5Ob3QgRm91bmQ8L1RJVExFPg0KPE1FVEEgSFRUUC1FUVVJVj0iQ29udGVudC1UeXBlIiBDb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXMtYXNjaWkiPjwvSEVBRD4NCjxCT0RZPjxoMj5Ob3QgRm91bmQ8L2gyPg0KPGhyPjxwPkhUVFAgRXJyb3IgNDA0LiBUaGUgcmVxdWVzdGVkIHJlc291cmNlIGlzIG5vdCBmb3VuZC48L3A+DQo8L0JPRFk+PC9IVE1MPg0K", "port": 80, "ip": "198.15.186.4"}
{"vhost": "104.148.103.201", "host": "104.148.103.201", "data": "SFRUUC8xLjEgNDAwIEJhZCBSZXF1ZXN0DQpDb250ZW50LVR5cGU6IHRleHQvaHRtbA0KRGF0ZTogV2VkLCAxMCBEZWMgMjAxNCAxMDoyODoyMSBHTVQNCkNvbm5lY3Rpb246IGNsb3NlDQpDb250ZW50LUxlbmd0aDogMzkNCg0KPGgxPkJhZCBSZXF1ZXN0IChJbnZhbGlkIEhvc3RuYW1lKTwvaDE+", "port": 80, "ip": "104.148.103.201"}
{"vhost": "23.73.210.133", "host": "23.73.210.133", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2OzMxNWRmNmQ4JiM0NjsxNDE4MjA3MzAwJiM0Njs2YWY5NzE5CjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.73.210.133"}
{"vhost": "23.6.1.2", "host": "23.6.1.2", "data": "SFRUUC8xLjAgNDAwIEJhZCBSZXF1ZXN0DQpTZXJ2ZXI6IEFrYW1haUdIb3N0DQpNaW1lLVZlcnNpb246IDEuMA0KQ29udGVudC1UeXBlOiB0ZXh0L2h0bWwNCkNvbnRlbnQtTGVuZ3RoOiAxOTMNCkV4cGlyZXM6IFdlZCwgMTAgRGVjIDIwMTQgMTA6Mjg6MjAgR01UDQpEYXRlOiBXZWQsIDEwIERlYyAyMDE0IDEwOjI4OjIwIEdNVA0KQ29ubmVjdGlvbjogY2xvc2UNCg0KPEhUTUw+PEhFQUQ+CjxUSVRMRT5JbnZhbGlkIFVSTDwvVElUTEU+CjwvSEVBRD48Qk9EWT4KPEgxPkludmFsaWQgVVJMPC9IMT4KVGhlIHJlcXVlc3RlZCBVUkwgIiYjNDc7IiwgaXMgaW52YWxpZC48cD4KUmVmZXJlbmNlJiMzMjsmIzM1OzkmIzQ2O2M1ODgwMmNjJiM0NjsxNDE4MjA3MzAwJiM0NjsxMmM5ZGEwCjwvQk9EWT48L0hUTUw+Cg==", "port": 80, "ip": "23.6.1.2"}

A sample request is shown below. Each line is terminated by a CRLF and an extra CRLF is appended to the request.

GET / HTTP/1.1
Host: ${IP_ADDRESS}
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2)
Accept-Encoding: none
Connection: close

The DAP utility can be used to process the JSON data as a stream:

$ curl -s https://scans.io/data/rapid7/sonar.http/20141209-http.gz | \
  zcat | head -n 1 | \
  bin/dap json + transform data=base64decode + decode_http_reply data + json
{
  "data.http_title": "Invalid URL",
  "data.http_body": "<HTML><HEAD>\n<TITLE>Invalid URL</TITLE>\n</HEAD><BODY>\n<H1>Invalid URL</H1>\nThe requested URL \"&#47;\", is invalid.<p>\nReference&#32;&#35;9&#46;bd01fea5&#46;1418207300&#46;175b82b4\n</BODY></HTML>\n",
  "data.http_date": "20141210T04:28:20",
  "data.http_server": "AkamaiGHost",
  "vhost": "23.41.188.80",
  "host": "23.41.188.80",
  "data": "HTTP/1.0 400 Bad Request\r\nServer: AkamaiGHost\r\nMime-Version: 1.0\r\nContent-Type: text/html\r\nContent-Length: 194\r\nExpires: Wed, 10 Dec 2014 10:28:20 GMT\r\nDate: Wed, 10 Dec 2014 10:28:20 GMT\r\nConnection: close\r\n\r\n<HTML><HEAD>\n<TITLE>Invalid URL</TITLE>\n</HEAD><BODY>\n<H1>Invalid URL</H1>\nThe requested URL \"&#47;\", is invalid.<p>\nReference&#32;&#35;9&#46;bd01fea5&#46;1418207300&#46;175b82b4\n</BODY></HTML>\n",
  "port": 80,
  "ip": "23.41.188.80",
  "data.http_code": 400,
  "data.http_message": "Bad Request",
  "data.http_raw_headers": {
    "connection": "close",
    "expires": "Wed, 10 Dec 2014 10:28:20 GMT",
    "content-type": "text/html",
    "mime-version": "1.0"
  }
}
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.