This is a project I started working on and abandoned in 2009. It is a spider which was specifically built for crawling websites which contained EVE Kill Mail's. In the game of EVE, every time you kill a player, an in-game 'mail' is sent to you containing information. Players would copy this informaiton and paste it into a website, which would show statistics about the kill.
The purpose of this project was to take information about all of the kills and display them on an interactive 2D heatmap. There would also be a slider which represented time, and a user could slide this slider. The purpose of the project was to show which areas in space were the most dangerous.
Unfortunately, I stopped playing EVE and abandoned the project. The crawler, which worked in 2009, is likely no longer useful. It would download pages and run regular expressions to find data, as most 'kill mail' websites used the same software. But, I'm sure the software has changed by now.
cache.txt ID of the last crawled page ccp_map_data.zip Solar system coordinate data; released by CCP crawler.php Main crawler downloader.php Page downloading class eve-map-bg.jpg Background graphic for map eve-map.xml.php PHP script for generating XML data for flash evecrawler.sql.7z 189MB database file of a bunch of data map.fla Raw map building flash file map.swf Compiled map flash file mysql.ssi.php Configuration file for database settings rendered.xml Example of the rendered XML data sample-crawl.htm An example of the kill mail pages circa 2009
I honestly don't remember what the state of the project is. I do know that the crawler works just fine; I was able to grab a couple million records. The flash map stuff is probably all broken and whatnot. Back in 2009 we didn't have any fancy canvas rendering, a better developer would replace the flash with canvas.
The crawler wasn't as efficient as it could be. Most of the slowness would be due to network latency, so paralell downloads should be used.
None of the interface development for clicking regions, doing heat maps, or scrolling through time was ever implemented.
Released under the BSD License.