added latest distribution crawler and result
forgot to set page back to 0
added latest ntriples file
latest crawl from 2006-11-26
send a better user agent to let data.gov.uk know who is requesting th…
…e pages. also do not stop when encountering http error.
bit of doc
moved over from bzr http://inkdroid.org/bzr/data-gov-uk