It'd be good to do a robots.txt endpoint to steer robots away from places they shouldn't go.
What happened here?, the commit 82d2647 is not in the repository. (it mentions something about a CLA not signed in #533 )
I think this pull request was closed by accident with lots of PR in June 2014.
Could anybody with permissions look at it?
@mrvdb I cannot find the mention to the CLA not signed, can you point us at it?