Request to omit all statements on csarven.ca

[csarven]

Hi. I'm the current owner of csarven.ca. I would appreciate it if the dataset, map, and anything else, can omit all statements pertaining to csarven.ca. If there is any information that's currently on csarven.ca that the crawler is discovering, please let me know, I can remove those. Same goes for any other place that I might have access to. Thanks.

[snarfed]

oh wow, hey @csarven. apologies, i totally missed this. absolutely, will do.

[snarfed]

done! your site is now gone from the dataset. details.

fwiw, here's what i see in http://csarven.ca/robots.txt right now. i do get that you may want to allow some crawlers but not others, like indie map.

User-agent: *
Disallow: /archives
Disallow: /scripts
Disallow: /url
Disallow: /labs
Disallow: /presentations
Disallow: /webstream
Disallow: /search
Disallow: /statistical-linked-dataspaces-and-analysis-overview
Allow: /labs/indexability
Allow: /archives/articles

[csarven]

Thanks!

Pardon me if I'm looking at the wrong code, but perhaps it is also worthwhile to update the user-agent value: https://github.com/snarfed/indie-map/blob/master/crawl/wget.sh#L9 ?

[snarfed]

interesting idea. you mean, set it to Indie Map? i could! technically the user agent still is wget though, right? Indie Map is just the use case? maybe I'm splitting hairs.

[csarven]

I would classify indie-map (the software of this repository) as the user-agent, as opposed to a particular library that's doing the fetching. Just as Firefox uses its own library to negotiate resources. But yes, generally it can be arbitrary.

So, you can do something like User-Agent: indie-map or if you're feeling adventurous use User-Agent: https://github.com/snarfed/indie-map/ or some other HTTP URI that can provide a structured description for the application eg view-source https://dokie.li/

[snarfed]

also fwiw, i've only actually done this whole crawl once, and i have no plans to do it again right now, regularly or otherwise.

still, good idea. thanks for the nudge.