scraperwiki comparison

[necrodome]

Hi Andrew,
Where do you see this project in comparison to ScraperWiki? ScraperWiki is also open-source, but it also allow developers to write their agent equivalents in multiple programming languages and also share them between users. I am interested to hear your thoughts if they overlap or serving different purposes ( given that ScraperWiki is in 4 years of development, it might be good to hear your thoughts about your project's philosophy)

[cantino]

Hi @necrodome, while I haven't used ScraperWiki extensively, I think that it's a pretty different beast. (It's also quite cool, especially the online editing of Ruby and Python code.) Huginn is a tool for connecting agents with events and for triggering events when certain things happen. ScraperWiki would be a great integration and a useful source of scrapers and ideas for Agents, but Huginn's focus is broader than just scraping data. Huginn is more about responding to- and acting upon- data. Does that make sense?

[necrodome]

I see, so the main differentiator is the "if this then that" part, right? (scraperwiki is generic enough to gather whatever data you want and manipulate as they allow a sandboxed programming environment. They also let you setup cron functionality similar to huginn.)

[cantino]

It's certainly possible that they have a lot in common. As I said, I haven't used ScraperWiki very much. Is it open source and self-hostable?

It would definitely be interesting to make a ScraperWikiAgent that interfaced with ScraperWiki scrapers.

[necrodome]

Yes, scraperwiki is open-source and self-hostable as well. You can take a glimpse at http://blog.scraperwiki.com/2013/03/19/from-future-import-x-scraperwiki-com/ for their new generation platform for some ideas. I really like how you can share agents (scrapers in their terms), might be a good idea for huginn as well.