Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Shopobot Fathead #14

Closed
wants to merge 3 commits into from

4 participants

Hank Stoever Robert Picard Gabriel Weinberg Zaahir Moolla
Hank Stoever

This is a pull request for our fathead to work with Shopobot.com. We are a consumer product site that tracks thousands of products across many different e-commerce sites, and we crawl their daily price information. This fathead is a corpus of all of our products with external links to where the product is listed (amazon, newegg, etc) as well as an image and snippet containing a short description and the 30-day high and low price.

Let us know any feedback or improvements we can make at hanks@shopobot.com or davem@shopobot.com

Robert Picard

Update: Okay, I installed rvm and ruby 1.9.3 and it's working now.

@hstove I'm getting this error when I try to run the parser:

parse.rb:1:in `require': no such file to load -- json (LoadError)
        from parse.rb:1

I take it that this is a problem with the json gem but could you shed some light on what the specific issue might be?

Gabriel Weinberg
Owner

Talked off-github about integrating it in a different way.

Gabriel Weinberg yegg closed this
Zaahir Moolla
Owner

@hstove Hey there, when you get a chance, would you mind sending an email to moollaza@duckduckgo.com please? Or if you don't mind, can I have your email address? (We have some goodies for our contributors!) Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 18, 2012
  1. Hank Stoever

    initial fathead setup

    hstove authored
  2. Hank Stoever

    removed download folder

    hstove authored
Commits on Apr 25, 2012
  1. Hank Stoever

    touches to parser and fetcher

    hstove authored
This page is out of date. Refresh to see the latest.
6 shopobot/README.txt
View
@@ -0,0 +1,6 @@
+Dependencies
+------------
+wget 1.13.4
+ruby 1.9.2 (with json gem)
+
+gem install json
2  shopobot/fetch.sh
View
@@ -0,0 +1,2 @@
+mkdir -p download
+wget -P download 'https://s3.amazonaws.com/DDG/ddgdump.json'
15 shopobot/meta.txt
View
@@ -0,0 +1,15 @@
+# This is the name of the source as people would refer to it, e.g. Wikipedia or PerlDoc
+Name: Shopobot
+
+# This is the base domain where the source pages are located.
+Domain: shopobot.com
+
+# This is what gets put in quotes next to the source
+# It can be blank if it is a source with completely general info spanning many types of topics like Facebook.
+Type: Shopping
+
+# Whether the source is from MediaWiki (1) or not (0).
+MediaWiki: 0
+
+# Keywords uses to trigger (or prefer) the source over others.
+Keywords: shop
36 shopobot/parse.rb
View
@@ -0,0 +1,36 @@
+require 'json'
+
+DOWNLOAD_DIR='download'
+
+def open(file)
+ file = File.new(file, "r")
+ json = file.readlines.to_s
+ all = JSON.parse(json)
+ prodList = JSON.parse(all[0])
+ puts prodList[0]
+ prodList.each do |product|
+ title = product['title']
+ if product['offers']
+ external_links = ''
+ product['offers'].each do |offer|
+ offer.each do |key, value|
+ if value != ""
+ external_links += "[#{value} #{key}]\\n"
+ end
+ end
+ end
+ end
+ image = product['image']
+ url = product['url']
+ description = "30 Day Price History: $#{product['high']}-$#{product['low']}. "
+ if product['description']
+ description += product['description']
+ end
+ puts "#{title}\tA\t\t\t\t\t\t\t#{external_links}\t\t#{image}\t#{description}\t#{url}"
+ end
+end
+
+
+Dir.new(DOWNLOAD_DIR).entries.each do |file|
+ open(DOWNLOAD_DIR+'/'+file) if file=~/\.json/
+end
3  shopobot/parse.sh
View
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+ruby parse.rb > output.txt
Something went wrong with that request. Please try again.