Skip to content


Folders and files

Last commit message
Last commit date

Latest commit



14 Commits

Repository files navigation


A crawler to collect reviews and product infomation on and save them to SQLite databases.

Quick Start Guide

Get reviews from one product

To get all reviews for a product, first get the Amazon Standard Identification Number (ASIN) of the product. It is the 10-character alphanumeric ID followed by /product/ in the url. Then add the following code to the main function

	Item samsungTab3 = new Item("B00D02AGU4");
	samsungTab3.writeReviewsToDatabase("reviewtest.db", false);

Data in the created SQlite database can be managed or exported using tools such as SQLiteStudio or RSQLite.

Get reviewer information

	GetReviewerInfo reviewer_crawler = new GetReviewerInfo("reviewer_ids.txt","reviewer_test.db");

where reviewer_ids.txt is a file whose content lists all reviewer ids, and reviewer_test.db is a SQlite db.

Get reviews from a product category

To get all reviews from a product category, find out the node ID for the category from's url (&node=). The node ID will be the first arugument in the GetASINbyNode() constructor. Then you should estimate about how many products there are and divide that number by 5, which will be the third argument in GetASINbyNode(). For example:

	GetASINbyNode getIDs = new GetASINbyNode("541966%2C1232597011", 1,	10);
	ItemList thelist = new ItemList("idlist.txt");
	thelist.writeReviewsToDatabase("reviews.db", false);

Get pricing information

To get the pricing information you need to register as an Amazon Associate ( Then you need to add your Associate tag in the signInput() function in

	variablemap.put("AssociateTag", "your_tag_here");

You also need your Product Advertising API Key & Secret Key and add them to

	private String awsAccessKeyId = "your_api_key";
	private String awsSecretKey = "your_seceret_key";

Once you have them you can change the second argument of writeReviewsToDatabase() to true, and pricing information will be saved in the same database in XML format.

To test your keys, try

	Item testItem = new Item("B00D02AGU4");

Common Exceptions

  1. means that the item no longer exist on You do not have to do anything with that item.
  2. means that connection to the website is taking too long. Rerun the crawler on the items with this exception.

Update Log

2015-11-14 updated to reflect changes on Amazon website.

2016-06-25 Amazon is no longer displaying total number of helpful votes.


The code is released into public domain. If you find the code useful in your research work, I appreciate if you can cite "Market Dynamics and User-Generated Content about Tablet Computers" by Xin (Shane) Wang, Feng Mai and Roger H.L. Chiang, Marketing Science 33.3 (2014): 449-458


A crawler to collect reviews and product information on







No releases published


No packages published