Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. License

About The Project

shopee-crawler is a simple project to crawling shopee's shop and product information. Some common information can be crawled:

• shop:
  • shopid
  • follower_count
  • shop_location
  • name
  • ...
• product: can crawl all shop's products by recursive function
  • itemid
  • shopid
  • name
  • view_count
  • like_count
  • status
  • categories
  • price
  • description
  • currency
  • attributes such as orgin, material,...
  • ...
• review: can crawl all product's reviews by recursive function
  • reviewid
  • itemid
  • orderid
  • rating
  • comment
  • images
  • videos
  • ...

Built With

• Node.js

Getting Started

To get a local copy up and running follow these simple example steps.

Prerequisites

Make sure that you had installed Node.js

• npm

  npm install npm@latest -g

Installation

1. Clone the repo

   git clone https://github.com/hoangkimminh/shopee-crawler

2. Install NPM packages

   npm install

Usage

1. Modify shops_username inside get_shops_details.js with your own shop's names

const shops_username = [
  "minastore_vn",
  "friday.official",
  "germe.vn",
  "sliky",
  "yinxx.vn",
];

2. Run scripts to get shop details

node get_shops_details.js

3. Start crawling all products and its reviews of each shop with

node index.js

Data of each shop will be saved in output/[shopname].json Make sure output folder created

License

Distributed under the MIT License. See LICENSE for more information.