Skip to content
Crawl Uniqlo online store with Scrapy, and build server application with Spring Boot.
Python Java
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
backend
spider
.gitignore
README.md
uniqlo_db.png

README.md

Uniqlo

Crawl Uniqlo online store, provide api for further analysis.

Installation

spider

Use Scrapy to crawl data.

  1. Go to uniqlo/spider folder, install dependencies.
pipenv install
  1. Set up Postgres, create database.
psql
\i spider/spider/db/init_db.sql
  1. Copy uniqlo/spider/config-template.py to uniqlo/spider/config.py, modify the databse related information (e.g., DB_HOST, DB_PORT, etc).

  2. Run scrapy. Go to uniqlo/spider/spider folder, run the following command:

scrapy crawl uniqlo_ca

backend

  • Use Spring Boot to provide database access and REST api support.
  • Use Shiro for authentication and authorization support.

Go to uniqlo/backend folder, run the application with following command:

./mvnw spring-boot:run

Database Design

uniqlo_db

You can’t perform that action at this time.