Skip to content

show full stack technology applications : Scrapy + webservice[restful] + websocket + VueJS + MongoDB

Notifications You must be signed in to change notification settings

fanqingsong/web_full_stack_application

Repository files navigation

Full Stack Web Application

Purpose

This is a demo project for showing how one full stack web application should be like.

Introduction

features:

  • scrawler get data into database
  • webserver push data to browser periodically

Demo

page

Architecture

+-----------+                 +------------+              +------------+               +--------------+
|           |      frontend   |            |              |            |      data     |              |
|   Browser +<----------------+  Vue Server|              |   RestAPI  +<--------------+    Scrapy    |
|           |                 |            |              |            |               |              |
+------+----+                 +------------+              +------+-----+               +--------------+
       ^                                                         |
       |                                                         |
       |                                                         |data
       |                                                         |
       |                                                         |
       |                                                         v
       |                     +-------------+              +------+------+
       |     data            |             |    data      |             |
       +---------------------+  Websocket  +<-------------+   MongoDB   |
                             |             |              |             |
                             +-------------+              +-------------+

Workflow

  1. scratch quotes from scrapy offical demo site(quotes), save to database(MongoDB), and show them through web.
  2. Front end is vue.js project, including WebSocket client(socketio) component.
  3. Back end is websocket server, which query data form MongoDB through ORM mongoose every period(1s), pushing quotes data to front end.
  4. data picker use scrapy to crawl quotes to save to MongoDB.

Technology Stack

Scrapy + webservice[restful] + websocket + VueJS + MongoDB

Component Description
Scrapy scratch data from web
webservice serve restapi for data storation and fetch
websocket push data to browser
VueJS frontend
MongoDB DB

Install

./bin/install.sh

Run

# run all services
./bin/start.sh

# stop all services
./bin/stop.sh

# start scrapy
./bin/start_scrapy.sh

Reference

About

show full stack technology applications : Scrapy + webservice[restful] + websocket + VueJS + MongoDB

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published