Our goal is to build a system that discover interesting facts from web data. The system consist of three major components, including web crawler, exploratory analysis, and data visualization.
Based on our experiences on web development and data science, as well as the descriptions mentioned above, we take Feb, 2016 as the 1st stage with the primary goal of prototyping our application following the development guild lines mentioned below. Here's the tentative timeline.
- [2016/02/08 - 2016/02/12] Project Selection, Plan Discussion, Proposal Draft Writing, Resource Discovery
- [2016/02/13 - 2016/03/07] System Design, Project Implementation
- Web Crawler
- Exploratory Analyzer / Recommender
- Data Visualization
- [2016/03/08 - 2016/03/15] Document Writing, User Manual Writing and Video Presentation Making
Details of each schedule and task will be added later.
- [BitTiger Project: AppStore - Website] https://slack-files.com/T0GUEMKEZ-F0J4G9QTT-274d3bc97e
- Javascript: Node.js, Express.js, AngularJS
- Database: MongoDB
- Cloud Platform: Cloud Foundry
- Modularity. Following the principle "loose coupling and high cohesion", each module should be standalone.
- Minimalism. Each module should be kept short, simple, and concise. Every piece of code should be transparent upon first reading.
- Easy extensibility. New modules (as new classes and functions) are should be simply add, and existing modules should be extended easily.
Crawler
MEAN Stack
MEAN is an acronym for MongoDB, Express.js , Angular.js and Node.js
MongoDB: MongoDB is an open-source, document database (NoSQL) designed for ease of development and scaling.
- MongoDBx: M101x Introduction to MongoDB using the MEAN Stack
- M101JS: MongoDB for Node.js Developers
- www.mongodb.org
- mongoose
Express.js: Fast, unopinionated, minimalist web framework for Node.js.
Angular.js: Angular is a development platform for building mobile and desktop web applications.
Node.js: Node.js is a JavaScript runtime built on Chrome's V8 JavaScript engine.
@team: Pikachu