A backend server and database of product information for an e-commerce web app. My goal was to replace an existing API with a back end system that can support the full data set for the project and to scale to meet the demands of production traffic.
After I designed a database(PostgreSQL) and the server, I also deployed to AWS EC2 and scaled the service to support at minimum 1k RPS(requests per second) on AWS EC2 using a t2.micro instance.
- Design and multiple database options to analyze, compare and select one database to use.
- ETL Process (Extract the data from csv files, and Load them into the database after Transformating)
- Design and build an API server to provide data to the client in the format specified by the API doc.
- Optimize the server by analyzing query times and server responses.
- Deploy the service and improve the performance of you service at sacle.
- Phase 0: Picking a project and initial setup
- Phase 1: Create the Database
- Phase 2: Create the API
- Phase 3: Performance Tune the Service
- Phase 4: Deploy and Benchmark Initial Performance
- Phase 5: Scale the Application
- Node.js
- Nodemon
- Node express
- Axios
- Npm
- PostgreSQL
- Webpack
- Babel
- Loader.io
- Artillery
- AWS EC2
CSV File Name | Size | Lines |
---|---|---|
product.csv | 288 MB | 1,000,012 |
features.csv | 85 MB | 2,218,659 |
skus.csv | 653 MB | 26,961,740 |
photos.csv | 3,650 MB | 13,463,360 |
styles.cvs | 158 MB | 4,660,355 |
related.csv | 96 MB | 4,510,018 |
Total | 4,930 MB | 52,814,144 |
Request Type | Endpoint | Returns |
---|---|---|
GET | /products | Retrieves the list of products |
GET | /products/:product_id | Returns all product level information for a specified product id |
GET | /products/:product_id/styles | Returns the all styles available for the given product |
GET | /products/:product_id/related | Returns the id's of products related to the product specified |
I used the Artillery for the local stress test and the Loader.io for the cloud-based load test.
API Service | Response Time/1 RPS | Median Response Time/100 RPS | ERROR Rate/100 RPS |
---|---|---|---|
Test Type | Postman | Artillery | Artillery |
List Products | 27ms | 5,711 ms | 98.4% |
Product Information | 188ms | 6,064 ms | 98.6% |
Product Styles | 53,568ms | N/A | 100% |
Related Products | 607ms | 5,487 ms | 56.6% |
API Service | Response Time/1 RPS | Median Response Time/100 RPS | ERROR Rate/100 RPS |
---|---|---|---|
Test Type | Postman | Artillery | Artillery |
List Products | 22 ms | 5 ms | 0% |
Product Information | 25 ms | 5 ms | 0% |
Product Styles | 28 ms | 8.9 ms | 0% |
Related Products | 19 ms | 4 ms | 0% |
- Navigate to the folder where it was cloned.
- With npm installed,
npm install
npm start
-> This will install all necessary packages, and start the express server. Open http://localhost:3000
- You can test the api by sending curl commands or a service to send requests for you such as Postman.
- Alternatively, you could install one of the front end clients and direct it to this api in order to fully use it.
- Byungchan Lee