This project was developed in a couple of days for a test assignment and needs many improvement to make it perfect. Jupyter Notebook and Pandas were used to extract the data from http://testing-ground.scraping.pro/table. Pandas Data Frame was used to insert the data into MySQL database. Flask was used for writing Rest APIs. Bootstrap, HTML, JavaScript, Jquery was used in front-end. HighChart.Js was used for data visualization.
Note: This project is not intended for production deployment.
- Create a database in MySQL named "application" with a table named "records".Import database_structure.sql to generate the database schema OR import database_structure_with_data.sql to generate database along with scraped data.
- If database is created with database_structure.sql (a database with no data), run the testing-ground.scraping.py OR testing-ground.scraping.ipynb in Jupyter Notebook to extract the data and insert it into MySQL database.
- Download the code and open it in your sublime. Use Python3
- Install all the necessary imports using the pip tool. (Install Flask, flask_sqlalchemy, pymysql, pandas)
- Run App.py (CTRL + B)
Create, Update, Get and Delete APIs were implemented to work with the data in the database. Adding OR Updating data may cause discrepancies in some cases since I scurried to finish the job.