This is a simple python script to scrape stock prices off NASDAQ API and feed it to MySQL. I have spent the past 2 weeks pondering on what beginner project I should do to apply my SQL knowledge and where do I even obtain the data from? I spent some time researching and found a few option. I can import a csv file into SQLite table using SQLite3 or DB browser for SQLite or scrape the data off a webpage and feed it into MySQL. I decided to take up the challenge to extract the data myself since that is a great way for me to practise on my python skill. Do note that I am scrapping stock prices per minute and not per day.
- Python 3.8x
- python-dateutil 2.8.1 or newer
- ujson 1.35 or newer
- urllib3 1.25.8 or newer
- mysql-connector-python 8.0.18 or newer
- MySQL 8.0.19 or newer
- Internet connection
- Python: You can install Python using Anaconda. It is the easiest way to install python and it comes with a good amount of data science packages.
- MySQL: I used MySQL Community server. Please remember your password as it will need to use it everytime you log into the server.
- python-dateutil,ujson,mysql-connector-python,urllib3 can be installed using pip.
$ pip install mysql-connector-python
conda install -c anaconda mysql-connector-python
What to do before running code
- Log in using:
mysql -u root -p
- Create a database call stock
CREATE DATABASE stock;
- Select database
- Create a table on your database call nasdaq
CREATE TABLE nasdaq ( datetime DATETIME, price FLOAT(2,2), symbol VARCHAR, created_at TIMESTAMP default now() );
- Edit the config file
- You will need to change the db_user and db_password to your own. This is an essential step to prevent the disclosure of your server details.
How to check if your code works
- Using SELECT * from nasdaq
SELECT * from nasdaq
- You should able to see 4 columns of data (datetime, price, symbol and created_at)
This scraper is licensed under the MIT license. You can check the information inside the LICENSE file. To make it short, I wanted it to be free and open, so that anyone can contribute to it.