This project is now archived. The visualization still works but has stopped being updated as of March 30th, 2022. Archival was set due to no longer wanting to pay for API usage.
Any data in this project or on my website is for informational purposes only and should not be taken as invesment advice.
- Extracts and transforms S&P 500 stock data with Python from a financial API.
- Data is loaded into Cloud Storage then transferred to BigQuery and rendered on my webpage.
- Python code runs on a scheduled cron job through a virtual machine with GCP Compute Engine.
- A cron job triggers
main.py
to run. main.py
calls the IEX Cloud API.- The data is processed and cleaned by removing commas, hyphens, and/or other extra characters from the company name column.
main.py
creates acsv
file with the prepared data.load.py
copies thecsv
file to a Cloud Storage bucket.- The
csv
file is loaded to BigQuery. - Using the BigQuery API and when the webpage is loaded, the data is queried and then displayed.
- None
- The file that connects to BigQuery to pull the data when the page loads is located in my wesbite repository since that renders the frontend.
- The pipeline does not account for holidays.
- APIs: IEX Cloud
- Google Cloud Services:
- Virtual Machine: Compute Engine
- Object Storage: Cloud Storage
- Data Warehouse: BigQuery
- Scheduler: cron
- Visualization: Flask and HTML