- Continuously ingest data from an API, store it efficiently and visualise the results
- This project aims to improve skills such as database management, data engineering, web development, and application design, with an emphasis on modularity and computational efficiency
The main objective is to develop a seamless pipeline for real-time monitoring of blood glucose levels using data from Libre Freestyle sensors. The project emphasizes efficient data processing, storage, and visualization, providing insights into glucose levels over time. The use of psycopg2 for database interactions and connection pooling are key components in enhancing the system's performance.
The data is sourced directly from my Libre Freestyle sensor, utilizing its API to fetch real-time Continuous Glucose Monitoring (CGM) data.
- Python
- PostgreSQL
- Requests
- NumPy
- psycopg2
- SQLAlchemy
- Flask
- Pandas
- Plotly
- schedule
- atexit
- Data Fetching: Utilizing the
requests
library to connect to the Libre Freestyle sensor API and fetch real-time CGM data. - Data Processing: Applying basic transformations such as data type conversions to ensure data integrity and compatibility with the database.
- Database Management: Employing
psycopg2
for efficient data loading into a PostgreSQL database, utilizing connection pooling to optimize database interactions and reduce overhead. - Visualization: Implementing a Flask application to visualize the blood glucose data using
Plotly
, providing real-time insights into glucose levels. - Continuous Operation: Using
schedule
for periodic data fetching andatexit
to ensure graceful shutdown of processes, the system operates continuously on a VM for uninterrupted data monitoring and analysis.
- Managing database connections, executing transactions, and retrieving data
- Relational database operations and SQL
- Optimising data storage and retrieval processes, improving data commit efficiency
- Fetching data from external APIs, integrating third-party data sources into application
- Handling API authentication and managing data queries based on specific user contexts
- Preprocessing and transforming raw data for database insertion
- Handling of various data types and management of missing or inconsistent data
- Analysing and visualising data using Flask for presentation
- Object-oriented programming, structuring functionality into Management classes like AuthenticationManagement
- Implementation of web-based interfaces using Flask for data visualisation
- Integration of data processing and visualisation into a web application, enhancing user interaction
- Managing data persistence using various formats (e.g., JSON, Parquet), enabling flexible data storage and access patterns.
- Management of data files, including saving, loading, and amending operations, to support the application's data lifecycle.
For more information or to discuss this project further, please reach out.
Email: delstonds@outlook.com