This project utilizes the Kaggle Netflix movies and shows dataset, which can be found here. In our environment, we have transformed this data into a relational schema. This schema not only embodies the original dataset but also fabricates additional tables and data related to users, user ratings, viewing sessions, and recommendations. The primary intent behind this project is dual-faceted:
- To provide analysts with access to raw data for analytical pursuits.
- To facilitate data scientists in their endeavors, especially while simulating or developing machine learning recommender systems.
- Database: PostgreSQL
- API Framework: FastAPI
- ORM: SQLModel
- Analysis and Development: Python/Python Notebooks
-
Dependencies Installation:
- Navigate to the
artifacts
folder and locate therequirements.txt
file. - Install the necessary Python packages by running:
pip install -r artifacts/requirements.txt
- Navigate to the
-
Setting up the Environment:
- If you're keen on accessing the system either as an analyst or a data scientist, ensure you create and configure your postgre db first, and then put your admin credentials in a .env file in your project directory.
- Run the files in the order they are listed below.
- Analysts have the flexibility to interact using the provided notebook or tools like pgAdmin.
- Data scientists can harness either the local API or the web API to perform simulations, especially when conceptualizing a recommender system.
Location: src/da_rawschema
This part is dedicated to analysts who need access to the raw data. It facilitates both ad hoc queries and structured analyses through Python notebooks.
- Load Raw Tables:
load_raw_csvs_1.ipynb
- This notebook is responsible for ingesting raw tables and populating them into a non-relational schema. - User Credentials Creation:
create_DA_creds_2.ipynb
- Utilize this notebook to generate a user with read privileges to the raw tables. - Analyst Notebook:
analyst_env_3.ipynb
- An environment for analysts to run read queries against the raw schema in the recommender database.
Location: src/ds_relational_schema
and src/db_api
The relational schema is built and managed through:
- Relational Schema Creation:
relational_schema_1.ipynb
- This notebook creates tables in the relational schema within the recommender database. - DS User Credential Creation:
create_DS_creds_2.ipynb
- It's designed to create a user with full control over the relational schema.
Following the schema creation, the local database API allows interaction with the relational schema:
- Local Database API:
db_local_api.py
andlocal_query.py
- This Python API permits read and write queries. - Demonstration Recommender:
demo_local_recommender.py
- A simple recommender system to illustrate the functionality of the local API.
Location: src/ds_api/ds_web_api
- FastAPI Web API:
ds_web_api
- This is a FastAPI powered API offering read, write, and delete endpoints (update functionality coming soon) that interact with the database through JSON payloads. - Data Models:
models.py
- Validates both incoming requests and responses using SQLModel to ensure integrity.
Location: src/ds_api/tests
This testing suite, while still under development, provides foundational tests for:
- DB Connection:
conftest.py
- Local Python API:
test_db_local_api.py
- Web API:
test_db_web_api.py
To run the tests, navigate to the src/ds_api/
directory and run:
export PYTHONPATH=/Users/bfaris96/Desktop/turing-proj/recommender_db/src/ds_api:$PYTHONPATH
Then, run:
pytest
- Database ERD: Find an image representation of our latest database Entity-Relationship Diagram (ERD) in the
artifacts
directory. - Documentation: Refer to
Doc/plan.ipynb
for initial plans, decision-making notes, and checklists to grasp our development trajectory.