OpenSea ETL Pipeline with Custom ORM

Overview

This project implements an ETL (Extract, Transform, Load) pipeline that fetches collection data from the OpenSea API, transforms it, and stores it into a SQLite, PostgreSQL, or MySQL database using a custom ORM (Object-Relational Mapping) system.

---

Features

• ETL Pipeline: Extracts OpenSea collections data, transforms it, and loads it into a configured database.
• Custom ORM: Supports full CRUD operations and schema modifications without relying on third-party ORM frameworks.
• Multi-Database Support: Compatible with SQLite, PostgreSQL, and MySQL.
• Data Storage: Persists both raw JSON responses and cleaned, structured database records.
• Filtering and Sorting: Implements query filtering via LIKE, ILIKE, and IN operators, as well as ORDER BY sorting.

---

Project Structure

Sweeft_Python_Engineering/
├── main.py              # Entry point; orchestrates the ETL pipeline
├── custom_orm.py        # Custom ORM implementation
├── etl_pipeline.py      # Core ETL logic (extract, transform, load)
├── requirements.txt     # Python dependencies
├── README.md            # Project documentation
├── .env                 # API key and database connection URL
└── opensea.db           # SQLite database file (auto-generated at runtime)

---

Installation and Setup

Prerequisites

• Python 3.8 or higher
• pip

Steps

1. Clone the repository

git clone https://github.com/Aleksandre16/Sweef_Python_Engineering
cd Sweef_Python_Engineering

2. Install dependencies

pip install -r requirements.txt

3. Configure environment variables

Create a .env file in the project root and set the following variables:

OPENSEA_API_KEY=your_api_key_here
DATABASE_URL=sqlite:///opensea.db

For PostgreSQL or MySQL, replace DATABASE_URL with the appropriate connection string.

4. Run the pipeline

python main.py

---

Configuration

Variable	Description	Example
OPENSEA_API_KEY	Your OpenSea API key	abc123...
DATABASE_URL	Database connection string	sqlite:///opensea.db

---

Author

Developed by Aleksandre Kolbini

• GitHub: github.com/Aleksandre16
• LinkedIn: linkedin.com/in/aleksandre-kolbini-266219254