This project builds an automated data pipeline to scrape, process, and analyze skincare product reviews from the Female Daily website. The pipeline is fully integrated within Docker, utilizing Selenium, Apache Airflow, Spark, and Metabase to automate data collection, transformation, and visualization.
├── dags
│ ├── resources
│ │ ├── scripts
│ │ │ └── scrape.py
│ │ ├── sql
│ │ │ ├── bronze
│ │ │ │ ├── category.sql
│ │ │ │ ├── product.sql
│ │ │ │ ├── review.sql
│ │ │ ├── silver
│ │ │ │ ├── category_transformed.sql
│ │ │ │ ├── product_transformed.sql
│ │ │ │ ├── review_transformed.sql
│ │ │ ├── gold
│ │ │ │ ├── dim_category.sql
│ │ │ │ ├── dim_product.sql
│ │ │ │ ├── dim_review.sql
│ │ │ │ ├── fact_product_review.sql
│ ├── scrape_product_dag.py
│ ├── transform_data_dag.py
│ ├── gold_layer_dag.py
│
├── docker
│ ├── docker-compose-airflow.yml
│ ├── docker-compose-postgres.yml
│ ├── docker-compose-spark.yml
│ ├── docker-compose-metabase.yml
│ ├── Dockerfile.airflow
│ ├── Dockerfile.spark
│
├── jars
│ ├── postgresql-42.2.18.jar
│
├── scripts
│ ├── entrypoint.sh
│ ├── metabase_create_conn.sh
│ ├── metabase_entrypoint_wait.sh
│ ├── metabase_entrypoint.sh
│
├── sql
│ ├── warehouse-setup.sql
│
├── Makefile
-
Scraping (Bronze Layer):
- Uses Selenium to scrape skincare product data and reviews.
- Stores raw data in PostgreSQL.
- Managed by
scrape_product_dag.pyin Apache Airflow.
-
Transformation (Silver Layer):
- Cleans and structures raw data using SQL transformations.
- Converts unstructured text into structured format.
- Managed by
transform_data_dag.py.
- Data Warehouse (Gold Layer):
- Aggregates and organizes data into dimensional models.
- Creates fact and dimension tables for analysis.
- Managed by
gold_layer_dag.py.
- Visualization:
- Uses Metabase to create dashboards for insights on product trends and reviews.
The dashboard is built using Metabase and includes several charts to analyze skincare product reviews:
- Filtering Options:
- Platform: E-commerce, supermarket, event, website.
- Skin Type: Dry, normal, oily, combination.
- Skin Tone: Dark, light, medium.
- Undertone: Cool, warm, neutral.
- Date Filter: Select time range for analysis.
- Overall Performance: Total products, total categories, and total reviews.
- Review Breakdown:
- Review per undertone (cool, warm, neutral).
- Review per skin tone (dark, light, medium).
- Review per skin type (dry, normal, oily, combination).
- Review per usage period.
- Review by rating.
- Review per reviewer age.
- Top Products Analysis:
- Top products by rating per category.
- The cheapest high-rated (rating > 4.5) product per category.
- Selenium – Web scraping
- Apache Airflow – Workflow orchestration
- Apache Spark – Data processing
- PostgreSQL – Data storage
- Metabase – Data visualization
- Docker – Containerization
git clone https://github.com/iisyafiraa/skincare-product-review.gitmake docker-build
make all- Access Airflow UI at
http://localhost:8080 - Enable and trigger:
scrape_product_dagtransform_data_daggold_layer_dag
- Go to
http://localhost:3000 - Configure PostgreSQL connection using
scripts/metabase_create_conn.sh - Create and explore dashboards
The project includes a Makefile to simplify Docker container management. Below are some useful commands:
make docker-build # Build all Docker images
make postgres # Start PostgreSQL container
make airflow # Start Airflow scheduler & webserver
make spark # Start Spark cluster
make metabase # Start Metabase for visualization
make all # Run all services (PostgreSQL, Airflow, Spark, Metabase)
make postgres-bash # Access PostgreSQL container shell
make give-all-permission # Set necessary file permissions- Implement sentiment analysis on reviews.
- Enhance data enrichment with additional sources.
- Optimize scraping process for scalability.
Author: Isyafira





