VideoMiner: YouTube and Vimeo Content Aggregator

VideoMiner is a project to extract, process, and centralize information from YouTube and Vimeo channels and videos. It is based on a microservices architecture that queries the official APIs, transforms the data, and exposes it through a REST API.

---

Table of Contents

• Features
• Architecture
• Requirements
• Configuration
• Getting Started
• API Documentation
• Usage Examples
• Technologies
• Notes and Best Practices
• Roadmap
• License

---

Features

• Channel data extraction: description, creation date, and video listing.
• Video details: title, description, publication date, comments, and subtitles (when available).
• Centralized storage: persistence in an H2 database and query/management through a REST API.
• Modular architecture: independent services for YouTube and Vimeo that publish data to the main service.

---

Architecture

The system consists of three services:

1. VideoMiner-main
   Core service. It exposes the REST API to manage and query channels, videos, comments, and subtitles stored in the H2 database.

2. YoutubeMiner-main
   Microservice that consumes the YouTube Data API. It locates channels, retrieves their videos, and extracts comments and subtitles, then sends them to VideoMiner.

3. VimeoMiner-main
   Microservice that consumes the Vimeo API. It performs equivalent functions for the Vimeo platform.

Workflow: YoutubeMiner / VimeoMiner retrieve data from the platforms and publish it to VideoMiner, where it is stored and made available for querying.

---

Requirements

• JDK 17 or higher
• Apache Maven
• An IDE (IntelliJ IDEA, Eclipse, VS Code) or terminal

---

Configuration

Each microservice may require environment variables or properties for API credentials and network settings. Common examples:

• YoutubeMiner-main
  • YOUTUBE_API_KEY: YouTube Data API v3 key.
• VimeoMiner-main
  • VIMEO_ACCESS_TOKEN: Vimeo OAuth access token.
• VideoMiner-main
  • H2 database settings and CORS configuration if it will be consumed from a frontend.

Values can be configured in:

• application.yml / application.properties
• System environment variables
• -D parameters when starting Maven/Spring Boot

---

Getting Started

Clone the repository and go to the folder:

git clone https://github.com/tu_usuario/VideoMiner.git
cd VideoMiner

Run each service in a separate terminal:

VideoMiner-main (port 8080):

cd VideoMiner-main
mvn spring-boot:run

VimeoMiner-main (port 8081):

cd VideoMiner-main/VimeoMiner-main
mvn spring-boot:run

YoutubeMiner-main (port 8082):

cd VideoMiner-main/YoutubeMiner-main
mvn spring-boot:run

Once running:

• http://localhost:8080 → VideoMiner
• http://localhost:8081 → VimeoMiner
• http://localhost:8082 → YoutubeMiner

---

API Documentation

OpenAPI/Swagger documentation is available at:

• VideoMiner API: http://localhost:8080/swagger-ui.html
• VimeoMiner API: http://localhost:8081/swagger-ui.html
• YoutubeMiner API: http://localhost:8082/swagger-ui.html

---

Usage Examples

Add a YouTube channel to VideoMiner

Make a POST request to the YouTube microservice so it can retrieve the data and publish it to VideoMiner:

curl -X POST   http://localhost:8082/api/youtube/v3/channels/{channelId}

Query channels stored in VideoMiner

curl -X GET http://localhost:8080/videominer/channels

---

Technologies

• Java 17
• Spring Boot 3
• Spring Data JPA + Hibernate
• H2 Database (in-memory)
• SpringDoc / Swagger UI for API documentation
• Maven for dependency management and build automation