Transcode

Goal: Leverage the Oracle Cloud Free Tier to transcode video files to AV1 to optimize storage space.

• Source: Original videos are stored in a local directory.
• Remote Processing: Files are transcoded to AV1 on remote Oracle Cloud instances.
• Retrieval: Completed transcoded videos are downloaded back to a local destination folder.
• Cleanup: Remote copies are purged after a successful transfer to maintain free space.

Operational Workflow: This project is designed for intermittent local availability. The local PC is only powered on when in active use by the user, not specifically for transcoding. The synchronization process is intended to run automatically in the background during these periods to manage the remote queue.

The project consists of two main components:

• transcode.sh - Designed to run as a cron job on the Oracle Cloud instance:

  1. Uses flock to ensure only one instance is active at a time.
  2. Scans for files, ignoring partial uploads (e.g., hidden files generated by rsync).
  3. Executes ffmpeg to perform the AV1 transcoding.
  4. Deletes the source file on the cloud once transcoding is successful.

• sync_transcoded.py - Designed to run automatically on the local Windows machine (e.g., via Task Scheduler) whenever it is powered on:

  1. Downloads finished transcoded files and removes them from the Oracle Cloud server.
  2. Verifies that sufficient disk space exists on the cloud instance before proceeding.
  3. Uploads new source files for the next batch of transcoding tasks.

---

Data Flow

To ensure stability and allow for future pipeline stages, the project uses a numbered directory structure:

1. Local Source: The local directory where you place raw video files.
2. 01_upload_staging (Server): Temporary area for incoming files (prevents processing partial uploads).
3. 02_transcode_queue (Server): Files moved here are picked up by transcode.sh for processing.
4. 03_transcode_staging (Server): Where ffmpeg writes the temporary output file during encoding.
5. 04_transcode_finished (Server): Final server destination for completed files, waiting for download.
6. Local Destination: The local directory where finished files are automatically retrieved.

Server instructions

Install

Clone repo + Setup cron job

cd /home/ubuntu
git clone https://github.com/FlippFuzz/transcode
cd transcode
chmod +x transcode.sh

# Add the cron job to run every minute (non-interactive and idempotent)
(crontab -l 2>/dev/null | grep -Fv "/home/ubuntu/transcode/transcode.sh"; echo "* * * * * /home/ubuntu/transcode/transcode.sh") | crontab -