This project is a proof of concept that utilizes the power of advanced Artificial Intelligence to extend the capabilities of standard video streaming. By integrating a state-of-the-art Large Language Model (LLM), the application does not just stream video content but enriches it with intelligent analysis in real-time.
GoPro Control API: A flexible API that integrates with the GoPro to control basic functions such as starting/stopping recordings and live streaming.
Video Streaming: The API supports converting MPEG-TS packets from the GoPro into HLS format, facilitating video streaming to various devices.
LLM Integration: Processing frames of the stream and passing it to a LLM gpt-vision-preview
TTS: Using a Text To Speech model to sent content as audio through HTTP Live Streaming
Celery with Redis for Background Processing: Video streaming functionality is handled through a background worker in Celery with Redis, optimizing server performance.
Frontend Interface: A simple user interface, developed using Vite, React, TypeScript, and Tailwind CSS.
PROJECT_JARVIS harnesses the power of advanced Artificial Intelligence to extend the capabilities of standard video streaming. By integrating a state-of-the-art Large Language Model (LLM), specifically gpt-4-vision-preview, the application does not just stream video content but enriches it with intelligent analysis in real-time.
- Utilizing FFMPEG, the server manages the conversion of incoming packets to the appropriate format and handles the periodic frame capture.
- Post-analysis, the AI-generated content is delivered as audio through the TTS model, complementing the visual feed.
- Robust background processing is achieved using Celery with Redis, ensuring seamless performance and scalability.
Backend: Python, FastAPI Frontend: Vite, React, TypeScript, Tailwind CSS Data Processing: Celery, Redis Video Processing: FFMPEG/HLS for streaming
- A GoPRO Hero 12
- Python (3.8)
- Nodejs
- Celery
- Redis
Make sure you have Python installed on your computer. For streaming, the API is running a background worker with Celery and Redis as a broker. It expects this to be running here ::redis://localhost:6379/0. If you don't want to use this background worker, you can change the code as you like.
This API provides a flexible interface for integrating with GoPro cameras, utilizing a wireless HTTP connection over WiFi. This enables users to effortlessly connect to their GoPro and send various commands, such as starting/stopping recordings and managing live streaming.
cd gopro-api
You can start installing the API by running the following command in your terminal:
pip install -r requirements.txt
This API is using FastAPI as a webserver. To run the app use the following command.
uvicorn main:app --reload
celery -A main.celery_app worker --loglevel=INFO
cd gopro-frontend
npm install
npm run dev