VideoGenAI

An AI-driven tool that suggests relevant footage for certain frame descriptions by automatically organizing/ summarizing/ tagging/ selecting video footage on mobile phone to assist video creation.

• Project Introduction Video
• Demo Video

Setup

Setup Environment:

cd videogen_backend
pip install -r requirements.txt

To run backend:

cd videogen_backend
flask --app app.py run # locally
python3 -m flask --app app.py run --host 0.0.0.0 -p 5000 # server

Frontend:

• Just open the frontend folder in xcode. Remember to install ios simulator, 16+ is recommended.

Features to includes

v0.1

Features(Higher Priority):

Infrastructure:

• [Backend]: Video file database, static database, support video preview (keyframes slideshow).
• [Backend]: Relevant keyframe extractions of video files
• [Backend]: Video footage content categorization, tagging, meta data storing.
• [Backend]: Indexer for video data indexing
• [Backend]: NLP searching of relevant footage with indexer
• [Backend]: Database for editing progress
• [-] [Backend]: Generate Video Summary

Library Page:

• [Backend]: Return all assets for Library via Pagination
• [Backend]: Receive Multiple Video files in one upload
• [Backend]: Delete Video from Library
• [Backend]: Search based on keywords
• [Frontend]: Select and upload a relevant footage
• [Frontend]: Select and upload multiple relevant footages
• [Frontend]: Loading when uploading
• [Frontend]: Fetch and show Video Libraries Efficiently
• [Frontend]: Delete Selected Videos
• [Frontend]: Library Page Layout
• [Frontend]: Show Searched Result
• [Frontend]: Implement actions for preview buttons of videos, copy video url & safe video file.
• [Frontend]: Video List Pagination Loading

Projects Page:

• [Backend]: Create Project and Save Project
• [Backend]: Update Current Editing Process in Database
• [Backend]: Delete a Project
• [Backend]: Return Projects List in Pagination
• [Frontend]: Video Projects Page with Pagination to select desired project to work on.
• [Frotnend]: Create Project & direct to Timeline Page
• [Frontend]: Remove Project

Timeline Page:

• [Frontend]: Project Title updates
• [Frontend]: Q&A Block Component
• [Frontend]: Video Timeline page for multiple video selection blocks and place the selected footage in a desired sequence.
• [-] [Frontend]: Update Video Sequence in frontend and reflects in the database
• [Frontend]: Preview Entire Timeline
• [Frontend]: Video downloads as well as export to third party apps like CapCut.
• [Backend]: Get Project Details

Block Page:

• [Frontend]: Block page for relevant footage queries and preview.
• [Frontend]: Remove Video if disliked
• [-] [Frontend]: Reorder Responses

Improvements/Known Issues (Lower Priority):

• [Backend][Improvement] Upload Large files that are larger than 100MB per upload request
• [Frontend][Bug] Efficient Library Videos Loading, app crashing because all videos are taking up the main thread.
• [Frontend][Improvement] Preview of the entire video clip
• [Frontend][Improvement] For files returned without "tags", mark them as "INDEXING" instead
• [Frontend][Improvement] Hide Search Bar when in edit mode.
• [Backend][Bug] Remove video meta data in pinecone indexer when removing the video from cloudinary.
• [Frontend][Improvement] Navigation Bar Title Style.
• [Frontend][Bug] Projects Page Long Press Position Error
• [Frontend][Bug] Library Page and Projects page Pagination not working after actions(ex. Delete, Search, etc).

Future Scope

• [Backend]: Script to Video Footage selections in sequence generation
• [Backend]: Auto Video length trimming functionality.
• [Backend]: User database and session management.
• [Backend]: Video Indexer per user
• [Frontend]: Script & video gen result preview page.
• [Frontend]: User Login and Profile Page

Video Assets Pre-process Procedure

1. Upload Video to Backend as Cache.
2. Upload video in the cache to Cloudinary, and store video id on Cloudinary.
3. Extract Key frames, and Store Summary, Tags, and Cloudinary ID in the pinecone database.
4. Query Using Natural Language and Parse into Pinecone DB Queries
5. Search and return Pinecone DB Ranks
6. Show in frontend