Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metauniService audio workflow for Replays #56

Open
dmurfet opened this issue Jan 7, 2024 · 1 comment
Open

metauniService audio workflow for Replays #56

dmurfet opened this issue Jan 7, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request replay Related to the replay system

Comments

@dmurfet
Copy link
Member

dmurfet commented Jan 7, 2024

Here's how I imagine this working. I record a replay, giving me an audio file (say 30min long) in mp3 format. I upload this to metauniService along with some identifier for the replay (consisting of the pocketId and replayId, the in-game UI should provide this string for me to just paste into a field on the webpage). Then the code

(a) splits the mp3 file into 6min chunks
(b) processes the mp3 file to generate a transcript (OpenAI has an API for this, should be easy)
(c) uploads the files to Roblox Cloud, gets asset IDs
(d) modifies the DataStore entry for the replay to add the asset IDs and timestamp data, along with the transcript data

This requires me to understand the data structure storing the replays (presumably straightforward) and to have additional fields added for transcripts (text).

Ideally we would generate embeddings as we were doing with Pinecone, but this is prohibitively expensive to do in a hosted manner at the moment, and further work in this direction would require hosting our own vector DB at metauniService (doable, maybe a day's work).

@dmurfet dmurfet added enhancement New feature or request replay Related to the replay system labels Jan 7, 2024
@dmurfet dmurfet self-assigned this Jan 7, 2024
@dmurfet
Copy link
Member Author

dmurfet commented Jan 7, 2024

We can SRT (subtitle) files from the OpenAI transcription API (https://platform.openai.com/docs/api-reference/audio/createTranslation). This means text along with timecodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request replay Related to the replay system
Projects
None yet
Development

No branches or pull requests

1 participant