Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transcribe: Add support for transcribing video files #9898

Merged
merged 2 commits into from Dec 19, 2023

Conversation

viren-nadkarni
Copy link
Member

@viren-nadkarni viren-nadkarni commented Dec 18, 2023

Motivation

This PR adds support for files with video streams to Transcribe. The list of supported media files remain the same. Like before, only the audio stream matters to Transcribe but now it differentiates the streams better for metadata extraction.

Tests

Updates existing test_transcribe_supported_media_formats to add new parameters with video file and its expected speech content. Confirmed passing.

The new media files are of different container formats: mkv and mp4, however the underlying audio stream codec is same: aac.


tests/aws/services/transcribe/test_transcribe.py::TestTranscribe::test_transcribe_supported_media_formats[../../files/en-us_video.mkv-one of the most vital] 
-------------------------------- live log setup --------------------------------
2023-12-18T11:34:59.923  INFO --- [  asgi_gw_12] localstack.request.aws     : AWS s3.CreateBucket => 200
-------------------------------- live log call ---------------------------------
2023-12-18T11:34:59.952  INFO --- [   asgi_gw_2] localstack.request.aws     : AWS s3.PutObject => 200
2023-12-18T11:34:59.963 DEBUG --- [   asgi_gw_8] localstack.utils.threads   : start_thread called without providing a custom name
2023-12-18T11:34:59.964 DEBUG --- [_run_transcription_job-functhread10766] localstack.services.transcribe.provider : Starting transcription: test-transcribe-64784b50
2023-12-18T11:34:59.965  INFO --- [   asgi_gw_8] localstack.request.aws     : AWS transcribe.StartTranscriptionJob => 200
2023-12-18T11:34:59.983  INFO --- [   asgi_gw_5] localstack.request.aws     : AWS transcribe.GetTranscriptionJob => 200
2023-12-18T11:35:00.003 DEBUG --- [_run_transcription_job-functhread10766] localstack.packages.api    : Installation of ffmpeg skipped (already installed).
2023-12-18T11:35:00.004 DEBUG --- [_run_transcription_job-functhread10766] localstack.packages.api    : Performing runtime setup for already installed package.
2023-12-18T11:35:00.004 DEBUG --- [_run_transcription_job-functhread10766] localstack.services.transcribe.provider : Determining media format
2023-12-18T11:35:00.005 DEBUG --- [_run_transcription_job-functhread10766] localstack.utils.run       : Executing command: /var/lib/localstack/lib/ffmpeg/4.4.1/ffmpeg-4.4.1-amd64-static/ffprobe -show_streams -show_format -print_format json -hide_banner -v error /tmp/tmpg1fh87j_
2023-12-18T11:35:00.012 DEBUG --- [_run_transcription_job-functhread10766] localstack.services.transcribe.provider : Media format detected as: matroska,webm
2023-12-18T11:35:00.013 DEBUG --- [_run_transcription_job-functhread10766] localstack.services.transcribe.provider : Transcoding media to wav
2023-12-18T11:35:00.013 DEBUG --- [_run_transcription_job-functhread10766] localstack.utils.run       : Executing command: /var/lib/localstack/lib/ffmpeg/4.4.1/ffmpeg-4.4.1-amd64-static/ffmpeg -y -nostdin -loglevel quiet -i '/tmp/tmpg1fh87j_' -ar 16000 -ac 1 '/tmp/tmp9p12a___.wav'
2023-12-18T11:35:01.433  INFO --- [_run_transcription_job-functhread10766] localstack.services.transcribe.provider : Transcription job completed: test-transcribe-64784b50
2023-12-18T11:35:01.999  INFO --- [   asgi_gw_6] localstack.request.aws     : AWS transcribe.GetTranscriptionJob => 200
2023-12-18T11:35:02.010  INFO --- [  asgi_gw_13] localstack.request.aws     : AWS s3.GetObject => 200
PASSED

Related

Closes: #9812

@viren-nadkarni viren-nadkarni self-assigned this Dec 18, 2023
@viren-nadkarni viren-nadkarni added the semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases label Dec 18, 2023
@coveralls
Copy link

coveralls commented Dec 18, 2023

Coverage Status

coverage: 84.044% (+0.002%) from 84.042%
when pulling da3d1c8 on transcribe-video-support
into dd9b2b0 on master.

@viren-nadkarni viren-nadkarni marked this pull request as ready for review December 18, 2023 11:46
Copy link

github-actions bot commented Dec 18, 2023

LocalStack Community integration with Pro

       2 files  ±0         2 suites  ±0   1h 10m 52s ⏱️ -48s
2 410 tests +2  2 182 ✔️ +2  228 💤 ±0  0 ±0 
2 411 runs  +2  2 182 ✔️ +2  229 💤 ±0  0 ±0 

Results for commit da3d1c8. ± Comparison against base commit dd9b2b0.

This pull request removes 6 and adds 8 tests. Note that renamed tests count towards both.
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.amr]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.flac]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.mp3]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.mp4]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.ogg]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.webm]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.amr-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.flac-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.mp3-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.mp4-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.ogg-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-gb.webm-hello my name is]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-us_video.mkv-one of the most vital]
tests.aws.services.transcribe.test_transcribe.TestTranscribe ‑ test_transcribe_supported_media_formats[../../files/en-us_video.mp4-one of the most vital]

♻️ This comment has been updated with latest results.

("../../files/en-gb.mp4", "hello my name is"),
("../../files/en-gb.ogg", "hello my name is"),
("../../files/en-gb.webm", "hello my name is"),
("../../files/en-us_video.mkv", "one of the most vital"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add a test case for mp4 video file, as stated in the github issue?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in da3d1c8

@sannya-singal
Copy link
Contributor

sannya-singal commented Dec 18, 2023

Just a minor nit for adding a mp4 video file test case, otherwise LGTM 🚀

@viren-nadkarni viren-nadkarni merged commit ef6d7ac into master Dec 19, 2023
30 checks passed
@viren-nadkarni viren-nadkarni deleted the transcribe-video-support branch December 19, 2023 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver: minor Non-breaking changes which can be included in minor releases, but not in patch releases
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Transcribe service always fails with sample_rate error for video codecs
3 participants