feature/support-processing-m3u8-video-playlist-uris #188
Conversation
evamaxfield
added
enhancement
Codecov Report
@@ Coverage Diff @@
## main #188 +/- ##
==========================================
+ Coverage 94.55% 94.60% +0.04%
==========================================
Files 50 50
Lines 2607 2630 +23
==========================================
+ Hits 2465 2488 +23
Misses 142 142
Continue to review full report at Codecov.
|
|
I have two actions running on my dev infrastructure to test:
if both succeed I will link to their page on my dev infra website and then likely merge after approvals! 🎉 |
| m3u8_To_MP4.download( | ||
| uri, | ||
| mp4_file_dir=dst.parent, | ||
| mp4_file_name=save_name, | ||
| ) |
There was a problem hiding this comment.
This downloads to an mp4 file in local storage - could that be a problem on GitHub actions with limited storage?
There was a problem hiding this comment.
We store the video on the disk regardless of m3u8 or native mp4. This slightly duplicates storage because it stores chunks and the converted but 🤷 we have never run into problems. Even when processing 8 events at a time
There was a problem hiding this comment.
I think the limit is 500mb. Are the temp files deleted after they have been hosted, that would help?
There was a problem hiding this comment.
500Mb wouldn't even be possible to store a docker image on.
I think it's 14GB: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources
There was a problem hiding this comment.
The 500MB must be for logs and artifact files then 😕
|
Both the normal event gather and the manual process special event succeeded https://jacksonmaxfield.github.io/cdp-dev/#/events/0913ecd21d43 @tohuynh you happy with this PR if so I think I can merge. 🎉 |
yes |
evamaxfield
deleted the
feature/support-processing-m3u8-video-playlist-uris
branch
Link to Relevant Issue
CouncilDataProject/cdp-scrapers#96 (comment)
Description of Changes
Include a description of the proposed changes.
Allows processing of m3u8 URIs.
What will happen is during
resource_copywe will download and convert the m3u8 playlist to an mp4. Then during determining the video host, we check the original session video uri and see if it ends with".m3u8"and if so, we host the already converted and saved locally m3u8 file.