Call Mist to transcode input manifest and push renditions to S3 #41

thomshutt · 2022-09-09T15:06:55Z

curl --location --request POST 'http://localhost:4949/api/transcode/file' \ 
--header 'Authorization: Bearer IAmAuthorized' \ 
--header 'Content-Type: application/json' \ 
--data-raw '{
  "source_location": "s3+https://REDACTED:REDACTED@storage.googleapis.com/alexk-dms-upload-test/avsample.mp4",
  "callback_url": "https://webhook.site/YOUR-UNIQUE-URL-HERE",
  "manifestID": "somestream123",
  "profiles": [
        {
        "name": "360p",
          "width": 640,
          "height": 360,
          "bitrate": 200000,
          "fps": 24
        }
    ], 
  "verificationFreq": 1}'

This PR still needs some tidying, async transcode (i.e HTTP call shouldn't wait for that) added back in and lots of unit tests - ideally mostly at the function rather than HTTP level since we don't have a good story around mocking MistProcLivepeer yet.

config/config.go

handlers/handlers_test.go

handlers/misttriggers/live_tracklist.go

config/config.go

handlers/misttriggers/live_tracklist.go

handlers/transcode.go

handlers/handlers_test.go

cache: - Add transcoding streams to system cache misttriggers: - add handlers for LIVE_TRACK_LIST and PUSH_END triggers transcode: - add handler to launch MistProcLivepeer to kickoff transcode flow

This will be updated later to test a specific portion of the pipeline instead of an e2e test triggered by an html endpoint.

…add studio transcode support * exec-ing MistProcLivepeer and then opening pipes was causing it to sometimes exit prematurely. I used a similar method but refactored to fix this issue. * Added support for using one of two methods of transcode: 1. livepeer.studio w/ developer access-token: this uses production BOT nodes to perform the transcoding. 2. local hardcoded broadcaster: this uses a local broadcaster running in bg to perform the transcoding. These additional parameters were added to the json schema for the incoming transcode requests.

AlexKordic

Well done. Some of my comments may be out of scope of this PR

handlers/misttriggers/live_tracklist.go

handlers/transcode.go

yondonfu

Needs a go mod tidy.

handlers/misttriggers/live_tracklist.go

cache: - Add transcoding streams to system cache misttriggers: - add handlers for LIVE_TRACK_LIST and PUSH_END triggers transcode: - add handler to launch MistProcLivepeer to kickoff transcode flow

This will be updated later to test a specific portion of the pipeline instead of an e2e test triggered by an html endpoint.

…add studio transcode support * exec-ing MistProcLivepeer and then opening pipes was causing it to sometimes exit prematurely. I used a similar method but refactored to fix this issue. * Added support for using one of two methods of transcode: 1. livepeer.studio w/ developer access-token: this uses production BOT nodes to perform the transcoding. 2. local hardcoded broadcaster: this uses a local broadcaster running in bg to perform the transcoding. These additional parameters were added to the json schema for the incoming transcode requests.

codecov · 2022-09-29T06:42:49Z

Codecov Report

Merging #41 (d3b09bf) into main (7689ffc) will decrease coverage by 12.56932%.
The diff coverage is 0.36900%.

@@                 Coverage Diff                  @@
##                main         livepeer/catalyst#41          +/-   ##
====================================================
- Coverage   44.64286%   32.07354%   -12.56932%     
====================================================
  Files             15          16           +1     
  Lines            728         979         +251     
====================================================
- Hits             325         314          -11     
- Misses           364         628         +264     
+ Partials          39          37           -2

Impacted Files	Coverage Δ
cache/transcoding.go	`34.04255% <0.00000%> (-11.67174%)`	⬇️
handlers/misttriggers/live_tracklist.go	`0.00000% <0.00000%> (ø)`
handlers/misttriggers/push_end.go	`0.00000% <0.00000%> (ø)`
handlers/misttriggers/triggers.go	`10.41667% <0.00000%> (-0.10966%)`	⬇️
handlers/transcode.go	`7.85714% <0.00000%> (-36.58730%)`	⬇️
handlers/upload.go	`44.11765% <100.00000%> (-7.16441%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7689ffc...d3b09bf. Read the comment docs.

Impacted Files	Coverage Δ
cache/transcoding.go	`34.04255% <0.00000%> (-11.67174%)`	⬇️
handlers/misttriggers/live_tracklist.go	`0.00000% <0.00000%> (ø)`
handlers/misttriggers/push_end.go	`0.00000% <0.00000%> (ø)`
handlers/misttriggers/triggers.go	`10.41667% <0.00000%> (-0.10966%)`	⬇️
handlers/transcode.go	`7.85714% <0.00000%> (-36.58730%)`	⬇️
handlers/upload.go	`44.11765% <100.00000%> (-7.16441%)`	⬇️

Addressed all comments - github is confused after a force push?

thomshutt marked this pull request as draft September 9, 2022 15:07

thomshutt commented Sep 9, 2022

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

thomshutt commented Sep 9, 2022

View reviewed changes

handlers/handlers_test.go Show resolved Hide resolved

yondonfu reviewed Sep 21, 2022

View reviewed changes

handlers/misttriggers/live_tracklist.go Outdated Show resolved Hide resolved

yondonfu previously requested changes Sep 22, 2022

View reviewed changes

emranemran force-pushed the transcode branch 3 times, most recently from 1e1eed4 to 6cbd30c Compare September 28, 2022 01:53

emranemran requested review from AlexKordic, cyberj0g and yondonfu September 28, 2022 01:54

thomshutt and others added 3 commits September 27, 2022 21:17

transcode: implement initial version of VOD Transcode pipeline

b080512

cache: - Add transcoding streams to system cache misttriggers: - add handlers for LIVE_TRACK_LIST and PUSH_END triggers transcode: - add handler to launch MistProcLivepeer to kickoff transcode flow

tests: temporarily remove problematic test

7e2e2d5

This will be updated later to test a specific portion of the pipeline instead of an e2e test triggered by an html endpoint.

emranemran force-pushed the transcode branch from 6cbd30c to 286b675 Compare September 28, 2022 04:17

emranemran marked this pull request as ready for review September 28, 2022 04:23

emranemran requested a review from victorges September 28, 2022 04:23

emranemran force-pushed the transcode branch from 286b675 to 30f6fcc Compare September 28, 2022 04:28

AlexKordic approved these changes Sep 28, 2022

View reviewed changes

yondonfu reviewed Sep 28, 2022

View reviewed changes

handlers/misttriggers/live_tracklist.go Outdated Show resolved Hide resolved

handlers/misttriggers/live_tracklist.go Show resolved Hide resolved

emranemran and others added 7 commits September 28, 2022 21:33

misttriggers: update LIVE_TRACK_LIST trigger's handler to upload to S3

565e7f7

go mod: update go-tools

c1619d6

transcode: implement initial version of VOD Transcode pipeline

56305f3

cache: - Add transcoding streams to system cache misttriggers: - add handlers for LIVE_TRACK_LIST and PUSH_END triggers transcode: - add handler to launch MistProcLivepeer to kickoff transcode flow

tests: temporarily remove problematic test

94ba7ff

This will be updated later to test a specific portion of the pipeline instead of an e2e test triggered by an html endpoint.

misttriggers: update LIVE_TRACK_LIST trigger's handler to upload to S3

74c43f1

go mod: update go-tools

d6a49f3

emranemran force-pushed the transcode branch from 30f6fcc to c1619d6 Compare September 29, 2022 05:09

emranemran added 2 commits September 28, 2022 22:10

Merge branch 'delete-me' into transcode

0017ec2

github actions: address lint errors

39a03a1

emranemran added 2 commits September 28, 2022 23:10

tests: fix handlers/middleware tests

fee6812

go.mod: use go 1.19

d3b09bf

emranemran merged commit ff9b8ae into main Sep 29, 2022

hjpotter92 deleted the transcode branch September 29, 2022 06:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Call Mist to transcode input manifest and push renditions to S3 #41

Call Mist to transcode input manifest and push renditions to S3 #41

thomshutt commented Sep 9, 2022 •

edited

AlexKordic left a comment

yondonfu left a comment

codecov bot commented Sep 29, 2022

Call Mist to transcode input manifest and push renditions to S3 #41

Call Mist to transcode input manifest and push renditions to S3 #41

Conversation

thomshutt commented Sep 9, 2022 • edited

AlexKordic left a comment

Choose a reason for hiding this comment

yondonfu left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 29, 2022

Codecov Report

thomshutt commented Sep 9, 2022 •

edited