Requesting Support For Go90.com

[goldensun87]

Here are my example URLs.

https://www.go90.com/profiles/va_07d47f43a7b04eb5b693252f2bd1086b
https://www.go90.com/profiles/va_506ad122d74647a58282a05aa9d91283
https://www.go90.com/profiles/va_0b3c19a514954b8c8d1ec7c7b62a9e02

[goldensun87]

Thanks for the labels. Will the geo-restriction make it harder or impossible to add support?

[yan12125]

New example from #10215: https://www.go90.com/profiles/va_0a41e9fb10434ab4a483fc2c60e07885. It's still geo-restricted, though.

@goldensun87 Yes. It's still possible but harder.

[triple-j]

Hopefully the following will help the development of an extractor for Go90.

Download Videos from Go90.com

INFO

Go90 seems to use HTTP Live Streaming (aka Apple HLS).

How to download manually

First we need to go to a video page. We'll use this link for our example.

Open the Network inspector in your browser. (You may have to refresh the page to get this to work)

Find the first file with a .m3u8 extension that the browser downloads. This file will contain links to all the possible video resolutions that are available as additional .m3u8 files.

From the previous .m3u8 file, download the .m3u8 file that contains the resolution that you want. This file will contain links to small chunks of the video we are trying to download (and potentially a few Ads).

Now that we have the .m3u8 file we want, we still have to download all the small video chunks and merge them together. Luckily ffmpeg can do all that for us, but first we must remove the Ad segments from the .m3u8 file.

Open the .m3u8 file in a text editor and find a line that looks simular to #UPLYNK-SEGMENT: 9251bafa4a21407098d35f73f835afb6,00000000,ad (the 'numbers' will differ but note that it ends with ,ad). Delete that line and every line following it until you get to another line that starts with #UPLYNK-SEGMENT:. Repeat that for all Ad segments you find.

Now that we have an Ad free .m3u8 file we can have ffmpeg download and merge together all the video chunks.

ffmpeg -i hls-stream-noads.m3u8 -c copy temp-video.ts

Not all video players can open .ts media containers, so we'll want to move the audio and video into a more universal container like .mp4.

ffmpeg -i temp-video.ts -c copy -bsf:a aac_adtstoasc output-video.mp4

We now have a video downloaded from Go90.

[triple-j]

Is there already an extractor that supports HTTP Live Streaming (HLS) that can be used as a starting point or reference?

[yan12125]

@triple-j There's a function self._extract_m3u8_formats for handling m3u8 manifests.

[goldensun87]

@triple-j How exactly do I download the .m3u8 file I want?

And, as a general note to all, Firefox's Downloadhelper extension can detect the HLS streams, but it ultimately cannot download them (possibly because of the geo-restriction).

[triple-j]

@goldensun87 since you have Downloadhelper, use it to copy the URL of the .m3u8 file you want.

On the video's page open Downloadhelper. Hover over the video with the resolution you want, click the arrow icon on the right, and select Copy URL. Then give the URL to a general file downloading program like cURL.

curl -o hls-stream-withads.m3u8 "https://content-ause2.uplynk.com/fe8b89255702489583abadecd8148ff7/g.m3u8?pbs=cb5a5cc9f74942628332a9083396ff63"

Then follow the rest of my instructions above to remove the Ads and download the video.

[triple-j]

Thanks @yan12125, I started looking into youtube-dl's code yesterday and it looks to be easier to implement extractors than I thought.

I started a fork to implement an extractor for Go90. I was able to hack one together to be able to download videos. The problem I've run into is that the .m3u8 file contains Ads at different resolutions than the main video and hlsnative downloads and merges both into the final video file.

Go90 is using upLynk to distribute their files, and upLynk seems to add additional data to differentiate the Ad segment from the main video (#UPLYNK-SEGMENT:). I know I can strip out the Ad segments if I can intercept the .m3u8 file before it gets to hlsnative. Are there any suggestions on how I should do that? Where does youtube-dl select/call the downloader from?

[yan12125]

To remove ads, you can download the m3u8 manifest in the extractor, change its content and pass it to the downloader via encode_data_uri(). A similar use case can be found in youtube_dl/extractor/leeco.py.

Where does youtube-dl select/call the downloader from?

By get_suitable_downloader().

[remitamine]

did you try to remove the query from the url, i checked before an uplynk manifest in abc7news #9551 (comment).

[goldensun87]

So, I could not get curl to work. But, thanks to Downloadhelper, I tried getting youtube-dl to download the m3u8 link. Since I'm still using v2016.08.06, I was surprised that the stream actually downloaded. However, I encountered the same problem that @triple-j did. The ad and the main video got merged into one video, and they are at different resolutions. Is there any way to get youtube-dl to skip the ad?

[triple-j]

Thanks @yan12125, that information is extremely helpful. I now have a good idea where to go from here.

@remitamine, I did some preliminary manual testing without the query string. Opening the file in a text editor, it does look like the Ads are removed however both mplayer and ffmpeg error out saying that the .m3u8 file is invalid. I don't think youtube-dl's hlsnative is as picky, so this may still work. More testing is required.

@goldensun87, I'm working on removing the Ads. In the meantime, see if youtube-dl will download the video if you remove the query string (everything after ?) from the m3u8 link, and let us know if the Ads are still there.

[goldensun87]

@triple-j youtube-dl is not able to download the video with the query string removed. More specifically, it seems ffmpeg errors out without the query string.

[https @ 00000000005d8820] HTTP error 404 Not Found
Unable to open key file https://contentausc2.uplynk.com/check2b=fe8b89255702489583abadecd8148ff7&v=fe8b89255702489583abadecd8148ff7&r=g
[hls,applehttp @ 00000000004d7fe0] Error when loading first segment 'https://stg-ec-norcal-u.uplynk.
com/slices/fe8/98ac1613c7624a8387596b5d5e441064/fe8b89255702489583abadecd8148ff7/G00000000.ts?x=0&si=0'
https://content-ausc2.uplynk.com/fe8b89255702489583abadecd8148ff7/g.m3u8: Invalid data found when processing input

ERROR: ffmpeg exited with code 1

[remitamine]

i can confirm that the AES-128 keys can't be obtained without the pbs in the query.
so there is another way to get a manifest without ads by downloading the manifest without the query and then add the pbs to all urls in the manifest.

[remitamine]

i have added an extractor for uplynk, it will simplify adding support for sites that uses uplynk with pbs param(i found a simular case in viceland).
note that downloading will work only using hls_native downloader with pycrypto.

[yan12125]

note that downloading will work only using hls_native downloader with pycrypto.

I download an example video with youtube-dl -v test:uplynk. It uses ffmpeg, and the result file seems OK. Is there something wrong with ffmpeg?

[remitamine]

it's about:

support for sites that uses uplynk with pbs param

the pbs param should be added to the keys urls to be able to get it's content.

[yan12125]

OK I see.

[RuneMagle]

When are you gonna make it work?

[armerpunkt]

I was looking at the Go90 website today, and it appears they are now using MPEG-DASH rather than HLS.

In the network inspector, search for manifest, and then copy the url (should be on the domain revvel.io). You can then use that url directly with youtube-dl.

[goldensun87]

Yeah, the Go90 website got a major upgrade in the past months. 1080p video is now available, the true video urls are no longer on uplynk, which was a Verizon based server, and videos are no longer broken into multiple fragments and played using a playlist file. Since general Vevo support got broken, I got the idea to look into the source code of videos, in order to find the playlist url. I am going to look into the source code of a Go90 video, and look for the downloadable video links there.

[supermansaga]

Hello, I am using 5.18.2018 version and can't download https://www.go90.com/videos/1PRVCVNCJPK.

However, I managed to download some other clips. Not sure what's the latest holdup. Thx

[Go90] 1PRVCVNCJPK: Downloading JSON metadata
[Go90] 1PRVCVNCJPK: Downloading m3u8 information
[Go90] 1PRVCVNCJPK: Downloading MPD manifest
WARNING: Failed to download MPD manifest: HTTP Error 403: Forbidden
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.