Embedded Videos Not Supported/Detected + Possible Solution

[EnginePod]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.07.26.2. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2016.07.26.2

Before submitting an issue make sure you have:

• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

A bit of background

I'm an avid youtube-dl user and every once in a while I try to download a video.
It sometimes happens that the video is embedded from another site (similar to YouTube), but youtube-dl doesn't detect it. This makes me have to dig through the source code to find the code for the embedded player and enter that into youtube-dl and this gets a bit annoying after 100 videos.

Luckily pretty much all embedded YouTube players are supported, but others are not.
So my question is, why can't we detect those as well? There is already a list of the supported sites so wouldn't it only be a matter of checking those domains in the source code and seeing if they're found in an attribute?

So what the hell can we do?

One way is to manually find and add support for every single embed that youtube-dl currently supports - the other is to automate it like I'm going to describe below:

So let's say that I enter this page: http://unsupported-embed.com/video?id=123 and youtube-dl doesn't support it.
Youtube-dl would then find all the attributes containing URLs (e.g <iframe src="**http://free-video-embeds.com**"></iframe>), add them to an array, loop through the array and check in the supported site list to find the embed. Once and if the embedded player is found it runs a new query for that URL to finally get the video.

Now this could be a bit heavy to run in default so we it could simply be added as a super useful parameter like --extensive-embed-search.

Currently Supported Embed

YouTube
Vimeo

Currently Unsupported Embed

Twitter
Yahoo
LiveLeak
Ebaum's World
USANetwork
...and many more

[yan12125]

Please post real URLs for unsupported sites.

[SeanPM5]

I often run into this issue too, mostly with Brightcove embeds but sometimes with others.

Here is one such example. The embedded (autoplaying) video on this page will not be detected/does not download even though it is hosted on a supported site (The Platform): http://www.usanetwork.com/mrrobot/videos/rami-malek-talks-mr-robot-season20

I have to do a workaround for this example, click the share button in the top right of video player and then in the embed code find a link like this: http://player.theplatform.com/p/HNK2IC/qRe1IXDE_V0F/select/media/guid/2140458126/3052957?ec=f which downloads fine in youtube-dl. But it would be nice if I didn't have to do those extra step(s).

[EnginePod]

@yan12125: There are so many URLs that this happens to. Luckily SeanPM5 filled in with a website, but I'll see if I can gather a large list of pages with embedded players from multiple different sites that we can later use for testing as well.

@SeanPM5: that is exactly what I'm doing every single time there's an unsupported embedded player (almost always except when it's YouTube). I think this currently is a huge weakness in youtube-dl since it's so common to find embedded players.

[yan12125]

@SeanPM5's example looks like another form of ThePlatform embedded video. Indeed youtube-dl should support it but it's currently not.

[EnginePod]

@yan12125: exactly, it should be supported because there are so many pages using embedded players. As I explained in the OP it could be supported:

1. Going through the website source and extracting all the URLs from parameters like src (e.g http://unsupported-embed.com/video?id=123 from <iframe src="http://unsupported-embed.com/video?id=123"></iframe>)
2. Running a new query for each of the URLs found
3. Outputting the result

[yan12125]

Here's a line from the example given by @SeanPM5:

<div data-usa-tve-player="player" data-src="//player.theplatform.com/p/HNK2IC/qRe1IXDE_V0F/select/media/guid/2140458126/3052957" data-id="player" class="pdk-player-wrap"></div>

Scanning <iframe>s sounds reasonable. But do you want scanning data-src as well?

Anyway such a feature depends on #5587

[EnginePod]

@yan12125: I just used the iframe's src as an example since it's so commonly used for embeds.

In the example from @SeanPM5 the URL is in a div's data-src, but this can be supported as well simply by gathering all the data-src to the array and going through them as well (if their URL is in the supported domain list).

Basically what we need for this to work is a list of the domains for all the supported sites.
Once we have that we can just gather the URLs and check if they contain the domains, if they do we have youtube-dl run them and output their results.

[yan12125]

OK but please paste some links. A request without links is not valid.

[EnginePod]

Below I've made a list of a few supported and unsupported embeds.
Let me know if you find any sites that I've missed for the supported embeds.

All of the unsupported embeds would be working using the iFrame src or data-src method.

Currently Supported Embed

YouTube
Vimeo

Currently Unsupported Embed

Twitter
Yahoo
LiveLeak
Ebaum's World
USANetwork
...and many more

[yan12125]

Do you have real URLs? These look like human generated test cases.

[EnginePod]

@yan12125: I uploaded these to webdevout.com/test (something like Pastebin that runs HTML) for the reason of keeping the sources simple just to prove that barebone embeds without any additional HTML don't work.

There are a few "real" URLs in there as well though - have a look at USANetwork or Twitter.

[yan12125]

If this feature is implemented in the future, I'll add some test cases into youtube-dl. All test cases should be real videos.

By the way, the Twitter page does not contain a video - it's just a link.

[EnginePod]

@yan12125: Would that require every single supported site to contain test case for the embeds - because this would defy the point of the automatic extraction?

There is a video on the bottom of the Twitter page.

[yan12125]

Oops the Twitter video is blocked by my Ad blocker (again).

Would that require every single supported site to contain test case for the embeds

Nope. Supporting for sites is added on-demand, that is, once there's a test case.

[yan12125]

Note to myself: The twitter video has the same pattern as the example in #10089

[SeanPM5]

Here is another example, this time with Yahoo.

This embedded video will not work: https://www.yahoo.com/tv/39-agents-of-s-h-i-1430049850908726.html even though it's a supported website and service.

If you click the cog/gear icon in the bottom right of video player and look for link inside the iframe embed code: https://www.yahoo.com/tv/agents-h-e-l-d-121852367.html?format=embed that will download fine with youtube-dl.

[EnginePod]

Added to the list.
It does however seem that the source code is obfuscated so manual support would have to be added for that site.

[jnbdz]

I found a good example for Twitter: https://www.joe.co.uk/news/disgraceful-scenes-in-manchester-as-uk-against-hate-protesters-break-police-lines-129177