YouTube urls with `/embed/` in the url fail to embed

[probabble]

Issue Summary

Using an Embed block in a streamfield, we are able to embed videos from youtube.com like this https://youtu.be/2MW6CrxScLk or like this https://www.youtube.com/watch?v=2MW6CrxScLk but not like this https://youtube.com/embed/2MW6CrxScLk

Steps to Reproduce

Grab a youtube url by clicking on a link that says "Embed".

Technical details

• Python version: 3.4
• Django version: 1.11.
• Wagtail version: 1.13.1
• Browser version: Chrome 62 (but it's a backend thing)

[lb-]

I can confirm this unexpected behaviour, all that is rendered is an empty div, tested on the latest master branch (prep for 2.0).

<div class="block-embed_block"></div>

Even the help text suggests a URL in this format https://www.youtube.com/embed/SGJFWirQ3ks

I cannot really get to the bottom of the cause, the closest I got was that this type of URL does not seem to be matching in the oembed_providers.py. I am not a regex pro but it seems that the /emebd/ URL format is not explicitly listed there.

wagtail/wagtail/embeds/oembed_providers.py

Lines 16 to 29 in 26d9fb4

	youtube = {
	"endpoint": "http://www.youtube.com/oembed",
	"urls": [
	"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/watch.+$",
	"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/v/.+$",
	"^http(?:s)?://youtu\\.be/.+$",
	"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/user/.+$",
	"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/[^#?/]+#[^#?/]+/.+$",
	"^http(?:s)?://m\\.youtube\\.com/index.+$",
	"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/profile.+$",
	"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/view_play_list.+$",
	"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/playlist.+$"
	],
	}

When the URL cannot be matched it throws EmbedUnsupportedProviderException from within the get_embed function, but this is intentionally silently ignored.
https://github.com/wagtail/wagtail/blob/master/wagtail/embeds/embeds.py#L22

I could be way off here but this is as far as I got investigating this one, but definitely appears to be an issue.

[gasman]

Even the help text suggests a URL in this format https://www.youtube.com/embed/SGJFWirQ3ks

@lb- Where are you seeing that? As far as I can see, all references to youtube.com in the codebase use youtube.com/watch/ URLs (with the exception of test code that's checking the final embed output).

I don't think this is a bug as such, but I've seen people caught out by it before - it's not necessarily obvious that the URL we want is the one for the actual page on Youtube, and that as an editor you don't have to go through Youtube's own embedding mechanism. (In some cases people have tried pasting in the full embed HTML snippet, not just the URL...) So, I'm in favour of making the URL matching as permissive as possible.

[lb-]

@gasman my apologies, this is actually something specific to the bakerydemo (code below), nonetheless I think we should try to handle this URL format as it seems a reasonable assumption by users/developers.

I am right in thinking that we will just need to add an appropriate additional youtube regex?

class BaseStreamBlock(StreamBlock):
    """
    Define the custom blocks that `StreamField` will utilize
    """
    heading_block = HeadingBlock()
    paragraph_block = RichTextBlock(
        icon="fa-paragraph",
        template="blocks/paragraph_block.html"
    )
    image_block = ImageBlock()
    block_quote = BlockQuote()
    embed_block = EmbedBlock(
        help_text='Insert an embed URL e.g https://www.youtube.com/embed/SGJFWirQ3ks',
        icon="fa-s15",
        template="blocks/embed_block.html")

https://github.com/wagtail/bakerydemo/blob/c60a6129ee5403ccd9c9d3ea7e61159ef8340c12/bakerydemo/base/blocks.py#L53-L67

[gasman]

Yep, try adding

"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/embed.+$",

to the list in oembed_providers.py. Can't say for sure that it will work - we pass the matched URL on to Youtube's oembed endpoint to generate the embed snippet, so it's up to the handler at Youtube's end whether or not they recognise /embed/... as a valid URL for that purpose. (If not, I guess we'd need to implement some extra logic to rewrite it into a URL they do recognise.)

[lb-]

I have the URL being accepted locally but the embed is still not working.

It appears that the https://www.youtube.com/oembed endpoint will not accept YouTube URLs with the /embed/ format.

Works

• https://www.youtube.com/oembed?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DeYW4J6FyyZU&format=json
• Provided URL: https://www.youtube.com/watch?v=eYW4J6FyyZU

Does not work

• https://www.youtube.com/oembed?url=https%3A%2F%2Fwww.youtube.com%2Fembed%2FeYW4J6FyyZU&format=json
• https://www.youtube.com/embed/eYW4J6FyyZU

Potential solution

This might be a bit heavy handed but we could add the ability to set templates to settings for oembed_providers.

eg.

youtube = {
    "endpoint": "http://www.youtube.com/oembed",
    "urls": [
        "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/watch.+$",
        "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/v/.+$",
        #...
        "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/playlist.+$",
    ],
    "templates": [
        ("^(?P<base>http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com)/embed/(?P<ref>.+)$", '\g<base>/?v=\g<ref>')
    ],
}

If the template is present, when parsing through the urls we would also check for matches against the template, but instead of using the original url we would expand -match.expand(template) .

This would open a way to solve things like people pasting in the entire embed iframe code from YouTube. It adds a fair bit of complexity though but is quite backwards compatible.

Otherwise we should close this ticket, maybe updating some doc/file explaining why that type of URL is not supported.

[lb-]

I have a working version (needs tests/docs before I put in a PR) of the templates idea.

See the revised oembed code here:
https://github.com/lb-/wagtail/blob/4127_embed_youtube_url/wagtail/embeds/finders/oembed.py

A working template that here:
https://github.com/lb-/wagtail/blob/e7b8315f80a68b7b07ff83dce43ee20fe5b48f52/wagtail/embeds/oembed_providers.py#L29-L31

Would love some feedback if it is worth adding this feature, happy to add tests / document this correctly.

[maxwaiyaki]

Can Anyone help me figure out why this particular video gives "Cannot find an embed for this URL" error when trying to embed it while other youtube videos embed just fine?
Video - https://www.youtube.com/watch?v=SLCzF0C3LZk

[gasman]

@maxwaiyaki The uploader of the video has disabled embedding. From the youtube page, click "Share" then "Embed" and you'll see "Embedding disabled by request".

[maxwaiyaki]

@gasman Thank you.

[maxwaiyaki]

You can check this Magazine Out. Build with Wagtail - https://mag.mookh.com/