YouTube urls with `/embed/` in the url fail to embed #4127

probabble · 2017-12-13T18:50:41Z

Issue Summary

Using an Embed block in a streamfield, we are able to embed videos from youtube.com like this https://youtu.be/2MW6CrxScLk or like this https://www.youtube.com/watch?v=2MW6CrxScLk but not like this https://youtube.com/embed/2MW6CrxScLk

Steps to Reproduce

Grab a youtube url by clicking on a link that says "Embed".

Technical details

Python version: 3.4
Django version: 1.11.
Wagtail version: 1.13.1
Browser version: Chrome 62 (but it's a backend thing)

The text was updated successfully, but these errors were encountered:

lb- · 2017-12-15T07:31:40Z

I can confirm this unexpected behaviour, all that is rendered is an empty div, tested on the latest master branch (prep for 2.0).

<div class="block-embed_block"></div>

Even the help text suggests a URL in this format https://www.youtube.com/embed/SGJFWirQ3ks

I cannot really get to the bottom of the cause, the closest I got was that this type of URL does not seem to be matching in the oembed_providers.py. I am not a regex pro but it seems that the /emebd/ URL format is not explicitly listed there.

wagtail/wagtail/embeds/oembed_providers.py

Lines 16 to 29 in 26d9fb4

    
           youtube = { 
        
               "endpoint": "http://www.youtube.com/oembed", 
        
               "urls": [ 
        
                   "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/watch.+$", 
        
                   "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/v/.+$", 
        
                   "^http(?:s)?://youtu\\.be/.+$", 
        
                   "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/user/.+$", 
        
                   "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/[^#?/]+#[^#?/]+/.+$", 
        
                   "^http(?:s)?://m\\.youtube\\.com/index.+$", 
        
                   "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/profile.+$", 
        
                   "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/view_play_list.+$", 
        
                   "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/playlist.+$" 
        
               ], 
        
           }

When the URL cannot be matched it throws EmbedUnsupportedProviderException from within the get_embed function, but this is intentionally silently ignored.
https://github.com/wagtail/wagtail/blob/master/wagtail/embeds/embeds.py#L22

I could be way off here but this is as far as I got investigating this one, but definitely appears to be an issue.

gasman · 2017-12-15T10:51:17Z

Even the help text suggests a URL in this format https://www.youtube.com/embed/SGJFWirQ3ks

@lb- Where are you seeing that? As far as I can see, all references to youtube.com in the codebase use youtube.com/watch/ URLs (with the exception of test code that's checking the final embed output).

I don't think this is a bug as such, but I've seen people caught out by it before - it's not necessarily obvious that the URL we want is the one for the actual page on Youtube, and that as an editor you don't have to go through Youtube's own embedding mechanism. (In some cases people have tried pasting in the full embed HTML snippet, not just the URL...) So, I'm in favour of making the URL matching as permissive as possible.

lb- · 2017-12-15T11:00:38Z

@gasman my apologies, this is actually something specific to the bakerydemo (code below), nonetheless I think we should try to handle this URL format as it seems a reasonable assumption by users/developers.

I am right in thinking that we will just need to add an appropriate additional youtube regex?

class BaseStreamBlock(StreamBlock):
    """
    Define the custom blocks that `StreamField` will utilize
    """
    heading_block = HeadingBlock()
    paragraph_block = RichTextBlock(
        icon="fa-paragraph",
        template="blocks/paragraph_block.html"
    )
    image_block = ImageBlock()
    block_quote = BlockQuote()
    embed_block = EmbedBlock(
        help_text='Insert an embed URL e.g https://www.youtube.com/embed/SGJFWirQ3ks',
        icon="fa-s15",
        template="blocks/embed_block.html")

https://github.com/wagtail/bakerydemo/blob/c60a6129ee5403ccd9c9d3ea7e61159ef8340c12/bakerydemo/base/blocks.py#L53-L67

gasman · 2017-12-15T11:13:20Z

Yep, try adding

"^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/embed.+$",

to the list in oembed_providers.py. Can't say for sure that it will work - we pass the matched URL on to Youtube's oembed endpoint to generate the embed snippet, so it's up to the handler at Youtube's end whether or not they recognise /embed/... as a valid URL for that purpose. (If not, I guess we'd need to implement some extra logic to rewrite it into a URL they do recognise.)

lb- · 2017-12-17T01:22:31Z

I have the URL being accepted locally but the embed is still not working.

It appears that the https://www.youtube.com/oembed endpoint will not accept YouTube URLs with the /embed/ format.

Works

Does not work

Potential solution

This might be a bit heavy handed but we could add the ability to set templates to settings for oembed_providers.

eg.

youtube = {
    "endpoint": "http://www.youtube.com/oembed",
    "urls": [
        "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/watch.+$",
        "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/v/.+$",
        #...
        "^http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com/playlist.+$",
    ],
    "templates": [
        ("^(?P<base>http(?:s)?://(?:[-\\w]+\\.)?youtube\\.com)/embed/(?P<ref>.+)$", '\g<base>/?v=\g<ref>')
    ],
}

If the template is present, when parsing through the urls we would also check for matches against the template, but instead of using the original url we would expand -match.expand(template) .

This would open a way to solve things like people pasting in the entire embed iframe code from YouTube. It adds a fair bit of complexity though but is quite backwards compatible.

Otherwise we should close this ticket, maybe updating some doc/file explaining why that type of URL is not supported.

lb- · 2017-12-17T02:07:44Z

I have a working version (needs tests/docs before I put in a PR) of the templates idea.

See the revised oembed code here:
https://github.com/lb-/wagtail/blob/4127_embed_youtube_url/wagtail/embeds/finders/oembed.py

A working template that here:
https://github.com/lb-/wagtail/blob/e7b8315f80a68b7b07ff83dce43ee20fe5b48f52/wagtail/embeds/oembed_providers.py#L29-L31

Would love some feedback if it is worth adding this feature, happy to add tests / document this correctly.

maxwaiyaki · 2019-05-10T15:38:39Z

Can Anyone help me figure out why this particular video gives "Cannot find an embed for this URL" error when trying to embed it while other youtube videos embed just fine?
Video - https://www.youtube.com/watch?v=SLCzF0C3LZk

gasman · 2019-05-10T15:42:37Z

@maxwaiyaki The uploader of the video has disabled embedding. From the youtube page, click "Share" then "Embed" and you'll see "Embedding disabled by request".

maxwaiyaki · 2019-05-16T11:39:43Z

@gasman Thank you.

maxwaiyaki · 2019-05-16T11:41:35Z

You can check this Magazine Out. Build with Wagtail - https://mag.mookh.com/

thibaudcolas · 2019-11-28T15:39:00Z

I have stumbled upon something similar recently where a URL that has a video id and a playlist like: https://www.youtube.com/watch?v=tdB1I_gSCeY&list=PLEyaio0l1qoEIUFM9bnRKoN6VKEUOdxAn

… renders as a player showing just the playlist: https://www.youtube.com/embed/videoseries?list=PLEyaio0l1qoEIUFM9bnRKoN6VKEUOdxAn. As a user of the CMS I was expecting to be embedding the video I can see on YouTube from that URL rather than the first one in the playlist.

This behavior comes from YouTube’s oembed response for that URL, https://www.youtube.com/oembed?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DtdB1I_gSCeY%26list%3DPLEyaio0l1qoEIUFM9bnRKoN6VKEUOdxAn&format=json.

Like @gasman says (#4127 (comment)) it makes me wonder whether it might be valuable to have extra logic for this scenario – it’s very confusing that the resulting embed is different from the YouTube UI.

Note: the workaround in this scenario is of course to drop the playlist parameter from the query string, so https://www.youtube.com/watch?v=tdB1I_gSCeY.

Edit: also related, https://wagtail.io/blog/fixing-youtubes-oembed-implementation-custom-finder-class/.

cjoshmartin · 2022-03-18T03:39:40Z

Hello y'all I was looking for answer for days but I found it here:
https://docs.wagtail.org/en/stable/topics/writing_templates.html#responsive-embeds

for some reason it did not show up on google 🤔

Short of it is:

in your base settings file add this WAGTAILEMBEDS_RESPONSIVE_HTML = True
In the template file you are using the embed with add these styles:

.responsive-object {
    position: relative;
}

.responsive-object iframe,
.responsive-object object,
.responsive-object embed {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    height: 100%;
}

and BOOM responsive embeddeds

lb- added component:Streamfield component:Embeds type:Bug labels Dec 15, 2017

gasman added type:Enhancement and removed type:Bug labels Dec 15, 2017

YouTube urls with `/embed/` in the url fail to embed #4127

YouTube urls with `/embed/` in the url fail to embed #4127

probabble commented Dec 13, 2017 •

edited

lb- commented Dec 15, 2017

gasman commented Dec 15, 2017

lb- commented Dec 15, 2017

gasman commented Dec 15, 2017

lb- commented Dec 17, 2017

lb- commented Dec 17, 2017

maxwaiyaki commented May 10, 2019 •

edited

gasman commented May 10, 2019

maxwaiyaki commented May 16, 2019

maxwaiyaki commented May 16, 2019 •

edited

thibaudcolas commented Nov 28, 2019 •

edited

cjoshmartin commented Mar 18, 2022 •

edited

YouTube urls with /embed/ in the url fail to embed #4127

YouTube urls with /embed/ in the url fail to embed #4127

Comments

probabble commented Dec 13, 2017 • edited

Issue Summary

Steps to Reproduce

Technical details

lb- commented Dec 15, 2017

gasman commented Dec 15, 2017

lb- commented Dec 15, 2017

gasman commented Dec 15, 2017

lb- commented Dec 17, 2017

lb- commented Dec 17, 2017

maxwaiyaki commented May 10, 2019 • edited

gasman commented May 10, 2019

maxwaiyaki commented May 16, 2019

maxwaiyaki commented May 16, 2019 • edited

thibaudcolas commented Nov 28, 2019 • edited

cjoshmartin commented Mar 18, 2022 • edited

YouTube urls with `/embed/` in the url fail to embed #4127

YouTube urls with `/embed/` in the url fail to embed #4127

probabble commented Dec 13, 2017 •

edited

maxwaiyaki commented May 10, 2019 •

edited

maxwaiyaki commented May 16, 2019 •

edited

thibaudcolas commented Nov 28, 2019 •

edited

cjoshmartin commented Mar 18, 2022 •

edited