Embed Youtube, Images into Posts

[christicehurst]

If Flarum wants to be modern than this a needed featured. Wordpress has it and it's a drag and drop for Facebook as well. Currently I can;t get my Youtube shows to appear so people can't watch from the forum itself.

[JoshyPHP]

For reference, third-party content from supported sites can be embedded via the MediaEmbed plugin. Links to static images can be transformed by the Autoimage plugin.

[brainscript]

@tobscure @franzliedke Just implement OEmbed. With this you can paste a link, for example from YouTube, and it gets replaced by a video player.

[JoshyPHP]

@brainscript Out of curiosity, is that just a passing thought or is it an idea that you've already fleshed out in your mind? If it's the latter, how would you implement it and how would you select OEmbed providers and ?

[brainscript]

@s9e Maybe the DEVs could look into the source code of HumHub, they implemented it very well.

[d1a8lo24]

So the concept is once a person enters a URL on the editor/reply box what ever you want to call it, and it sends that information through the API to get process, figure out what type of URL it is if you need to process it for video or a site. Now we can do a quick call using CURL to either get the raw html and process it or use a service like embedly an get a quick reply with the information that is needed.

So whats needed; the title from the html meta data, the description and if not available a paragraph from the body, some of the images from the page if they exist and they should be a certain size.

Get the response quickly and present it in a highlighted box of whatever color on the post or reply and if possible have a small thumbnail chooser to choose from the images that were found.

Okay so here a small contribution for an extension to use embedly, I am not good at JavaScript or Ajax calls html design etc... so that I will leave it to the expert in that department but I have created some crawlers in the past to get information from sites so here is a very basic code to start things off

So the following code is from one of my projects and it just follows very basic instructions from embedly's api documentation.

function embedly_extract($url = FALSE)
{
    // Here we should check if something is being pass at the
    // same time it should check if its a valid URL I am hoping such
    // validation should already be in the application for now just 
    // a simple is there something there.
    if ( ! $url )
    {
        return FALSE;
    }

    // Now lets check if there is an http or https 
    // if there is none lets add it.
    if (substr($url, 0, 7) != 'http://' && substr($url, 0, 8) != 'https://')
    {
        $url = 'http://'.$url;
    }

    // Now lets create the URL to embedlys API to call it and ask it to 
    // return a response in JSON format and to get any pictures with
    // a maximum with of 800px. 
    $embedly_url = "http://api.embed.ly/1/extract?key=$embedly_api_key&url=".urlencode($url)."&maxwidth=800&format=json";

    // Yes I saw Guzzle somewhere in flarum's core 
    // but we are just doing a quick get using CURL
    // With just a few lines you will get your response
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $response = curl_exec($ch);
    curl_close($ch);

    // yes if there is no response return FALSE
    if (empty($response))
    {
        return FALSE;
    }

    // if there was lets decode the JSON response 
    // and create an Object/Array 
            $response = json_decode($response);

    // If we have an error again lets return FALSE
    if ($response->type == 'error')
    {
        return FALSE;
    }

    // Now if everything worked fine lets create an object to return
    // it and be able to use that information somewhere else. 
    $row               = new stdClass();
    $row->original_url = $response->original_url;
    $row->title        = $response->title;
    $row->domain       = $response->provider_display;
    $row->description  = $response->description;

    if ( ! empty($response->images) )
    {
        foreach ($response->images as $i)
        {
            $image          = new stdClass();

            $image->url     = $i->url;
            $image->width   = $i->width;
            $image->height  = $i->height;

            $row->images[] = $image;
        }
    }

    return $row;
}

Remember this will get you the RAW data so make sure to use Flarum's tools to do the rest like image manipulation, text formatting etc...

Hope this will get things started.

[JoshyPHP]

The problem with OEmbed is not implementing the protocol, afaik it's fairly simple. The problem is that the OEmbed is clunky and hard to use properly. You have to contact a third party service and it will return some HTML, which leads to the two biggest issues.

When do you retrieve the HTML? Not at posting time, because it's not supposed to be cached indefinitely otherwise you'd get stale HTML whenever the external site changes. You have to do it at rendering time, which adds a lot of latency to the page and you still have to cache the result at least temporarily otherwise a popular post with a couple of OEmbeds would DDoS your server à la Slowloris.

What do you do with the HTML anyway? You still have to filter it. Here are OEmbed's recommandation for XSS protection/mitigation:

When a consumer displays HTML (as with video embeds), there's a vector for XSS attacks from the provider. To avoid this, it is recommended that consumers display the HTML in an iframe, hosted from another domain. This ensures that the HTML cannot access cookies from the consumer domain.

I don't think people would want to setup a quarantined (sub)domain just to display videos, right?

I think that XenForo got it mostly right with their Media Sites functionality.

[d1a8lo24]

Here are some very basic functions to accomplish some quick data mining just using curl and simple_html_dom to parse the raw html.

Again this is very basic but gets most of the hard work done, from there all there really needs to do is a little more processing with the content, specially with the emoji characters and other special characters that may be on the content.

As far as video OEmbed should do the work.

include_once('./path_to_the_file/Simple_html_dom.php');

// Lets try to remove any HTML code
// Yes I know the strip_tags function exist
// But I have had better results doing this way.
function remove_html($html_content = FALSE) 
{
    if ( ! $html_content)
    {
        return FALSE;
    }

    // create spaces between tags, to remove them with out removing any text
    // Find the rest of the tags and some special characters
    $html_content = preg_replace("/<[\w ]+>/", "\\0 ", $html_content);
    $html_content = preg_replace("/<\/[\w ]+>/", "\\0 ", $html_content);
    $html_content = preg_replace('/<[^>]*>/', ' ', $html_content);
    $html_content = preg_replace("/&nbsp;/", " ", $html_content);
    // remove double spacing
    $html_content = preg_replace("/\s+/", " ", $html_content);        

    // Lets check again if we still have some content
    // you never know.
    if (empty($html_content))
    {
        return FALSE;
    }
    // Lets not forget to TRIM 
    return trim($html_content);
}

// Lets go get that info!!!
function get_html($url = FALSE)
{
    // Here we should check if something is being pass at the
    // same time it should check if its a valid URL I am hoping such
    // validation should already be in the application for now just 
    // a simple is there something there.
    if ( ! $url )
    {
        return FALSE;
    }

    // just doing a quick get using CURL
    // With just a few lines you will get the raw html
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $raw_html = curl_exec($ch);
    curl_close($ch);

    if (empty($raw_html))
    {
        return FALSE
    }

    // I like to use tidy to correct the raw html
    // to better manipulate the DOM
    // but we can always do a check if is available
    if (class_exists('tidy')) 
    {
        $config = array(
                       'indent'         => true,
                       'output-xhtml'   => true,
                       'wrap'           => 200
                   );

        $tidy = new tidy;
        $tidy->parseString($raw_html, $config, 'utf8');
        $tidy->cleanRepair();
        $raw_html = $tidy;
    }

    // Now lets create the object that we are going to return
    $row = new stdClass();
    $row->original_url = $url;
    $row->title = '';
    $row->description = '';
    $row->images = array();

    // Will be using simple html dom 
    // to parse the Raw HTML
    $dom = new Simple_html_dom();

    $raw_html_array = $dom->load($raw_html);

    // Lets find what we need and remove whatever 
    // HTML is there.
    $tag_title  = array();
    @$tag_title = $raw_html_array->find('title');
    if (! empty($tag_title) )
    {
        $row->title = remove_html($tag_title);
    }

    // If there is no description find all the P tags
    // and put them together if the new text is larger than
    // 200 characters lets cut the string down
    $tag_desc  = array();
    @$tag_desc = $raw_html_array->find('head meta[name=description]', 0);
    if ( ! empty($tag_desc) )
    {
        $row->description = remove_html($tag_desc->content);
    } else {
        @$tag_desc = $raw_html_array->find('body p');

        if ( ! empty($tag_desc) )
        {
            $tag_desc = remove_html(implode(' ', $tag_desc));

            if (strlen($tag_desc) > 200)
            {
                $tag_desc = trim(substr($tag_desc, 0, 200)).'&#8230;';
            }
            $row->description = $tag_desc;
        }
    }

    // Now lets look for all the images
    // on the page and create an array with their 
    // URL to use it.
    $images = array();
    @$images = $raw_html_array->find('img');                
    if ( ! empty($images) )
    {
        foreach ($images as $image)
        {
            if ( ! preg_match('/^https?:?/', $image->src)  )
            {
                $image->src = 'http:'.$image->src;
            }
            $row->images[] = $image->src;
        }
    }
    return $row;
}

[tobyzerner]

I think @JoshyPHP's MediaEmbed extension covers this well enough?

[d1a8lo24]

@tobscure yes @JoshyPHP mediaembed does a great job for embedding media and I really didn't read the whole thing here as I was directed from a discussion on the forum about quick info boxes for links and I just notice that this was mainly for media embedding.

I know many forums and social networks have this functionality where you put a link to an article or website and it grabs quick info to create a small informational snippet and if available an image or favico, or the website's app image.

I think discourse calls it "OneBox" and like I said many other applications have it is nothing new.

Anyway I don't know if @JoshyPHP medianembed covers this and if its implemented in Flarum?