Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
376 lines (340 sloc) 19.6 KB
<!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Zooming Past the Competition</title>
<link href="" type="application/atom+xml" rel="alternate" title="Another Datum Full Atom Feed" />
<!-- Bootstrap Core CSS -->
<link href="" rel="stylesheet">
<!-- Custom CSS -->
<link href="" rel="stylesheet">
<!-- Code highlight color scheme -->
<link href="" rel="stylesheet">
<!-- CSS specified by the user -->
<link href="" rel="stylesheet">
<!-- Custom Fonts -->
<link href="" rel="stylesheet" type="text/css">
<link href=',700,400italic,700italic' rel='stylesheet' type='text/css'>
<link href=',400italic,600italic,700italic,800italic,400,300,600,700,800' rel='stylesheet' type='text/css'>
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src=""></script>
<script src=""></script>
<meta name="description" content="How to create an Augmented Reality app that allows a user to get content recommendations.">
<meta name="author" content="Yoel Zeldes">
<meta name="tags" content="deep learning">
<meta name="tags" content="web">
<meta name="tags" content="hackathon">
<meta name="tags" content="augmented reality">
<meta property="og:locale" content="en">
<meta property="og:site_name" content="Another Datum">
<meta property="og:type" content="article">
<meta property="article:author" content="">
<meta property="og:url" content="">
<meta property="og:title" content="Zooming Past the Competition">
<meta property="article:published_time" content="2018-10-10 21:00:00+03:00">
<meta property="og:description" content="How to create an Augmented Reality app that allows a user to get content recommendations.">
<meta property="og:image" content="">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:site" content="@YZeldes">
<meta name="twitter:title" content="Zooming Past the Competition">
<meta name="twitter:image" content="">
<meta name="twitter:description" content="How to create an Augmented Reality app that allows a user to get content recommendations.">
<!-- Navigation -->
<nav class="navbar navbar-default navbar-custom navbar-fixed-top">
<div class="container-fluid">
<!-- Brand and toggle get grouped for better mobile display -->
<div class="navbar-header page-scroll">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<a class="navbar-brand" href="">Another Datum</a>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
<li><a href="">Posts</a></li>
<li><a href="">about me</a></li>
<li><a href="">Resources</a></li>
<!-- /.navbar-collapse -->
<!-- /.container -->
<!-- Page Header -->
<header class="intro-header" style="background-image: url('images/hackathon_2018/cover.png')">
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2 col-md-10 col-md-offset-1">
<div class="post-heading">
<h1>Zooming Past the Competition</h1>
<span class="meta">Posted on 10 October 2018</span>
<!-- Main Content -->
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2 col-md-10 col-md-offset-1">
<!-- Post Content -->
<p>Imagine you’re walking down the street and you see a nice car you’re thinking of
buying. Just by pointing your phone camera, you can see relevant content about
that car. How cool is that?!</p>
<p>That was our team’s idea that awarded us first place in the recent Taboola R&amp;D
hackathon aptly named — <a href="">Taboola Zoom</a>!</p>
<p>Every year Taboola holds a global R&amp;D hackathon for its 350+ engineers aimed at
creating ideas for cool potential products or just some fun experiments in
<p>This year, 33 teams worked for 36 hours to come up with ideas that are both
awesome and helpful to Taboola. Some of the highlights included a tool that can
accurately predict the users’ gender based on their browsing activity and an
integration to social networks for Taboola Feed.</p>
<p>Our team decided to create an
<a href="">AR</a> (Augmented Reality)
application that allows a user to get content recommendations, much like
Taboola’s recommendations widget, based on whatever they’re pointing their phone
camera at.</p>
<h1>What is Zoom?</h1>
<p><img alt="" src="images/hackathon_2018/demo.png" /></p>
<p>The app itself is an AR experience similar to that of Google Glass, which allows
you to interact with the world using your phone camera. Using the app, one just
needs to point their camera onto an object to immediately get a list of stories
from the web related to that object.</p>
<p>To make this idea a reality we used technologies from several domains —
<a href="">AI</a>, <a href="">Web
Applications</a> and
<a href="">Microservices</a>.</p>
<h3>Under the Hood</h3>
<p><img alt="" src="images/hackathon_2018/architecture.png" /></p>
<p>The flow is pretty simple:</p>
<li>The user opens a web application on his phone — an
<a href="">HTML5</a> page that
behaves like a native app.</li>
<li>The app sends a screenshot of the captured video every second to a remote
<li>The server processes the image using computer vision technologies.</li>
<li>The server searches for web articles with thumbnails that are the most similar
to the processed image.</li>
<li>The retrieved images are filtered using a similarity threshold, and are sent
back to the user to be shown in a slick widget atop of the user’s display.</li>
<li>When the user clicks the widget the relevant article is opened in the browser.</li>
<h3>Zooming In</h3>
<p>Each component of the system is implemented using different technologies:</p>
<h4>Web Application</h4>
<p>We decided to implement the user interface using HTML5, which allowed us access
to native capabilities of the phone such as the camera. Additionally, we used
<a href="">WebRTC API</a> and
<a href="">Canvas API</a>.</p>
<h4>Computer Vision Service</h4>
<p><img alt="" src="images/hackathon_2018/vision.jpg" /></p>
<p>For every image, the service processes the image and returns an
<a href="">embedding</a>
— a numerical vector representing information extracted from the content of the
<p>Understanding the content of an image is a well known problem with plenty of
solutions. We chose to use Google’s
<a href="">Inception</a>
model, which is a <a href="">DNN</a> (Deep
Neural Network) trained to classify the object found in an image.</p>
<p>A DNN is a construct of layers of neurons — similar to nodes in a graph, where
each layer learns certain patterns in the image. The first layers learn to
output simple patterns such as edges and corners, while the last layer outputs
the type of the object, e.g. dog, cat etc.</p>
<p>We chose to use the output of the layer before last — as that produced the best
<p>The only component that was already available to us was Taboola’s internal
database of articles, containing, among other things, the thumbnail and the
article URL.</p>
<p>If such a database was not readily available to us, we could have just built our
own by <a href="">scraping</a> images using a
library such as <a href="">BeautifulSoup</a>.</p>
<p>We used Flask as our web server. On startup, the server queries Taboola’s
internal database of articles from the web. It then sends each image to the
computer vision service, which returns an embedding. The embeddings are then
stored into a designated data structure called
<a href="">FAISS</a>
(Facebook AI Similarity Search). It allows us to perform a <a href="">nearest neighbors
<p>Each image sent from the user is similarly transformed into an embedding. It is
then used as a query to the above data structure to retrieve its nearest
embeddings, meaning, images with similar patterns and content. Only images which
are considered similar above a predefined threshold are then returned to the
<p>So to recap, the entire architecture relies on three main components:</p>
<li>web application</li>
<li>computer vision service</li>
<li>articles database</li>
<h1>Do it Yourself</h1>
<p><img alt="" src="images/hackathon_2018/diy.jpg" /></p>
<p>The entire project was developed in under 36 hours by a team of 5 people.</p>
<p>This app touches several interesting and exciting domains that are “hot” in the
industry — AR and AI. It was a breeze to implement thanks to the commonly
available tools and libraries.</p>
<p>If there’s only one thing we want you to take from this is that it’s not that
<p>We invite you to be aware of the potential that lies in these domains and to be
on the lookout for interesting and exciting ideas. Once you find one, go and
have your own private hackathon!</p>
<hr />
<p>If you want to play around with the app, open
<a href=""></a> <strong>on your phone</strong> — use
Chrome on Android and Safari on iOS, and try hovering over different objects to
see the various results — keep in mind this is a pre-alpha hackathon project.</p>
<p>We want to thank our wonderful teammates who worked tirelessly to create this
amazing app –</p>
<p><a href="">Amir Keren</a>, <a href="">Yoel
Zeldes</a>, <a href="">Elina
Revzin</a>, <a href="">Aviv
Rotman</a> and <a href="">Ofri
<hr />
<p><em>Originally published at
<a href=""></a>
by me and <a href="">Amir Keren</a>.</em></p>
<div class="tags">
<p>tags: <a href="">deep learning</a>, <a href="">web</a>, <a href="">hackathon</a>, <a href="">augmented reality</a></p>
<!-- Begin MailChimp Signup Form -->
<link href="//" rel="stylesheet" type="text/css">
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:300px;}
#mc_embed_signup form{padding: 0;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
<div id="mc_embed_signup">
<form action=";id=c6f34ad6b7" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate>
<div id="mc_embed_signup_scroll">
<h2>Get updated of new posts</h2>
<div class="mc-field-group">
<label for="mce-EMAIL">Email Address </label>
<input type="email" value="" name="EMAIL" class="required email" id="mce-EMAIL">
<div id="mce-responses" class="clear">
<div class="response" id="mce-error-response" style="display:none"></div>
<div class="response" id="mce-success-response" style="display:none"></div>
</div> <!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_6894d7badcfb253606fa3fb54_c6f34ad6b7" tabindex="-1" value=""></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button"></div>
<script type='text/javascript' src='//'></script><script type='text/javascript'>(function($) {window.fnames = new Array(); window.ftypes = new Array();fnames[0]='EMAIL';ftypes[0]='email';fnames[1]='FNAME';ftypes[1]='text';fnames[2]='LNAME';ftypes[2]='text';}(jQuery));var $mcj = jQuery.noConflict(true);</script>
<!--End mc_embed_signup-->
<hr />
<div class="comments">
<h2>Comments !</h2>
<div id="disqus_thread"></div>
<script type="text/javascript">
var disqus_shortname = 'anotherdatum';
var disqus_identifier = 'taboola-hackathon-2018.html';
var disqus_url = '';
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = '//';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
<noscript>Please enable JavaScript to view the comments.</noscript>
<!-- Footer -->
<div class="container">
<div class="row">
<div class="col-lg-8 col-lg-offset-2 col-md-10 col-md-offset-1">
<ul class="list-inline text-center">
<a href="">
<span class="fa-stack fa-lg">
<i class="fa fa-circle fa-stack-2x"></i>
<i class="fa fa-linkedin fa-stack-1x fa-inverse"></i>
<a href="">
<span class="fa-stack fa-lg">
<i class="fa fa-circle fa-stack-2x"></i>
<i class="fa fa-github fa-stack-1x fa-inverse"></i>
<a href="">
<span class="fa-stack fa-lg">
<i class="fa fa-circle fa-stack-2x"></i>
<i class="fa fa-facebook fa-stack-1x fa-inverse"></i>
<a href="">
<span class="fa-stack fa-lg">
<i class="fa fa-circle fa-stack-2x"></i>
<i class="fa fa-twitter fa-stack-1x fa-inverse"></i>
<p class="copyright text-muted">
Blog powered by <a href="">Pelican</a>,
which takes great advantage of <a href="">Python</a>.
<br />
Blog sources can be found <a href="">here</a>.
</p> </div>
<!-- jQuery -->
<script src=""></script>
<!-- Bootstrap Core JavaScript -->
<script src=""></script>
<!-- Custom Theme JavaScript -->
<script src=""></script>
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-83684090-1']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
<script type="text/javascript">
var disqus_shortname = 'anotherdatum';
(function () {
var s = document.createElement('script'); s.async = true;
s.type = 'text/javascript';
s.src = '//' + disqus_shortname + '';
(document.getElementsByTagName('HEAD')[0] || document.getElementsByTagName('BODY')[0]).appendChild(s);