Great article, but after scraping the article we need to display it somewhere...
\nFor me I want to display it on iOS devices.
\nDo you know any good libs or HTML templates to use the diffbot text in?<\/p>\n
Thanks for your help:)<\/p>","type":"post","date":"Sun, 19 Oct 2014 00:00:00 GMT"},{"id":3,"author":"Taher","text":"Is there any open source projects as good as diffbots?","diffbotUri":"post|3|-197836079","pageUrl":"http://www.sitepoint.com/diffbot-crawling-visual-machine-learning/","votes":0,"authorUrl":"https://disqus.com/by/disqus_CWGq6zNflN/","humanLanguage":"en","html":"
Is there any open source projects as good as diffbots?<\/p>","type":"post","date":"Sat, 19 Jul 2014 00:00:00 GMT"},{"id":4,"parentId":3,"author":"Bruno Skvorc","text":"The answers here might help you out: https://www.quora.com/Web-Scra...","diffbotUri":"post|3|-1403587915","pageUrl":"http://www.sitepoint.com/diffbot-crawling-visual-machine-learning/","votes":1,"authorUrl":"https://disqus.com/by/brunoskvorc/","humanLanguage":"en","html":"
The answers here might help you out: https://www.quora.com/Web-Scra...<\/a><\/p>","type":"post","date":"Sat, 19 Jul 2014 00:00:00 GMT"},{"id":5,"author":"anonymous","text":"i just tried out the demo on diffbots website, and out of 14 pages i feeded it, it could only properly process 3. with most of 'em it was just showing me the copyright notice / legal bla foo embeded in those pages","diffbotUri":"post|3|-2120915058","pageUrl":"http://www.sitepoint.com/diffbot-crawling-visual-machine-learning/","votes":0,"humanLanguage":"en","html":" i just tried out the demo on diffbots website, and out of 14 pages i feeded it, it could only properly process 3. with most of 'em it was just showing me the copyright notice / legal bla foo embeded in those pages<\/p>","type":"post","date":"Sat, 19 Jul 2014 00:00:00 GMT"},{"tags":[{"id":175464,"count":1,"label":"Application programming interface","uri":"http://dbpedia.org/resource/Application_programming_interface"}],"text":"That's where custom API comes in to save the day. Out of curiosity, though, which URLs did you try, and which information was missing?","pageUrl":"http://www.sitepoint.com/diffbot-crawling-visual-machine-learning/","votes":0,"humanLanguage":"en","type":"post","date":"Sat, 19 Jul 2014 00:00:00 GMT","id":6,"parentId":5,"author":"Bruno Skvorc","diffbotUri":"post|3|1316965997","authorUrl":"https://disqus.com/by/brunoskvorc/","html":" That's where custom API comes in to save the day. Out of curiosity, though, which URLs did you try, and which information was missing?<\/p>"}],"provider":"Disqus","humanLanguage":"en","confidence":1,"type":"discussion","participants":5,"rssUrl":"http://www.sitepoint.com/diffbot-crawling-visual-machine-learning/feed/","title":"Diffbot: Crawling with Visual Machine Learning","diffbotUri":"discussion|3|-1039854465","numPages":1},"pageUrl":"http://www.sitepoint.com/diffbot-crawling-visual-machine-learning","metaTags":[{"name":"ai"},{"name":"Artificial Intelligence"},{"name":"crawling"},{"name":"Diffbot"},{"name":"framework"},{"name":"laravel"},{"name":"machine learning"},{"name":"OOPHP"},{"name":"PHP"},{"name":"scraping"},{"name":"visual learning"}],"humanLanguage":"en","type":"article","date":"Sun, 27 Jul 2014 00:00:00 GMT","resolvedPageUrl":"http://www.sitepoint.com/diffbot-crawling-visual-machine-learning/","author":"Bruno Skvorc","title":"Diffbot: Crawling with Visual Machine Learning","diffbotUri":"article|3|-938093421","images":[{"height":533,"naturalHeight":727,"diffbotUri":"image|3|-851701004","primary":true,"width":780,"naturalWidth":1063,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624455201.png"},{"height":216,"naturalHeight":216,"diffbotUri":"image|3|762494522","width":523,"naturalWidth":523,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624487602.png"},{"height":184,"naturalHeight":184,"diffbotUri":"image|3|302236938","width":664,"naturalWidth":664,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624509003.png"},{"height":784,"naturalHeight":972,"diffbotUri":"image|3|-1836356546","width":780,"naturalWidth":966,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624552704.png"},{"height":156,"naturalHeight":184,"diffbotUri":"image|3|1297360030","width":780,"naturalWidth":918,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624650505.png"},{"height":157,"naturalHeight":188,"diffbotUri":"image|3|502449852","width":780,"naturalWidth":929,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624665606.png"},{"height":175,"naturalHeight":237,"diffbotUri":"image|3|-2007985802","width":780,"naturalWidth":1053,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624803507.png"},{"title":"Change the final output of the diffbotDemo() action to this:","height":520,"naturalHeight":604,"diffbotUri":"image|3|-140134863","width":780,"naturalWidth":906,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624809508.png"},{"height":533,"naturalHeight":727,"diffbotUri":"image|3|1129235416","width":780,"naturalWidth":1063,"url":"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624834309.png"}],"html":" Have you ever wondered how social networks do URL previews so well when you share links? How do they know which images to grab, whom to cite as an author, or which tags to attach to the preview? Is it all crawling with complex regexes over source code? Actually, more often than not, it isn’t. Meta information defined in the source can be unreliable, and sites with less than stellar reputation often use them as keyword carriers, attempting to get search engines to rank them higher. Isn’t what we, the humans, see in front of us what matters anyway?<\/p>\n If you want to build a URL preview snippet or a news aggregator, there are many automatic crawlers available online, both proprietary and open source, but you seldom find something as niche as visual machine learning. This is exactly what Diffbot<\/a> is – a “visual learning robot” which renders a URL you request in full and then visually extracts data, helping itself with some metadata from the page source as needed.<\/p>\n After covering some theory, in this post we’ll do a demo API call at one of SitePoint’s posts.<\/p>\n The PHP library for Diffbot is somewhat out of date, and as such we won’t be using it in this demo. We’ll be performing raw API calls, and in some future posts we’ll build our own library for API interaction.<\/p>\n If you’d like to take a look at the PHP library nonetheless, see here<\/a>, and if you’re interested in libraries for other languages, Diffbot has a directory<\/a>.<\/p>\n We said in the introductory section that Diffbot renders the request in full and then analyzes it. But, what about JavaScript content? Nowadays, websites often render some HTML above the fold, and then finish the CSS, JS, and dynamic content loading afterwards. Can the Diffbot API see that?<\/p>\n As a matter of fact, yes. Diffbot literally renders the page in full, and then inspects it visually, as explained in my StackOverflow Q&A here<\/a>. There are some caveats, though, so make sure you read the answer carefully.<\/p>\n Diffbot has several usage tiers. There’s a free trial tier which kills your API token after 7 days or 10000 calls, whichever comes first. The commercial tokens can be purchased at various prices<\/a>, and never expire, but do have limitations. A special case by case approach is afforded to open source and/or educational projects which provides an older model of the free token – 10k calls per month, once per second max, but never expires. You need to contact them directly if you think you qualify.<\/p>\n Diffbot guarantees a high uptime, but failures sometimes do happen – especially in the most resource intensive API of the bunch: Crawlbot. Crawlbot is used to crawl entire domains, not just individual pages, and as such has a lower reliability rate than other APIs. Not by a lot, but enough to be noticeable in the API Health<\/a> screen – the screen you can check to see if an API is up and running or currently unavailable if your calls run into issues or return error 500.<\/p>\n To prepare your environment, please boot up a Homestead Improved<\/a> instance.<\/p>\n Create a starter Laravel project by SSHing into the VM with In PHP Library<\/h2>\n
JavaScript Content<\/h2>\n
Pricing and API Health<\/h2>\n
Demo<\/h2>\n
Create Project<\/h3>\n
vagrant ssh<\/code>, going into the Code<\/code> folder, and executing composer create-project laravel/laravel Laravel --prefer-dist<\/code>. This will let you access the Laravel greeting page via http://homestead.app:8000<\/code> from the host’s browser.<\/p>\nAdd a Route and Action<\/h3>\n
app/routes.php<\/code> add the following route:<\/p>\n
![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624455201.png\")
<\/img><\/figure>\n![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624487602.png\")
<\/img><\/figure>\n![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624509003.png\")
<\/img><\/figure>\n![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624552704.png\")
<\/img><\/figure>\n![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624650505.png\")
<\/img><\/figure>\n![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624665606.png\")
<\/img><\/figure>\n![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624803507.png\")
<\/img><\/figure>\n![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624809508.png\")
<\/img>![\"\"](\"http://dab1nmslvvntp.cloudfront.net/wp-content/uploads/2014/07/140624834309.png\")
<\/img><\/figure>\n