Skip to content
Permalink
Fetching contributors…
Cannot retrieve contributors at this time
178 lines (135 sloc) 10.5 KB
title subhead authors date hero alt description tags
Making JavaScript and Google Search work together
Get the latest updates and insights from Google I/O 2019.
martinsplitt
ekharvey
2019-06-05
hero.png
A pile of gears.
We introduced a new Googlebot at Google I/O and took the opportunity to discuss improvements and best practices for making JavaScript web apps work well with Google Search.
post
seo
discoverability
javascript

Great things are happening with Google Search, and we were excited to share them at Google I/O 2019!

<iframe style="width:100%; height: 100%;position: absolute; top: 50%; left: 50%; transform: translate(-50%,-50%);" src="https://www.youtube.com/embed/ufcijo46LCU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

In this post we'll focus on best practices for making JavaScript web apps discoverable in Google Search, including:

  • The new evergreen Googlebot
  • Googlebot's pipeline for crawling, rendering and indexing
  • Feature detection and error handling
  • Rendering strategies
  • Testing tools for your website in Google Search
  • Common challenges and possible solutions
  • Best practices for SEO in JavaScript web apps

Meet the evergreen Googlebot

This year we announced the much-awaited new evergreen Googlebot.

Googlebot holding the Chrome logo

Googlebot is now running a modern Chromium rendering engine.

Googlebot now uses a modern Chromium engine to render websites for Google Search. On top of that, we will test newer versions of Chromium to keep Googlebot updated, usually within a few weeks of each stable Chrome release. This announcement is big news for web developers and SEOs because it marks the arrival of 1000+ new features — such as ES6+, IntersectionObserver, and Web Components v1 — in Googlebot.

Learn how Googlebot works

Googlebot is a pipeline with several components. Let's take a look to understand how Googlebot indexes pages for Google Search.

A diagram showing a URL moving from a crawling queue to a processing step that extracts linked URLs and adds them to the crawling queue, a rendering queue that feeds into a renderer which produces HTML. The processor uses this HTML to extract linked URLs again and index the content.

Fig. 1 — Googlebot's pipeline for crawling, rendering, and indexing a page.

The process works like this:

  1. Googlebot queues URLs for crawling.
  2. It then fetches the URLs with an HTTP request based on the crawl budget.
  3. Googlebot scans the HTML for links and queues the discovered links for crawling.
  4. Googlebot then queues the page for rendering.
  5. As soon as possible, a headless Chromium instance renders the page, which includes JavaScript execution.
  6. Googlebot uses the rendered HTML to index the page.

Your technical setup can influence the process of crawling, rendering, and indexing. For example, slow response times or server errors can impact the crawl budget. Another example would be requiring JavaScript to render the links can lead to a slower discovery of these links.

Use feature detection and handle errors

The evergreen Googlebot has lots of new features, but some features are still not supported. Relying on unsupported features or not handling errors properly can mean Googlebot can't render or index your content.

Let's look at an example:

 <body>
   <script>
     navigator.geolocation.getCurrentPosition(function onSuccess(position) {
       loadLocalContent(position);
     });
   </script>
 </body>

This page might not show any content in some cases because the code doesn't handle when the user declines the permission or when getCurrentPosition call returns an error. Googlebot declines permission requests like this by default.

This is a better solution:

 <body>
   <script>
     if (navigator.geolocation) {
       // this browser supports the Geolocation API, request location!
       navigator.geolocation.getCurrentPosition(
         function onSuccess(position) {
           // we successfully got the location, show local content
           loadLocalContent(position);
         }, function onError() {
           // we failed to get the location, show fallback content
           loadGlobalContent();
         });
     } else {
       // this browser does not support the Geolocation API, show fallback content
       loadGlobalContent();
     }
   </script>
 </body>

If you have problems with getting your JavaScript site indexed, walk through our troubleshooting guide to find solutions.

Choose the right rendering strategy for your web app

The default rendering strategy for single-page apps today is client-side rendering. The HTML loads the JavaScript, which then generates the content in the browser as it executes.

Let's look at a web app that shows a collection of cat images and uses JavaScript to render entirely in the browser.

A code box showing HTML that loads some scripts. A screenshot of a web page on mobile that shows placeholder images while loading the actual content.

The rendering strategy influences the performance and robustness of your web apps.

If you're free to choose your rendering strategy, consider server-side rendering or pre-rendering. They execute JavaScript on the server to generate the initial HTML content, which can improve performance for both users and crawlers. These strategies allow the browser to start rendering HTML as it arrives over the network, making the page load faster. The rendering session at I/O or the blog post about rendering on the web shows how server-side rendering and hydration can improve the performance and user experience of web apps and provides more code examples for these strategies.

If you're looking for a workaround to help crawlers that don't execute JavaScript—or if you can't make changes to your frontend codebase—consider dynamic rendering, which you can try out in this codelab. Note, though, that you won't get the user experience or performance benefits that you would with server-side rendering or pre-rendering because dynamic rendering only serves static HTML to crawlers. That makes it a stop-gap rather than a long-term strategy.

Test your pages

While most pages generally work fine with Googlebot, we recommend testing your pages regularly to make sure your content is available to Googlebot and there are no problems. There are several great tools to help you do that.

The easiest way to do a quick check of a page is the Mobile-Friendly Test. Besides showing you issues with mobile-friendliness, it also gives you a screenshot of the above-the-fold content and the rendered HTML as Googlebot sees it.

The mobile-friendly test shows the rendered HTML Googlebot sees after rendering the page

The mobile-friendly test shows you the rendered HTML Googlebot uses.

You can also find out if there are resource loading issues or JavaScript errors.

The Mobile-Friendly Test shows JavaScript errors and a stack trace.

It's recommended to verify your domain in Google Search Console so you can use the URL inspection tool to find out more about the crawling and indexing state of a URL, receive messages when Search Console detects issues and find out more details of how your site performs in Google Search.

The URL inspection tool showing a page that is indexed with information on discovery, crawling and indexing for one URL

The URL Inspection Tool in Search Console shows the status of a page in crawling, rendering, and indexing.

For general SEO tips and guidance, you can use the SEO audits in Lighthouse. To integrate SEO audits into your testing suite, use the Lighthouse CLI or the Lighthouse CI bot.

A lighthouse SEO report with a score of 44 and warnings about mobile-friendliness as well as warnings about content best practices

Lighthouse shows general SEO recommendations for a page.

These tools help you identify, debug, and fix issues with pages in Google Search and should be part of your development routine.

Stay up to date and get in touch

To stay up to date with announcements and changes to Google Search, keep an eye on our Webmasters Blog, the Google Webmasters Youtube channel, and our Twitter account. Also check out our developer guide to Google Search and our JavaScript SEO video series to learn more about SEO and JavaScript.

You can’t perform that action at this time.