Search Engine Optimization for single page apps that cannot spawn additional processes.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Spoon.Standalone.Connector
Spoon.Standalone.Example
Spoon.Standalone
.gitignore
LICENSE
README.md
Spoon.Standalone.sln

README.md

Spoon.Standalone

Deprecation notice

Google now supports crawling of web applications that use JavaScript to present their content, and has deprecated the escaped fragment Ajax crawling scheme that Spoon was designed to take advantage of. As such, this library is no longer needed, and no longer maintained.

Overview

Spoon Standalone is an alternative solution to Spoon, built to work in cases where it is not possible to launch a separate process to crawl your site. Azure Web Sites are a good example of this.

How does it work?

Spoon Standalone works in much the same way as Spoon does, crawling your site and creating snapshots of your dynamic content that can be served up to search engines and web crawlers when they request it. There are a few key differences between the two projects, however.

  • Spoon Standalone is not embedded into a web application like Spoon is. It is launched manually once a given site is up and running.
  • Spoon Standalone stores its snapshots in an Azure blob storage container and not in a directory in your web application's hierarchy.
  • Spoon Standalone is written in Node.js. That said, because the project is not integrated into your web application you do not actually need to know anything about Node.js in order to use it. The only requirement is that you have Node.js installed on your system so that you can run it.
  • Spoon Standalone will work with any server-side web application architecture. The Spoon Standalone Connector which streamlines the retrieval of snapshots is written for ASP.NET web applications, but is not required to use Spoon Standalone.

How do I use it?

Step 1 - Prepare Node.js

Node.js is required to run Spoon Standalone. Install it on your system if necessary, then navigate to the directory where you have downloaded Spoon Standalone and run npm install to install all necessary dependencies.

Step 2 - Prepare an Azure blob storage container

To use Spoon Standalone you will need an Azure storage account with an empty blob storage container. This container must be reserved exclusively for Spoon, as Spoon will clear its contents before uploading any snapshots in order to use as little storage space as possible. When you create the container, be sure to give it Public Blob access.

Step 3 - Configure Spoon Standalone

Spoon Standalone needs a bit of information in order to crawl your site. Fill in the configuration file located at /src/config.js with the following information.

  • The name of your Azure storage account.
  • The access key of your Azure storage account.
  • The name of the container in which you wish to store your snapshots.
  • The list of pages to crawl.

There are actually two ways of telling Spoon Standalone which pages you want it to crawl. The best way is to provide it with the URL of a simple sitemap. A simple sitemap is a basic text file with one URL per line. Spoon Standalone does not at the moment support XML-style Sitemaps, but could in the future if there was demand for such a feature.

If your site does not have a simple sitemap, you can tell Spoon Standalone which pages to crawl by providing it with an array of URLs. Be aware that if you specify both a sitemap and an array of URLs to crawl, Spoon Standalone will only crawl the URLs listed in the sitemap.

Step 4 - Add snapshot retrieval code to your site

ASP.NET MVC

Install the Spoon Standalone Connector from NuGet (PM> Install-Package Spoon.Standalone.Connector) and rewrite your main Action method as follows.

public async Task<ActionResult> Index(string _escaped_fragment_)
{
    if (_escaped_fragment_ == null)
        return View();

    // NOTE: Failure to respond correctly to _escaped_fragment_ requests could result in your entire site
    // not being indexed. You should at the very least log all errors before handling them appropriately.
    try
    {
        // TODO: Fill in your Azure storage account and storage container names.
        return await SnapshotManager.GetSnapshotAsync(_escaped_fragment_, "", "");
    }
    catch (ArgumentNullException)
    {
        // The _escaped_fragment_ is null. You must verify that it is not null before calling GetSnapshotAsync.
        throw;
    }
    catch (ArgumentException)
    {
        // The Azure storage account name or storage container name is null or whitespace.
        throw;
    }
    catch (HttpRequestException)
    {
        // The _escaped_fragment_ does not correspond to a snapshot generated by Spoon Standalone.
        throw;
    }
    catch
    {
        // If you end up here please create an issue at https://github.com/LeviBotelho/spoon-standalone/issues.
        throw;
    }
}

Other platforms

You will need to add code to your application to return an Azure blob as an HTML document whenever an _escaped_fragment_ is passed as a GET parameter in a request to your site. The code to do this is very simple, as Azure blobs are accessible directly via a URL. No Azure storage API is required. The URLs that Spoon uses to store snapshots are constructed as follows.

http://[Storae Account].blob.core.windows.net/[Container]/_escaped_fragment_=[_escaped_fragment_ value].html

Note that [_escaped_fragment_ value] will be empty when the snapshot refers to the application's home page.

Step 5 - Publish your site

Your site must be up and running in order for Spoon to crawl it and generate snapshots.

Step 6 - Run Spoon Standalone

Run Spoon Standalone by executing node run.js from Spoon's root directory.

Step 7 - Test it out

Test that everything is working by making a sample request to your site with an _escaped_fragment_ using Fiddler or a similar tool. Dynamic content should be correctly rendered.

Problems? Don't like what you see?

I wrote this project out of personal necessity but would be happy to adapt it for use by a wider audience if the need arises. Please feel free to make feature requests or bug reports in the issue tracker.