Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

README.md

ABOUT

a crawler for ajax applications using hashbang urls.

creating a html snapshot of your site for googlebot by walking through hashbang links.

sample usage:

./crawlajax.py 'http://2011.jsconf.us/#!/schedule'

this command creates following folder structure and files:

|- www
 \|- schedule
  |\|- index.html
  |- news
  |\|- index.html
  |- speakers
  |\|- index.html
  ...

you can use any web server you preferred now to serve your static html snapshots by forwarding requests with escaped_fragment parameter to this folder.

http://2011.jsconf.us/?_escaped_fragment_=/schedule => http://2011.jsconf.us/googlebot/schedule/index.html

using phantomjs to take html snapshot of rendered dom. you can change snapshot command via construction parameter snapshot_cmd.

for making ajax applications crawlable, check instructions at:

we tried using htmlunit as google suggests at first, but we had performance problems, so we chose this way.

project satisfies only our applications requirements for now, you're free to fork and enrich it according to your needs.

INSTALLATION

dependencies:

  • python-2.6.6
  • phantomjs-1.3.0

see instructions to build and install phantomjs:

TEST

you can run unit tests via cmd:

./testcrawlajax.py

About

ajax site crawler to create snapshots for googlebot

Resources

License

Releases

No releases published

Packages

No packages published