Automated web browsing with javascript / cookies support.
Python JavaScript Shell
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


= Introduction =

Spynner is a stateful programmatic web browser module for Python. It is based upon [ Qt] and [ WebKit], so it supports Javascript, AJAX, and every other technology that !WebKit is able to handle (Flash, SVG, ...). Spynner takes advantage of [ jQuery], a powerful Javascript library that makes the interaction with pages and event simulation really easy.  

Using Spynner you would able to simulate a web browser with no GUI (though a browsing window can be opened for debugging purposes), so it may be used to implement crawlers or acceptance testing tools.

= Dependencies =

  * [ Python] (>=2.5)
  * [ PyQt] (>=4.4.3): Python wrappers for the [ Qt] framework.

= Install =

  * A [ release] version (recommended):

$ wget
$ tar xvzf spynner-VERSION.tgz
$ cd spynner-VERSION
$ sudo python install


$ sudo easy_install spynner

  * The bleeding edge version:

$ svn checkout spynner-trunk
$ cd spynner-trunk
$ sudo python install

= API =

You can generate the API locally (will create docs/api directory):

$ python gen_doc

= Usage =

A basic example:

import spynner

browser = spynner.Browser()
browser.runjs("console.log('I can run Javascript')")
browser.runjs("console.log('I can run jQuery: ' + jQuery('a:first').attr('href'))")"#esen")
browser.fill("input[name=enit]", "hola")"input[name=b]")
print browser.url, browser.html

Sometimes you'll want to see what is going on:

browser = spynner.Browser()
browser.debug_level = spynner.DEBUG

See more examples in the repository:

= Running Javascript = 

Spynner uses jQuery to make Javascript interface easier. By default, two modules are injected to every loaded page:

  * [ jQuery core]: Amongst other things, it adds the powerful [ jQuery selectors], which are used internally by some Spynner methods (_fill_, _select_, _click_, _check_, ...). Of course you can also use jQuery when you inject your own code into a page.

  * [ jQuery _simulate_ plugin]: Makes it possible to simulate mouse and keyboard events (for now spynner uses it only in the _click_ action). Look up the library code to see which kind of events you can fire.

Note that you must use __jQuery(...)_ instead of _jQuery(...)_  or the common shortcut _$(...)_. That prevents name clashing with the jQuery library used by the page.

= Cook your soup: parsing the HTML = 

You can parse the HTML of a webpage with your favorite parsing library ([ BeautifulSoup], [ lxml], ...). Since we are already using Jquery for Javascript, it feels just natural to work with [ pyquery], its Python counterpart:

import spynner
import pyquery

browser = spynner.Browser()
d = pyquery.Pyquery(browser.html)
href = d("#somelink").attr("href"), open("/path/outputfile", "w"))

= Running Spynner without X11 ==

Spynner needs a X11 server to run. If you are running it in a server without X11 you must install the virtual [ Xvfb server]. Debian users can use the small wrapper (xvfb-run). If you are not using Debian, you can download it here:

$ xvfb-run python

= Feedback =

Open an [ issue] to report a bug or request a new feature. Other comments and suggestions can be directly emailed to me: [mailto://].