A Python Qt based tool for extracting content from complex websites.
from webkit_browser import Browser from lxml import html b = Browser() b.open('http://google.com/search?q=python') content = b.main_frame['content'].read() dom = html.fromstring(content) results = dom.xpath('//*[@id="ires"]/ol/li') for result in results: print result.find('h3').text_content()
Assuming that you're on a Linux box, you need
in your system to make it work. If you plan to run it on a server, you
may want to use
xvfb, since Qt needs a display backend.
If you get it working on another environment, please contribute to this README. :)
Note that this software is not intended to replace tools like
Mechanize nor others simple tools for doing web scraping. I would
not use it if the page I want would be downloadable with a simple
Browser class that works similarly to the Mechanize's
Browser, but without all that extra functionality. You can follow
the above example and interact with the
When the page is loaded, the code looks for all the page frames,
recursively, and puts them up in a dictionary. Each "frame" has three
'content' (a file-like object) and
'children' (a list containing child frames, if they exist). As you
can see in the example,
main_frame serves as the root frame.
Additionally, if you're running the code in a graphical environment, a mini-browser window will open, showing what's happen under the hoods.
If you need to handle authentication and/or your page goes through many
redirects until you finally get what you want, consider providing a
validate function for it. Example:
def proceed_after_redirect(qwebview): """ After all the redirects, a page with title 'Home' will be displayed. Note that you'll be handing the ``QWebView`` instance in this function, not a ``Browser`` object. """ if 'Home' in qwebview.page().mainFrame().title: return True
You can always mix these to get what fits your problem. ;)
This project is licensed under the DWTFYW (Do What The Fck You Want) license. No, I'm kidding. It's *MIT licensed; anyway you are free to do anything you want with it.
If this program saved your day, please consider sending me some soda or donate to my PayPal account (firstname.lastname@example.org) so I can buy it here. :)
Note: this code needs packaging. Why don't you fork it and make it a Python package?