A web scraping comparison for LXML, BeautifulSoup with additional tools to investigate Selenium and Scrapy.
If you'd like to use virtual environments, please follow the following instructions. It is not required for the tutorial but may be helpful.
For more details on virtual environments
If you don't have virtual env wrapper and/or pip:
$ easy_install pip
$ pip install virtualenvwrapper
and read the additional instructions here
$ mkvirtualenv scraper_tutorial
$ pip install -r requirements.txt
You will need both LXML and Selenium to run all of the tests in this repository.
If you are using a Mac, I would highly recommend using Homebrew. It will help make pip install very easy for you to use.
- Homebrew and LXML Help
- More help on Installing LXML on Mac
- And additional suggestions for LXML on Mac
If you are using Windows, it might be worth it to run this within a Linux Virtual Machine. If you are a Windows + Python guru, please follow these installation instructions. I can help as needed but I have not programmed on Windows in more than 5 years.
Firefox comes as the default web driver for Selenium. To use Selenium easily, please download and install Firefox.
/msg kjam on freenode or @kjam on twitter