Skip to content

Latest commit

 

History

History
172 lines (109 loc) · 5.27 KB

install.rst

File metadata and controls

172 lines (109 loc) · 5.27 KB
Raw

Installation guide

Installing Scrapy

Note

Check intro-install-platform-notes first.

The installation steps assume that you have the following things installed:

  • Python 2.7
  • pip and setuptools Python packages. Nowadays pip requires and installs setuptools if not installed. Python 2.7.9 and later include pip by default, so you may have it already.
  • lxml. Most Linux distributions ships prepackaged versions of lxml. Otherwise refer to http://lxml.de/installation.html
  • OpenSSL. This comes preinstalled in all operating systems, except Windows where the Python installer ships it bundled.

You can install Scrapy using pip (which is the canonical way to install Python packages).

To install using pip:

pip install Scrapy

Platform specific installation notes

Windows

  • Install Python 2.7 from https://www.python.org/downloads/

    You need to adjust PATH environment variable to include paths to the Python executable and additional scripts. The following paths need to be added to PATH:

    C:\Python27\;C:\Python27\Scripts\;

    To update the PATH open a Command prompt and run:

    c:\python27\python.exe c:\python27\tools\scripts\win_add2path.py

    Close the command prompt window and reopen it so changes take effect, run the following command and check it shows the expected Python version:

    python --version

  • Install pywin32 from http://sourceforge.net/projects/pywin32/

    Be sure you download the architecture (win32 or amd64) that matches your system

  • (Only required for Python<2.7.9) Install pip from https://pip.pypa.io/en/latest/installing.html

    Now open a Command prompt to check pip is installed correctly:

    pip --version

  • At this point Python 2.7 and pip package manager must be working, let's install Scrapy:

    pip install Scrapy

Ubuntu 9.10 or above

Don't use the python-scrapy package provided by Ubuntu, they are typically too old and slow to catch up with latest Scrapy.

Instead, use the official Ubuntu Packages <topics-ubuntu>, which already solve all dependencies for you and are continuously updated with the latest bug fixes.

If you prefer to build the python dependencies locally instead of relying on system packages you'll need to install their required non-python dependencies first:

sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

You can install Scrapy with pip after that:

pip install Scrapy

Archlinux

You can follow the generic instructions or install Scrapy from `AUR Scrapy package`:

yaourt -S scrapy

Mac OS X

Building Scrapy's dependencies requires the presence of a C compiler and development headers. On OS X this is typically provided by Apple’s Xcode development tools. To install the Xcode command line tools open a terminal window and run:

xcode-select --install

There's a known issue that prevents pip from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions:

  • (Recommended) Don't use system python, install a new, updated version that doesn't conflict with the rest of your system. Here's how to do it using the homebrew package manager:

    • Install homebrew:

      ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

    • Update your PATH variable to state that homebrew packages should be used before system packages (Change .bashrc to .zshrc accordantly if you're using zsh as default shell):

      echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc

    • Reload .bashrc to ensure the changes have taken place:

      source ~/.bashrc

    • Install python:

      brew install python

    • Latest versions of python have pip bundled with them so you won't need to install it separately. If this is not the case, upgrade python:

      brew update; brew upgrade python
  • (Alternative) Force system python to load the user installed packages first:

    • Update your PYTHONPATH variable (Change .bashrc to .zshrc accordantly if you're using zsh as default shell):

      echo "export PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH" >> ~/.bashrc

    • Reload .bashrc to ensure the changes have taken place:

      source ~/.bashrc

    • Install pip:

      sudo easy_install pip

After any of these workarounds you should be able to install Scrapy:

pip install Scrapy