Skip to content
Scrapy proxy authentication example for Smartproxy
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
LICENSE Create LICENSE Jul 17, 2019
README.md Update README.md Aug 19, 2019
smartproxy_spider.py Moving Scrapy to a new DIR Jul 10, 2019

README.md

Smartproxy logo

🏠 Main Repository 🏠

Disclaimer

The following example is a simple script showing how to utilize Smartproxy with Scrapy. We suggest to reseach Scrapy documentation in order to continue development with this tool.

Prerequisites

To get started with Scrapy you will first need to install it using methods provided in their documentation. Check here for more information

Installation

Once you get Scrapy up and running if you have not yet, make sure that you create your project folder

scrapy startproject yourprojectname

When project directory is setup, you can now copy our test spider code:

  1. Open Terminal/Command prompt.
  2. Open your project folder using cd .\yourprojectname\yourprojectname\spiders\
  3. Run command curl https://raw.githubusercontent.com/Smartproxy/Scrapy/master/smartproxy_spider.py > smartproxy_spider.py
  4. Run the script using scrapy crawl smartproxy command.

How to check if it works?

As mentioned this script only sends a basic request to a certain website to check what IP address is currently visiting it.

If you done all the steps correctly, you should see the result as {'ip': '\nYour Public IPv4 is: XX.XX.XX.XX'} along with other actions performed by Scrapy in the Terminal window.

Need help?

Email - sales@smartproxy.com
Live chat 24/7

You can’t perform that action at this time.