Skip to content

Scraping the free proxy list and use sqlalchemy to save it to the database.

Notifications You must be signed in to change notification settings

mosiahr/free_proxy

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Free Proxy List

Scraping free proxy list and use SQLAlchemy to save it to database.

Getting Started

  1. Clone this repo with command:
    git clone git@github.com:mosiahr/free_proxy.git
    cd free_proxy/
  2. Create a virtual environment (either pipenv, virtualenv or venv)
  3. Install requirements (either with pipenv or pip install -r requirements.txt)
  4. Run the command to get help:
    python3 main.py --help

Save proxies to CSV file

Run command: python3 main.py --file csv
or with short option: python3 main.py -f csv

Save proxies to SQLite database

Run command: python3 main.py --database sqlite
or with short option python3 main.py -db sqlite

Save proxies to MySQL database

  1. Before run program you need to set variables SCHEMA_MYSQL, USER_MYSQL, PASS_MYSQL, HOST_MYSQL, DB_NAME_MYSQL in settings.py

  2. Run command: python3 main.py --database mysql
    or with short option python3 main.py -db mysql

How to use proxies in your program?

You can get a proxy list just by writing:
scraper = FreeProxyScraper()
proxies = scraper.scraping()

Use echo

That see logs use argument: --echo yes
For example:
python3 main.py -db sqlite --echo yes