Skip to content
Google, Naver multiprocess image web crawler (Selenium)
Branch: master
Clone or download
Yoongi Kim
Latest commit ee397ba Aug 12, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
chromedriver chromedriver update, auto version checking Aug 12, 2019
docs animation Feb 19, 2019
.gitignore Initial Commit Nov 21, 2018
LICENSE Initial commit Nov 21, 2018 Update Jul 23, 2019 google full update for new site Aug 12, 2019
keywords.txt Initial Commit Nov 21, 2018 chromedriver update, auto version checking Aug 12, 2019
requirements.txt Fix dependency conflict for issue Aug 6, 2019


Google, Naver multiprocess image crawler (High Quality & Speed & Customizable)

How to use

  1. Install Chrome

  2. pip install -r requirements.txt

  3. Write search keywords in keywords.txt

  4. Run ""

  5. Files will be downloaded to 'download' directory.



python3 [--skip true] [--threads 4] [--google true] [--naver true] [--full false] [--face false]
--skip true        Skips keyword if downloaded directory already exists. This is needed when re-downloading.

--threads 4        Number of threads to download.

--google true      Download from (boolean)

--naver true       Download from (boolean)

--full false       Download full resolution image instead of thumbnails (slow)

--face flase       Face search mode

Full Resolution Mode

You can download full resolution image of JPG, GIF, PNG files by specifying --full true

Data Imbalance Detection

Detects data imblance based on number of files.

When crawling ends, the message show you what directory has under 50% of average files.

I recommend you to remove those directories and re-download.

Remote crawling through SSH on your server

sudo apt-get install xvfb <- This is virtual display

sudo apt-get install screen <- This will allow you to close SSH terminal while running.

screen -S s1

Xvfb :99 -ac & DISPLAY=:99 python3


You can make your own crawler by changing

You can’t perform that action at this time.