Skip to content
Collection of scripts to aggregate image data for the purposes of training an NSFW Image Classifier
Branch: master
Clone or download
Alexander Kim
Alexander Kim add more urls
Latest commit b94ae29 May 11, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
scripts add more urls May 11, 2019
LICENSE Create LICENSE Jan 11, 2019 Update Mar 26, 2019
confusion_matrix.png init Jan 11, 2019

NSFW Data Scraper

Disclaimer: the data is noisy - do not use to train a production model


This is a set of scripts that allows for an automatic collection of tens of thousands of images for the following (loosely defined) categories to be later used for training an image classifier:

  • porn - pornography images
  • hentai - hentai images, but also includes pornographic drawings
  • sexy - sexually explicit images, but not pornography. Think nude photos, playboy, bikini, etc.
  • neutral - safe for work neutral images of everyday things and people
  • drawings - safe for work drawings (including anime)

Here is what each script (located under scripts directory) does:

  • - iterates through text files under scripts/source_urls downloading URLs of images for each of the 5 categories above. The Ripme application performs all the heavy lifting. The source URLs are mostly links to various subreddits, but could be any website that Ripme supports. Note: I already ran this script for you, and its outputs are located in raw_data directory. No need to rerun unless you edit files under scripts/source_urls.
  • - downloads actual images for urls found in text files in raw_data directory.
  • - (optional) script that downloads SFW anime images from the Danbooru2018 database.
  • - (optional) script that downloads SFW neutral images from the Caltech256 dataset
  • - creates data/train directory and copy all *.jpg and *.jpeg files into it from raw_data. Also removes corrupted images.
  • - creates data/test directory and moves N=2000 random files for each class from data/train to data/test (change this number inside the script if you need a different train/test split). Alternatively, you can run it multiple times, each time it will move N images for each class from data/train to data/test.


  • Java runtime environment:
    • debian and ubuntu:sudo apt-get install default-jre
  • Linux command line tools: wget, convert (imagemagick suite of tools), rsync, shuf

On Windows

  • option 1: download a linux distro from windows 10 store and run the scripts there

  • option 2

    • download and install git from here. Git also installs Bash on your pc
    • download and install wget from here and add it to PATH
    • run the scripts

On Mac

The only difference I encountered is that OS X does not have shuf command, but has gshuf instead that can be installed with brew install coreutils. After installation either create an alias from gshuf to shuf or rename shuf to gshuf in

How to run

Change working directory to scripts and execute each script in the sequence indicated by the number in the file name, e.g.:

$ bash # has already been run
$ find ../raw_data -name "urls_*.txt" -exec sh -c "echo Number of URLs in {}: ; cat {} | wc -l" \;
Number of URLs in ../raw_data/drawings/urls_drawings.txt:
Number of URLs in ../raw_data/hentai/urls_hentai.txt:
Number of URLs in ../raw_data/neutral/urls_neutral.txt:
Number of URLs in ../raw_data/sexy/urls_sexy.txt:
Number of URLs in ../raw_data/porn/urls_porn.txt:
$ bash
$ bash # optional
$ bash # optional
$ bash
$ bash
$ cd ../data
$ ls train
drawings hentai neutral porn sexy
$ ls test
drawings hentai neutral porn sexy

I was able to train a CNN classifier to 91% accuracy with the following confusion matrix: alt text

As expected, anime and hentai are confused with each other more frequently than with other classes.

Same with porn and sexy categories.

Note: anime category was later renamed to drawings

Download Trained Models

This repo provides you with the data. To train or download a trained model you can access both here:

You can’t perform that action at this time.