Web crawler.
Clone or download
Latest commit b763532 Dec 6, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
image Add files via upload Nov 29, 2018
src/main defualt Nov 10, 2018
.gitignore init Oct 29, 2018
.travis.yml Update .travis.yml Nov 11, 2018
LICENSE Create LICENSE Oct 30, 2018
PornBot.jar Update PornBot.jar Nov 10, 2018
README.md Update README.md Dec 6, 2018
config.properties defualt Nov 10, 2018
pom.xml Update pom.xml Nov 11, 2018
pronbot.flow.svg Add files via upload Nov 8, 2018

README.md

Web Vedeo Bot

java-1.8 license: MIT Build Status HitCount

  • Easy install, all dependencies are already included, just need to have a java environment.

porn_flow

Environment, Architecture

  • Java1.8

  • Crawler4j

  • Spring Boot x H2 Db

Run

java -jar PornBot.jar

Demo h2_console

Database Description

http://localhost:8000/h2-console/

JDBC URL: jdbc:h2:~/porn/porn-db

User Name: sa

Password: empty

Record Table:

 Table_Name               :PORN_RECORD
 viewKey                  :The website's video unique key.
 imageUrl                 :Image url of video.
 linkUrl                  :Video jump to Website`s link
 videoUrl                 :Video adrress.
 videoTitle               :Title of video.
 videoDuration            :Video click count.
 videoQuality             :Defualt quality - 240, 480, 960, 1280p.
 download                 :Has been downloaded. True or false.
 createdTime              :The record created time.
 filePath                 :The video downloaded path.

Configuration

 FILE_PATH                :Video download path.
 VIDEO_DOWNLOAD_SIZE      :Maximum download size of the video.(Byte)
 MAX_PAGE_SIZE            :Crawling page size.
 CONCURRENT_THREAD_SIZE   :Muilti-threads request amount.
 START_URL                :Crawling url.
 DOWNLOAD_VIDEO           :Download video. Y or N.

See Default Configuration