Skip to content

jooservices/XCrawler

develop
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
app
 
 
bin
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

XCrawler ( aka XGallery )

Codacy Badge Code Coverage codecov Scrutinizer Code Quality Build Status Maintainability Rating

A small project ( but took so long time ) based on Laravel for web Crawling ( with porn targeted ) data

How to install

Requirements

Database

  • MySQL
    • Primary data
    • Telescope
  • MongoDb
    • Logging

Supervisor

  • Redis

Cache

  • APCu

Worker / Supervisor

  • Horizon
    • api : For 3rd API request
      • maxProcesses: 2
    • crawling : For Crawling purpose
      • maxProcesses : 5
    • default : General usage
      • maxProcesses : 10

UnitTest

  • composer test for full test All crawler must be provided with 2 tests
  • Mock
  • Without mock ( request directly to target server )

Contribute

  • Create PR from develop
  • Make sure you have UnitTest fully
  • PR must be passed all required conditions