Skip to content

techhat/webflayer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Web Flayer

Web Flayer is a data mining tool that makes use of the Salt loader system to determine which sites to process, and how. Think of it as wget with plugins.

Requirements

The following python libraries are required:

  • beautifulsoup4
  • psycopg2
  • pyyaml
  • requests
  • salt
  • termcolor

Web Flayer also requires PostgreSQL. Schema can be found in the schema directory.

About

Data mining tool

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages