Skip to content

egorka13/web-crawler

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Web Crawler + Parser

This project may help you to parse data from web-sites.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

  1. To install all dependencies simply type:
$ pip3 install selenium bs4 requests
  1. Next, you need to download web driver from Chromium
  2. Update the path to your driver in dynamic.py
webdriver.Chrome(executable_path="your_path")
  1. Navigate to web-crawler/linkChecker/linkChecker/spiders folder and execute script
$ python dynamic.py

About

university project

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 95.5%
  • C 3.1%
  • XSLT 0.8%
  • C++ 0.3%
  • Objective-C 0.1%
  • Roff 0.1%
  • Other 0.1%