Skip to content

A high performance multi-threaded web-scraper using python and beautifulsoup

License

Notifications You must be signed in to change notification settings

Zummation/IQNewsClip-Web-Scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

45 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

README

Prerequisites

  • Anaconda 4+
  • Internet connection
  • Permission to access the website (via CUNET User Account)

Set up environment

  • Open Anaconda Prompt
  • Go to the root directory of this project. Then create its Conda environment from environment.yml by entering the following command:
    • conda env create -f environment.yml
  • Activate the environment by:
    • activate IQNewsClip-Web-Scraper
  • Install packages from requirements.txt by:
    • pip install -r requirements.txt
  • Install the local package by:
    • pip install . --no-cache-dir

Note

  • To remove the environment:
    • activate root
    • conda remove -n IQNewsClip-Web-Scraper --all

Useful links