Skip to content
This repository was archived by the owner on Sep 20, 2022. It is now read-only.

kaanbasal/url-scraper

Repository files navigation

Url Scrapper

The code tries to fetch given url recursively to find all the links and lists them.

Implementation has several approaches to do the same thing.

How to run

  • If docker is installed, Makefile can be used to build an image and run it using make run
  • If you want to run directly using python 3 use python main.py after installing required packages
  • If you want to use PyCharm, there is a need to change run configuration settings as shown in the below image.

PyCharm

TODO

  • Planning to implement QueueExecutor to process in parallel, using asyncio semaphore
  • Planning to implement tests

About

Fetch all links recursively from given url

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published