Python Web Scraping

• https://www.geeksforgeeks.org/web-scraping-from-wikipedia-using-python-a-complete-guide/
• https://pypi.org/project/setuptools/

HTTP Clients

1. https://github.com/psf/requests

Packages

https://github.com/YoongiKim/AutoCrawler/blob/master/requirements.txt

1. https://pypi.org/project/certifi/
2. https://pypi.org/project/chardet/
3. https://pypi.org/project/idna/
4. https://pypi.org/project/requests/
5. https://pypi.org/project/selenium/
6. https://pypi.org/project/webdriver-manager/

https://github.com/cocrawler/cocrawler/blob/main/setup.py

• https://pypi.org/project/aiodns/ Simple DNS resolver for asyncio
• https://pypi.org/project/aiohttp/ Async http client/server framework (asyncio)
• https://pypi.org/project/cchardet/ cChardet is high speed universal character encoding detector.
• https://pypi.org/project/PyYAML/ YAML parser and emitter for Python
• https://pypi.org/project/uvloop/ Fast implementation of asyncio event loop on top of libuv
• https://pypi.org/project/yarl/ Yet another URL library

Parsers

• https://medium.com/analytics-vidhya/scrapy-vs-selenium-vs-beautiful-soup-for-web-scraping-24008b6c87b8

1. https://beautiful-soup-4.readthedocs.io
2. https://www.selenium.dev

• 2.1. https://github.com/SeleniumHQ
• 2.2. https://github.com/SeleniumHQ/selenium

3. https://scrapy.org

• 1.1. https://github.com/scrapy/scrapy

Wrapper Directory

Python API Wrappers

W

1. Wikipedia

• 1.1. https://github.com/goldsmith/Wikipedia
• 1.2. https://github.com/martin-majlis/Wikipedia-API

Python HTTP Wrappers

I

1. IMDB

• 1.1. https://github.com/alberanid/imdbpy

CLI

• https://en.wikipedia.org/wiki/CURL
  • https://curl.se
  • https://curl.se/docs/manpage.html
• https://en.wikipedia.org/wiki/Wget