Web-Scraping

It's a web scraping tool, that harvests social media links.

Input:

• The input can be done in the command line or after you run the script:

  • at runtime:

    

  • in the command line:

    

• The input can be of three types:

  • URL (contains HTTP:// or HTTPS:// or www. or combination between them).

    • it will try to scrap it directly

      

  • domain (ex: megacorpone.com).

    • will try to scrap it as it is first then try all the different combinations until it finds a valid one.

    

  • Ip (ex: 127.0.0.1).

    • will try to scrape it as it is then try to add HTTP and then HTTPS to change the port.

      

The script checks if the website is valid.

The script checks for captcha.

The script has a whiteList that filters the links that we are interested in (you can add or remove elements to it).

The script has a blacklist that filters the links that contains any of the keywords in there subdirectory filtering any non profile url

(you can add or remove elements to it).

The script Saves two files, one file with all the found links and another with the whitelisted items.

• All Links:

  

• Whitelisted Links:

  

Log file:

Other Features:

not allowing redirects.

bypassing certificates.

Filtering duplicate links.

filter non profile links (feed, stories, stores, etc.).

Bypassing INCAPSULA.

Log file when running the script.