Normalize a URL

Extract and decompose (fuzzy) URLs (including emails, which are conceptually a part of URLs) in texts with Area-Pattern-based modularity

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

Remove clutter from URLs and return a canonicalized version

🔗🧹 Normalize URLs to a standardized form. HTTPS by default, flexible configuration, custom protocols, domain extraction, humazing URL, and punycode support. Both CJS & ESM modules available.

Get a stable, canonical version of any URL, with DNS and HTTPS checks, redirects, tracker stripping, and canonical link extraction!

Allows you to remove ad/tracking query params from a given URL in Scala

URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical

🔗 Lightweight Ruby library for parsing URLs and extracting domain components with accurate multi-part TLD support. Handles nested subdomains, query parameters, and URL normalization. Perfect for web scraping, analytics, and URL manipulation. Built on URI and public_suffix gem.

🔗 Pathor is a PHP library for normalizing, analyzing, and comparing URLs.

Natural Language Precessing related notebooks (Machine Learning)

This project contains a Python script to extract all unique absolute URLs from a webpage and write them into a text file. This can be useful for indexing purposes.

web crawler from scratch as it possible by using python

A simple plugin to rewrite multiple domains or subdomains to a single domain in the site's HTML output.

A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.

http url normalization for web crawlers