squatt.ing

This repository contains data scraping scripts and results from scraping the entire .ing TLD for English work domain hacks a few months after the release of the TLD. The results contain:

• WHOIS data
• Pricing for registration, transfer and renewal as provided by Gandi
• DNS records
• HTTP responses from both urllib as well as Firefox
• Website screenshots from Firefox

For more context, please refer to this blog post.

Usage

If you want to perform your own scraping run, start by checking out the repository:

git clone https://github.com/ioces/squatt.ing.git
cd squatt.ing

Clean up any cached results by deleting the wordlists and results folder, as well as main.log:

rm -r results wordlists main.log

Ensure you have Firefox installed, as the scraping process requires it. Then, install all of the Python dependencies:

pip install -r requirements.txt

From there, you can execute the main code and walk away. You can edit the script as you see fit to backfill results or similar:

python main.py

Results

Results are cached in the results/ folder. Each domain has a single folder associated with it, containing up to four files:

• dns.json - DNS results for the base domain
• http.json - HTTP responses from both urllib and Firefox, as well as a screenshot of the site
• registrar.json - Details of the domain and pricing from Gandi
• whois.json - WHOIS results

Note, these files can be read on-demand by using the Domain class:

from domain import Domain

d = Domain('abandon.ing', 'results/abandon.ing')
print(f"{d.name} costs {d.price} and contains the following words:\n{d.words}")

License

This code and these results are licensed under the MIT License.