Many ASNs are missing certain fields

[BlobTheKat]

E.g AS15 does not have any information such as:

• Type
• Description
• Domain
• Abuse email
• Org
• Updated
• ...

Performing an ASN lookup via ipapi's /whois api shows no results either, despite other services showing this information, and despite ipapi's IP lookup indeed showing the correct information in the "asn" field

[ipapi-is]

I will look into this and fix this very soon.

[BlobTheKat]

Thanks. Since then I have also noticed:

• Some still active ASNs are missing
• Many ASNs with types are not correct (they show a different type to api.ipapi.is)

(ps ipapi doesn't appear to be using up any of my one time credits despite that I'm doing a crap ton of requests, don't fix this ;))

[ipapi-is]

Thanks man. I probably don't use the newest ASN data pipeline to generate the database. Will definitely fix this very soon.

Some still active ASNs are missing

Do you have some examples?

ps ipapi doesn't appear to be using up any of my one time credits despite that I'm doing a crap ton of requests, don't fix this ;)

I fear I updated the backend and auth / billing system of the API during the past days and one time credits are billed correctly now :/

It was a low priority (have to make data quality better)

[BlobTheKat]

Some still active ASNs are missing

Do you have some examples?

400864 is the most recent I could find, with the associated prefixes ["198.145.42.0/23"]

I fear I updated the backend and auth / billing system of the API during the past days and one time credits are billed correctly now :/

😭 I've been using this to scrape a comprehensive offline database of accurate IP information, which we use to defend against bots.
HOWEVER, it has been brought to my attention that this is not enough, because of residential proxies. I own a site that gets a large amount of traffic from such proxies, and am interested in working with you to bring data about residential proxies to ipapi.is. Would you be interested? My email is blob.kat@hotmail.com, but you can also find me on discord (blobk.at)

[ipapi-is]

Please don't scape ipapi.is :) I see you fired a lot of POST requests / bulk lookups.

Please subscribe instead. How many requests per day do you need?

HOWEVER, it has been brought to my attention that this is not enough, because of residential proxies.

This is very correct. ipapi.is cannot reliably detect residential proxies.

I do have my own solution to detect residential and mobile proxies though and around 200k requests per day: https://proxydetect.live/

This is not enough to build a reliable offline database though. I would need a really huge site to source a offline database. At least 100M requests per day or something like this.

[BlobTheKat]

I've been buying these requests, I say it's "scraping" because it's one-time to build a DB
Indexed properly this DB weighs 10MB and has lookup times of under a microsecond, if you're interested.

Ours did peak at 20M requests a day, but has drastically reduced since then. I was talking about a potentially more thorough / concrete detection that doesn't rely on a single source of traffic (like your or my site)

[ipapi-is]

So I understand correctly that you looked up millions of IP's from ipapi.is and now you built your own IP address API?

How do you manage to get a lookup speed of 1 microsecond? Do you also use a sweep line algorithm?

Downside: Your API will be outdated within a week :/

[BlobTheKat]

Yes, 1,105,602 IPs, one corresponding to each range I do not use a sweep line algorithm, I instead use a Q-tree algorithm, which requires on average 10 array lookups per IP The resulting offline database is available for lookup as an NPM package Downside: I need to purchase 40 cents worth of credit every week to keep it updated enough Upside: I save countless credits on requests from my site, and lookups are inexpensive enough that they can be used to completely block hosting-based DDOS attacks

…

On 1 Aug 2023, at 09:45, ipapi.is ***@***.***> wrote: So I understand correctly that you looked up millions of IP's from ipapi.is and now you built your own IP address API? How do you manage to get a lookup speed of 1 microsecond? Do you also use a sweep line algorithm? Downside: Your API will be outdated within a week :/ — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APJ4WGOZRPEG6GFVDYQEJLDXTC62DANCNFSM6AAAAAA2VXXK4M>. You are receiving this because you authored the thread.

[ipapi-is]

I am closing this now since "AS15" is an inactive ASN and I don't have additional meta data for inactive ASNs such as:

• Type
• Description
• Domain
• Abuse email
• Org
• Updated

I won't WHOIS query inactive ASNs for some time since my focus is on active ASNs (at least one prefix assigned to it).