urlscan.io ?

[ZeroDot1]

Is it possible to use urlscan.io to search for subdomains and include it in Subfinder?
https://urlscan.io/about-api/
https://urlscan.io/

https://github.com/heywoodlh/urlscan-py/blob/master/bin/urlscan

[Ice3man543]

@ZeroDot1 Seems like a lovely addition. Thank you for this. I will add it tonight.

[ZeroDot1]

@Ice3man543 Thank you for your good work :)
Subfinder is very good.

[Ice3man543]

@ZeroDot1 This service doesn't seems to give many subdomains?
It gives a few. Can you send me a link showing the pages it gives subdomains on? It would be good if you are quick.

[ZeroDot1]

@Ice3man543 https://urlscan.io/search/#page.domain%3Agithub.io
I can find very many subdomains.

[Ice3man543]

@ZeroDot1 But this would only work if somebody had scanned it before?

[Ice3man543]

We add only those sources in subfinder which we see providing real value to the users. We don't add any source we can get without getting results. In case of urlscan.io it seems really less likely that it will hold any good subdomain data. Also, their API looks like broken.
Try this - curl "https://urlscan.io/api/v1/search/?q=domain:google.com"

[ZeroDot1]

Yes, only subdomains that have already been scanned are displayed.
So it is possible to find many unknown urls, because many users scan urls.

[ZeroDot1]

For Subdomains:

curl "https://urlscan.io/api/v1/search/?q=page.domain:gitlab.io"

[ZeroDot1]

Yes, it shows some wrong urls, :(

[ZeroDot1]

@Ice3man543 I sent an email to URLSCAN a few days ago, here's the answer.

Hello there,

the API is for searching single submissions. We do not have an API to
enumerate subdomains yet.

If this is something you're interested in we might put it on the list
for future features ;)

Regards,
Jojo

[Ice3man543]

Yeah, tell them we are surely interested in that. Another good data source :)

[ZeroDot1]

It's already done.
Now we have to wait, I hope that the new function will be available soon.

[picatz]

@ZeroDot1 @Ice3man543 I'm not sure what ya'll may have been seeing before, but I just got this ( for reference ):

$ curl "https://urlscan.io/api/v1/search/?q=page.domain:gitlab.io"

{
  "results": [
    {
      "task": {
        "visibility": "public",
        "method": "manual",
        "options": {},
        "time": "2018-06-23T16:52:30.456Z",
        "source": "web",
        "url": "http://zerodot1.gitlab.io"
      },
      "stats": {
        "uniqIPs": 2,
        "consoleMsgs": 0,
        "dataLength": 3906,
        "encodedDataLength": 3257,
        "requests": 1
      },
      "page": {
        "country": "US",
        "server": "",
        "city": "Boydton",
        "domain": "zerodot1.gitlab.io",
        "ip": "52.167.214.135",
        "asnname": "MICROSOFT-CORP-MSN-AS-BLOCK - Microsoft Corporation, US",
        "asn": "AS8075",
        "url": "http://zerodot1.gitlab.io/",
        "ptr": ""
      },
      "uniq_countries": 1,
      "_id": "f1a037f5-a411-4aed-82cb-d8e3abc13e69",
      "result": "https://urlscan.io/api/v1/result/f1a037f5-a411-4aed-82cb-d8e3abc13e69"
    },
    {
      "task": {
        "visibility": "public",
        "method": "manual",
        "options": {},
        "time": "2018-06-23T16:51:10.981Z",
        "source": "web",
        "url": "https://zerodot1.gitlab.io/CoinBlockerListsWeb/"
      },
      "stats": {
        "uniqIPs": 2,
        "consoleMsgs": 0,
        "dataLength": 161933,
        "encodedDataLength": 74834,
        "requests": 11
      },
      "page": {
        "country": "US",
        "server": "",
        "city": "Boydton",
        "domain": "zerodot1.gitlab.io",
        "ip": "52.167.214.135",
        "asnname": "MICROSOFT-CORP-MSN-AS-BLOCK - Microsoft Corporation, US",
        "asn": "AS8075",
        "url": "https://zerodot1.gitlab.io/CoinBlockerListsWeb/",
        "ptr": ""
      },
      "uniq_countries": 1,
      "_id": "eb7a3c32-4a47-4b09-bc56-b0e4fd15f13d",
      "result": "https://urlscan.io/api/v1/result/eb7a3c32-4a47-4b09-bc56-b0e4fd15f13d"
    },
    {
      "task": {
        "visibility": "public",
        "method": "manual",
        "options": {},
        "time": "2018-06-18T14:26:52.230Z",
        "source": "web",
        "url": "https://globalrentals.gitlab.io/"
      },
      "stats": {
        "uniqIPs": 3,
        "consoleMsgs": 0,
        "dataLength": 1062648,
        "encodedDataLength": 1026483,
        "requests": 12
      },
      "page": {
        "country": "US",
        "server": "",
        "city": "Boydton",
        "domain": "globalrentals.gitlab.io",
        "ip": "52.167.214.135",
        "asnname": "MICROSOFT-CORP-MSN-AS-BLOCK - Microsoft Corporation, US",
        "asn": "AS8075",
        "url": "https://globalrentals.gitlab.io/",
        "ptr": ""
      },
      "uniq_countries": 2,
      "_id": "14ad053b-36ef-457b-87cb-75ebef127be2",
      "result": "https://urlscan.io/api/v1/result/14ad053b-36ef-457b-87cb-75ebef127be2"
    },
    {
      "task": {
        "visibility": "public",
        "method": "manual",
        "options": {},
        "time": "2018-04-26T00:52:09.350Z",
        "source": "web",
        "url": "http://xuhaiyang1234.gitlab.io"
      },
      "stats": {
        "uniqIPs": 2,
        "consoleMsgs": 0,
        "dataLength": 3906,
        "encodedDataLength": 3257,
        "requests": 1
      },
      "page": {
        "country": "US",
        "server": "",
        "city": "Boydton",
        "domain": "xuhaiyang1234.gitlab.io",
        "ip": "52.167.214.135",
        "asnname": "MICROSOFT-CORP-MSN-AS-BLOCK - Microsoft Corporation, US",
        "asn": "AS8075",
        "url": "http://xuhaiyang1234.gitlab.io/",
        "ptr": ""
      },
      "uniq_countries": 1,
      "_id": "07feb290-6ab3-4d51-9f56-b848b8c4bd37",
      "result": "https://urlscan.io/api/v1/result/07feb290-6ab3-4d51-9f56-b848b8c4bd37"
    },
    {
      "task": {
        "visibility": "public",
        "method": "manual",
        "options": {},
        "time": "2017-03-07T13:32:26.891Z",
        "source": "web",
        "url": "http://jorgegarciadev.gitlab.io"
      },
      "stats": {
        "uniqIPs": 2,
        "consoleMsgs": 0,
        "dataLength": 1424252,
        "encodedDataLength": 1424252,
        "requests": 34
      },
      "page": {
        "country": "US",
        "server": "",
        "city": "Boydton",
        "domain": "jorgegarciadev.gitlab.io",
        "ip": "104.208.235.32",
        "asnname": "MICROSOFT-CORP-MSN-AS-BLOCK - Microsoft Corporation, US",
        "asn": "AS8075",
        "url": "http://jorgegarciadev.gitlab.io/",
        "ptr": ""
      },
      "uniq_countries": 1,
      "_id": "2c3bca3b-3ef2-4ac7-8b8f-4e5ec5e85318",
      "result": "https://urlscan.io/api/v1/result/2c3bca3b-3ef2-4ac7-8b8f-4e5ec5e85318"
    }
  ],
  "total": 5
}

[ZeroDot1]

@picatz It is definitely possible to extract the subdomains from the Json results, but I don't know how useful this is at the moment.

[Ice3man543]

@ZeroDot1 its been done in the new version coming soon! Closing this now!