Added Urlscan as passive source

[CaptainFreak]

This needs a lot of work.
@Ice3man543 @codingo can you please give me a developer setup and testing guide for testing out the new features while adding them. also a simple to-do list of making this a complete PR would be good. as i have noticed, due to such modularity of this project there are multiple places where i will have to make changes to completely plug in the new source. This PR just contains a very initial work for right now.

[Ice3man543]

@CaptainFreak Thanks for this Pull request.

[Ice3man543]

First, you need to import the source here -
https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/engines/passive/passive.go#L54-L56)

Next, add a boolean flag for this source here - https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/engines/passive/passive.go#L93-L94

https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/engines/passive/passive.go#L126-L127 - Set s.Urlscan = true here

https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/engines/passive/passive.go#L186-L194 - Add a switch here with true statement, https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/engines/passive/passive.go#L257-L262 - add one here in "all" block and one in a case "urlscan" block.

Then a if and printf here https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/engines/passive/passive.go#L383-L385

Add to domainDiscoverPool here - https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/engines/passive/passive.go#L560-L562

https://github.com/subfinder/subfinder/blob/37367bfe060cb8f6b4cc249c5f06539b36e8eecb/libsubfinder/helper/state.go#L51-L66 - add your api key field here. This should be the minimum required lol.

[CaptainFreak]

Looks good ? @Ice3man543

[Ice3man543]

@CaptainFreak looks okay 👍

I will test it tonight once i am free.

[Ice3man543]

@CaptainFreak The urlscan api doesn't works that way. What we actually need to do here is make a request to get all the pre-searched urls that contain domain as uber.com

See this - #82 (comment)

Instead what you are doing is submitting a url for scanning, that is not the purpose i think. Can you change this to be that way?

[ZeroDot1]

This looks interesting, why isn't it in use yet?

[Ice3man543]

It isn't complete yet. The pr doesn't works. And the core is somehow messed up. Don't worry, we'll be releasing something way better soon. Sorry for the delay @ZeroDot1

[ZeroDot1]

@Ice3man543 No problem, many thanks for the information, good luck for the future work.

[ehsandeep]

Thanks for the pull request, it is added in the latest version.