Using HTTP instead of CURL

[sumgr0]

I've come across certain scenarios, where CURL would have a blank response while HTTP response confirms the sub-domain is redirected.

Basis the fingerprint by 0xpatrik for Amazon AWS at EdOverflow/can-i-take-over-xyz#36.

Wouldn't using HTTP instead of CURL result in fewer false positives?

[anshumanbh]

Maybe? This would need some digging and a vulnerable subdomain to compare both ways and seeing what results in a TP. Do you have an example?

[sumgr0]

I was looking at the status of brochure.belkin.com via tko-subs:
brochure.belkin.com,s3-website-us-west-1.amazonaws.com.,amazonaws,true,false,Can't CURL it but dig shows a dead DNS record

The curl's response was blank, while the HTTP returned:

HTTP/1.1 301 Moved Permanently
Content-Length: 0
Date: Tue, 28 May 2019 23:28:28 GMT
Location: http://www.belkin.com/us/belkinbusiness/
Server: AmazonS3
x-amz-id-2: 63dEzBVOsJaPo3cFk0zEsihspVuTYAxeGHLgsigexQe8nxCh/6l4LI0f3SVEehmIPHIhOAI3gmY=
x-amz-request-id: 627231689EBCB87D

This is just one of the examples I was looking and saw the false positive.

[anshumanbh]

Since it had been a while since we wrote this, I took a look at the code and found this - https://github.com/anshumanbh/tko-subs/blob/master/tko-subs.go#L475

It looks like we default to assume its vulnerable if curl doesn't return anything. I assume this is reasonable as compared to calling it not vulnerable.

@mhmdiaa what do you think?

[sumgr0]

@anshumanbh instead of curl, is it possible we test the http command. The outputs of both can then be compared to see lesser false-positives.

[mhmdiaa]

Hi @sumgro and @anshumanbh!

I'm going to explain the issue quickly to make sure we are on the same page.

The reason tko-subs assumes brochure.belkin.com is vulnerable is, as @anshumanbh said, we've defaulted to assuming domains are vulnerable if they don't respond over https (or http if specified in the providers-data.csv file). And when this domain is requested over https, it times out (although it responds over http).

I think @sumgro's suggestion is to request each domain over both http and https and call it vulnerable only if we don't get a response over either protocol, which is a reasonable solution although it could decrease performance if used for all providers.

I believe this is a good opportunity to rethink how we're doing things. @anshumanbh, is there a reason we use https by default here instead of http? I took a random sample of our known providers and all of them seem to respond normally over http (except for AWS type services where you can run a server on any port, basically), and some of them respond over http only as we know, so wouldn't http make more sense as the default protocol?

So my suggested solution is to use http by default and deprecate the "http" field of the providers' data CSV file for a new field that specifies whether a provider needs to be tested over both http and https (as @sumgro suggested).

Let me know what you think and I'll write a fix ASAP!

P.S. We don't actually use curl, the command-line utility, for http requests BTW. It's kind of a misnomer that has been there for a while since we used to use curl in the early versions of tko-subs. Currently, we use Go's http package. Admittedly, we should edit the code/documentation to reflect this fact.

[anshumanbh]

@mhmdiaa The only reason I can think of why I included the http field in the first place is because I believe it was Tumblr or Heroku (either one of these) that wouldn't respond properly on HTTP. It had to be sent over HTTPS. I think trying both should be a reasonable middle ground here. Don't think it would impact the performance by a lot. And, maybe we can remove the http field from the providers file all together?

[mhmdiaa]

@anshumanbh you mean Tumblr and Heroku wouldn't respond over https? (That's what the current providers-data.csv file indicates.) They do respond over http like most of the other providers.

Not sure about the performance impact. I'm going to have to run some performance tests and compare both approaches. But I mean, if we can avoid making that extra request in about 90% of the cases, this must be the better way.

[sumgr0]

Hey,

It’s been a fairly long time.

Checking back on any updates to testing use of https and http to resolve the targets.

Thanks.