Change hostmap-crtsh to split SANs on newline

[mimi89999]

No description provided.

[mimi89999]

@bonsaiviking Could you please review?

[nnposter]

Besides the need to split the SANs, the script deserves a broader refresh:

• Ignore wildcard certificates
• Use the JSON parser
• Additional error checking
• Remove unused variables

Please give it a try and report back.

[mimi89999]

It works as expected, but I have a couple of thoughts:

• Ignore wildcard certificates

I think that while they shouldn't be added to targets, it might still be useful to report them.

• Use the JSON parser

Does it make sense to parse the entire result if we only use one field?

Sometimes there might be very strange things in certs and there can be basically anything besides domains and email addresses. There might be pure garbage or worse. See https://crt.sh/?id=312556089 as an example, be there are much worst. Certs in the CT log don't have to be signed by a trusted CA, so that field can't be trusted to not break completely Nmap parser logic.

[mimi89999]

That one is great: https://crt.sh/?id=3612039160

[nnposter]

Regarding reporting wildcard certificates, this feels like a mission creep. The purpose of the script is stated as "Finds subdomains of a web server by querying...". That said, I have made change in the code where the script will now try to use the static portion of a wildcard name. This means that *.foo.bar.com found in the context of bar.com will report foo.bar.com. On the other hand, *.bar.com will not be reported, because it already matches existing bar.com; no new subdomain is inferred.

I have added a hostname validation code to skip over entries other than hostnames. I agree that the name_value field cannot be trusted but if it is possible to "...break completely Nmap parser logic" then we have a bigger issue that needs to be addressed by fixing the JSON parser itself. If you are aware of such case then please submit it as a new issue.

I do not get your point about https://crt.sh/?id=3612039160. Its name_value seems pretty clean to me:

{
  "issuer_ca_id":51379,
  "issuer_name":"C=BR, O=ICP-Brasil, OU=Secretaria da Receita Federal do Brasil - RFB, CN=Autoridade Certificadora do SERPRORFB SSL",
  "common_name":"sso.supop.serpro",
  "name_value":"sso.supop.serpro",
  "id":3612039160,
  "entry_timestamp":"2020-11-06T18:12:54.017",
  "not_before":"2020-11-06T18:12:52",
  "not_after":"2021-10-10T19:05:08",
  "serial_number":"66d0250862febba63cbf4454"
}

What am I missing?