Potential bug on canonical URL detection

[konklone]

We heard from NASA that in the situation where the HTTP endpoints are off, but the HTTPS endpoints have cert errors, the "canonical URL" is the HTTP version, which is confusing. I haven't verified yet.

cc @egyptiankarim for details or an example hostname

[IanLee1521]

I think I might have stumbled upon a way to reproduce this, or something very similar as I was working on my new github.com/llnl/hscs repo.

In the el7/ directory in that repo is a Vagrantfile to spin up a CentOS 7 + Apache 2.4 configuration (it's still a work in progress and I need to document things in the readme a bit better still). If you then try to scan the system (I have my local /etc/hosts file set to point vagrant.local -> 192.168.33.10), you'll get the behavior you describe:

$ ~/.virtualenvs/pshtt/bin/pshtt --json 192.168.33.10
Error validating certificate.
[
  {
    "Base Domain": "33.10",
    "Canonical URL": "http://192.168.33.10",
    "Defaults to HTTPS": false,
    "Domain": "192.168.33.10",
    "Domain Enforces HTTPS": null,
    "Domain Supports HTTPS": null,
    "Domain Uses Strong HSTS": false,
    "Downgrades HTTPS": false,
    "HSTS": false,
    "HSTS Entire Domain": null,
    "HSTS Header": null,
    "HSTS Max Age": null,
    "HSTS Preload Pending": false,
    "HSTS Preload Ready": false,
    "HSTS Preloaded": false,
    "HTTPS Bad Chain": null,
    "HTTPS Bad Hostname": true,
    "HTTPS Expired Cert": null,
    "Live": true,
    "Redirect": true,
    "Redirect To": "https://vagrant.local/",
    "Strictly Forces HTTPS": true,
    "Valid HTTPS": null,
    "endpoints": {
      "http": {
        "headers": {
          "Connection": "Keep-Alive",
          "Content-Length": "206",
          "Content-Type": "text/html; charset=iso-8859-1",
          "Date": "Fri, 02 Jun 2017 05:22:11 GMT",
          "Keep-Alive": "timeout=5, max=100",
          "Location": "https://vagrant.local/",
          "Server": "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips"
        },
        "live": true,
        "redirect": true,
        "redirect_eventually_to": "https://vagrant.local/",
        "redirect_eventually_to_external": true,
        "redirect_eventually_to_http": false,
        "redirect_eventually_to_https": true,
        "redirect_eventually_to_subdomain": false,
        "redirect_immediately_to": "https://vagrant.local/",
        "redirect_immediately_to_external": true,
        "redirect_immediately_to_http": false,
        "redirect_immediately_to_https": true,
        "redirect_immediately_to_subdomain": false,
        "redirect_immediately_to_www": null,
        "status": 302,
        "url": "http://192.168.33.10"
      },
      "https": {
        "headers": {
          "Accept-Ranges": "bytes",
          "Connection": "Keep-Alive",
          "Content-Length": "9",
          "Content-Type": "text/html; charset=UTF-8",
          "Date": "Fri, 02 Jun 2017 05:22:13 GMT",
          "ETag": "\"9-550f2d3aeab7a\"",
          "Keep-Alive": "timeout=5, max=100",
          "Last-Modified": "Fri, 02 Jun 2017 04:45:47 GMT",
          "Server": "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips",
          "Strict-Transport-Security": "max-age=31536000"
        },
        "hsts": false,
        "hsts_all_subdomains": null,
        "hsts_header": null,
        "hsts_max_age": null,
        "hsts_preload": null,
        "https_bad_chain": null,
        "https_bad_hostname": true,
        "https_expired_cert": null,
        "https_valid": null,
        "live": true,
        "redirect": null,
        "redirect_eventually_to": null,
        "redirect_eventually_to_external": null,
        "redirect_eventually_to_http": null,
        "redirect_eventually_to_https": null,
        "redirect_eventually_to_subdomain": null,
        "redirect_immediately_to": null,
        "redirect_immediately_to_external": null,
        "redirect_immediately_to_http": null,
        "redirect_immediately_to_https": null,
        "redirect_immediately_to_subdomain": null,
        "redirect_immediately_to_www": null,
        "status": 200,
        "url": "https://192.168.33.10"
      },
...
}

Namely, there is a cert error, pshtt is detecting the redirect to HTTPS:

    "Redirect To": "https://vagrant.local/",
    "Strictly Forces HTTPS": true,

But the canonical url shows as http:

    "Canonical URL": "http://192.168.33.10",

There are also a whole bunch of nulls but best I can tell those are related to the cert issues (still working on that one).

[konklone]

Closing without a reproducible public hostname to measure this on. @egyptiankarim if this is still an issue, please feel free to re-open with an example.

[egyptiankarim]

Gah! Sorry. This fell off my todo list. Here're a few annotated examples I dug up:

• mobile.neacc.nasa.gov - Nmap shows only 443 is listening; queries to the HTTP endpoint timeout; queries to the HTTPS endpoint resolve after some time; pshtt shows "Live = False"; pshtt shows "Canonical URL = http:// mobile.neacc.nasa.gov"; I believe this is a pshtt timeout issue in that it's counting the target as not live, but even if that were the case (which it isn't) the canonical URL is wrong; because this target is being detected as "Live = False" this is closer to If the domain is not reachable over HTTP, canonical URL should be blank #34, and I was mischaracterizing it when I first created this issue.
• msonce-a.ndc.nasa.gov - Nmap shows only 443 is listening; queries to the HTTP endpoint timeout; queries to the HTTPS endpoint resolve; pshtt shows "Live = True"; pshtt reveals certificate error (resulting from a domain name mismatch); pshtt shows "Canonical URL = http:// msonce-a.ndc.nasa.gov; this matches the original problem case pretty clearly, I think.

I used to have more examples, but a lot of them have fixed their certificates and so don't match the problem case anymore. I think ultimately this has to do with the order of tests being performed and the way we keep variables up-to-date as we go through them.

[konklone]

Thank you, we can work with that!