Measure whether a domain is actually in the Chrome preload list

[konklone]

Get the Chrome preload list from source control somehow, and measure whether the domain is actually in there. This can help eliminate the gap between what we think is preload-ready and what actually made it.

[garrettr]

@konklone Do you think it would be worth implementing this upstream in site-inspector?

[konklone]

cc @benbalter to get his thoughts. My suspicion is no, since it's looking up the domain's status in an external source, rather than inspecting the domain's technical configuration.

[benbalter]

Assuming we can write a script to wget the HSTS preload list and vendor it into the Gem, I'd gladly merge a pull request that adds a check as to whether the domain is on the list.

[konklone]

@benbalter Cool, that's only upside for site-inspector. @garrettr contributed Python code to do exactly this in our scanner: https://github.com/18F/domain-scan/blob/master/scanners/inspect.py#L18-L41

One crucial aspect of it for batch scans is caching the HSTS list once, at the top of the scan, to avoid redownloading/reparsing it once for every domain.