HSTS blocks use of trusted.ci and cert.ci

[yaroslavafenkin]

Service(s)

cert.ci.jenkins.io

Summary

I'm getting TLS certificate errors when trying to access CERT CI instance. Previously there used to be an option to proceed anyway AFAIR, now it's not there.

Last line says:

You cannot visit cert.ci.jenkins.io right now because the website uses HSTS. 
Network errors and attacks are usually temporary, so this page will probably work later.

Reproduction steps

1. Connect to jenkins-infra VPN
2. Navigate to cert.ci.jenkins.io

[daniel-beck]

Same with trusted.ci on Chrome.

Don't know whether existing certificate exceptions are honored in Firefox or whether it's Chrome specific.

Proposed solution: Don't set the subdomain flag.

[timja]

I think it'll be from hitting the jenkins.io base domain which redirects to www.jenkins.io.

There's also:
#3091

[dduportal]

+1 with @timja : rolling back the HSTS setting would be painful, while we (infra team) can fix the root problem by providing a real life valid certificate to both controllers.

I do not have Chrome, but I confirm that I do not have the issue on my Firefox (if it helps on short term?)

[daniel-beck]

rolling back the HSTS setting would be painful

Does this include just not including subdomains?

[dduportal]

rolling back the HSTS setting would be painful

Does this include just not including subdomains?

I'm not sure to be honest. We need to check how much time it takes to the change to propagate (I remember HSTS settings having a loooong TTL making it not useful)

[dduportal]

@lemeurherve @smerle33 on short term, while checking HSTS settings, I'm going to generate manually an initial LE certificate and install it on the machines to unblock the JenSec team.

Renewal automation will come after once they're not blocked anymore

[daniel-beck]

how much time it takes to the change to propagate

If this is a good solution that should not be a blocker IMO (assuming there's workarounds like clearing browser cache), given there's just a few people affected (only trusted.ci / cert.ci users).

[dduportal]

jenkins-infra/jenkins-infra#2572 show the "short term but persistent" change I've applied to configure Apache to use manually-generated LE certificates for both server using DNS challenge.

@yaroslavafenkin @daniel-beck is it working for you now with Chrome?

for the infra-team: we now have to configure renewal for both machines (e.g. setting up Azure API credential limited to the DNS zone, and set up the puppet letsencrypt module to automatically renew using Azure DNS and these credentials).

(edit) worst case: certbot renew on each machine in 2.5 months + update of the Azure TXT record.

[dduportal]

Closing this issue as the Jenkins Security team confirmed they are unblocked.

Next steps:

• Automate LE certificate for trusted.ci.jenkins.io (outside of Azure) tracked in Valid ssl certificate for trusted.ci.jenkins.io #3091
• Automate LE certificate for cert.ci.jenkins.io (inside Azure) tracked in Valid ssl certificate for cert.ci.jenkins.io #3337