Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Post: Stack Overflow: the long road to HTTPS #26

Closed
NickCraver opened this issue Mar 11, 2017 · 26 comments
Closed

New Post: Stack Overflow: the long road to HTTPS #26

NickCraver opened this issue Mar 11, 2017 · 26 comments

Comments

@NickCraver
Copy link
Owner

NickCraver commented Mar 11, 2017

I need to do a post to follow-up on Stackoverflow.com: the road to SSL. This is to both track things so I remember and a place for people to add suggestions.

  • Previous post update
  • Priorities
    • What took so long?
  • Why?
    • Why go HTTPS at all?
    • HTTP2?
    • Why not Let's Encrypt?
    • CloudFlare/Fastly
    • Search ranking
    • Mitigation
    • Why move third-party things and not proxy?
      • What went wrong when we tried
  • Changes along the way
    • NPN/ALPN
    • HTTP/2
    • CloudFlare/Fastly
    • Railgun
  • Measuring
    • Logging
    • ClientTimings
  • Setup
    • Reference architecture post
    • HAProxy
    • Ciphers (Mozilla)
    • Combined certificates
    • Dev certs
    • Multiple Domains
    • IP Pooling (HTTP/1.1)
    • .internal structure
    • New SVG diagrams!
    • Caching/Not-caching
  • Related Moves
    • Blog
    • SendGrid
    • Hubspot
    • Cookie security
  • Unknowns
    • Google Analytics
    • Google Webmasters (damn captchas!)
    • Search analytics/impact
    • Scary search gaps
  • Migration Items
    • Wildcards
    • Meta.* domains
      • Search and rebake code
      • How per-site redirects work
    • Global Login
    • StackExchange.com Sites DB (and history)
      • Newsletters
      • Tag Sets
    • Area 51
    • Code Assumptions
    • Static links
    • Local setup for devs
    • Local vs. Dev vs. Prod
    • Websockets
    • Ads
      • Calculon
      • AdZerk
      • Tag Sponsorships
    • Redirects for http://
    • Mixed content
      • Why protocol-relative URLs don't really work
        • Mobile app
        • Email
      • Content-Security-Policy-Report-Only header
      • Posts
        • Migrating existing
        • Preventing new http://
        • Handling non-https:// existing
        • Images/YouTube
      • AboutMe profiles
      • Image proxy research
      • Help Center
      • Privileges
      • Avatars
        • ...and chat
        • ...and Area 51
      • Dev Stories
      • /jobs
        • Job postings
        • Company pages
      • StackSnippets
      • Per-site JavaScript includes
      • Trackers (Quantcast/GA)
    • Rendering Code
      • All image paths
      • APIv2
      • Tier unification
      • Cache breaker simplification
      • Default HTTPS
      • Redirects (301/302)
      • All .Url() calls
      • Migration of .Url()
      • Flair
      • JavaScript http:// assumptions. Oh so many.
      • Search engine fixes (<a>/<img>)
  • Launch steps
    • Prep everything
    • Migrate local
    • Migrate dev
    • Migrate prod
      • Meta.SO
      • Meta.SE
      • Security.SE, Super User
      • ...wait on Google...
      • Everything else
  • Problems Migrating
    • Revocation of dev cert & races
    • 301 Proxy Caching (VCL)
    • Help Center SNAFU
    • We're not done yet, I bet good money we'll add to this list
  • Performance
    • Client Timings
    • Server performance
    • HTTP/2 CDN performance
  • What comes next
    • HSTS (HTTP Strict-Transport-Security)
      • Migration/escalation plan
    • HPKP (HTTP Public Key Pinning)
    • HSTS preloading
      • Chromium addition
    • Secure cookie move
    • HTTP/2 to origin
      • Push content
  • Open source projects spawned

Please leave comments about anything I'm forgetting to cover. I want this to be informative now just about what we did, but also what we didn't do, to help anyone else in their https:// migration.

@AKTheKnight
Copy link

Based on what you and the team went through migrating StackOverflow, what advice regarding HTTPS would you offer to others looking to migrate or those creating websites (HTTPS by default?)?

@NickCraver
Copy link
Owner Author

@AKTheKnight I think that's the overall goal of the post - to convey everything we learned along the way, so that people can learn from our mistakes and not repeat them. Hopefully, some tricks we learn will help.

@daniel-white
Copy link

What about the mobile app?

@nikoskip
Copy link

Do SLL impact on performance (hardware/network)?

@NickCraver
Copy link
Owner Author

@daniel-white Mobile has always been over HTTPS, but it has some fun interactions from the fact of not being a browser - I'll cover some bits in the protocol-relative downsides section, thanks!

@nikoskip Not really at this point, but I'll detail metrics since this will come after the rollout so we can compare real numbers here.

@JakeDEvans
Copy link

Obviously you must have alpn for http2 support, but what differences did you see between alpn and npn...did you specifically wait for alpn support in your upstream. What OS are you running HAProxy on, did you build your own openssl binary.

@NickCraver
Copy link
Owner Author

@JakeDEvans we didn't specifically wait as we were blocked by a great many other things listed above, but it was interesting to see how that transition happened at the provider level (Fastly/CloudFlare), since we're a single-active-datacenter operation and need the local termination endpoints.

I'll add to the list above to cover this in the post - thanks!

@dexterdtg
Copy link

What you let fastly/cloudflare cache/not cache and for how long with regards to dynamic content. Any cache invalidation when questions get answered etc or all requests passed.

@Eng-Fouad
Copy link

Will the old http URLs be redirected automatically to https?

@MarcinHoppe
Copy link

Given that you are looking at HSTS, I'd like to learn if you look into pinning (HPKP) and if you use/plan to use OCSP stapling (or how do you otherwise handle possible revocation of your certs).

Thanks for all the posts so far!

@NickCraver
Copy link
Owner Author

@dexterdtg @Eng-Fouad @MarcinHoppe All added to items to cover, thanks!

Short answers on redirects: yes, the redirects are already in place as we migrate sites - that's actually an old mechanism for renaming sites we had in place that was updated a bit for https://

@bmaia
Copy link

bmaia commented Mar 13, 2017

Just a small suggestion, the Content-Security-Policy-Report-Only header is really useful to spot mixed-content. Washington Post used it during their HTTPS migration:

https://developer.washingtonpost.com/pb/blog/post/2015/12/10/moving-the-washington-post-to-https/

@NickCraver
Copy link
Owner Author

@bmaia honestly we forgot about that phase of the rollout after the full conversion (since our content should be there), but still useful for ads - added to the list and I'll get to it this week.

@kelunik
Copy link

kelunik commented Mar 14, 2017

What's your plan for user posted images? Proxy them like GitHub does?

@daniel-white
Copy link

@kelunik theres a community project to move them and update posts

@kelunik
Copy link

kelunik commented Mar 14, 2017

@daniel-white What's with images in chats? Will non-HTTPS images just no longer be one-boxed?

@NickCraver
Copy link
Owner Author

@kelunik On the sites, users can no longer post http:// embeds, so it's only a matter of cleanup on the relatively few we weren't able to move to https://. As for chat - I don't honestly know yet. There are so many moving pieces and dependencies in this chain to deploy that I don't have the mental capacity to think through chat yet. It's next up on our list after we finish the sites.

I'll be sure to cover some thoughts in the post, if I can turn them into coherent sentences :)

@jaxidian
Copy link

I've always been terrified of HSTS. Firstly, I don't really have any experience with it from the sys admin side of the world. Secondly, I fully understand how it's a one-way path where you essentially hit a point of no return once you flip that switch publicly.

How does one prepare to flipping that switch?

I know you can internally/privately flip that switch with custom DNS and test, but that's a pretty large amount of testing required and I have a feeling SO isn't going to commit (or even has) that many resources to thoroughly test it.

I know you could crank up the analytics/metrics to try to verify that you can use logs to detect anything/everything not yet moved over to HTTPS but I have a feeling getting 100% coverage with your analytics would be tough or even impossible.

What other strategies are there since any kind of public trial is nearly impossible since you can't really unflip that switch?

@JakeDEvans
Copy link

@jaxidian hsts is just another header tag and expires like any other, you can set hsts on and make the header good for 1 hour if you're not confident yet, it's really just to protect from MITM where you flip between a trusted and untrusted network, but retain that the site should be https

@jaxidian
Copy link

@JakeDEvans I assume you're talking about the max-age field of the HSTS header.

However, one problem for me is that I would like to leverage Cloudflare's HSTS toggle for controlling this. Maybe this is a terrible idea, especially given all of the "dragons lie here" warnings, but that is one thing I'd like to enable. Again, a major reason why I'm afraid of HSTS. The last thing I need is all of my users to have a value set for 6 months on a domain that isn't quite ready and now I have 6 months of downtime (in reality, I'm not sleeping until I get that/those systems back up so it'll be far less than 6 months but you get the idea).

I suspect a better idea is to tackle this on a per-domain/application basis and then only turn on the Cloudflare toggle once I believe I have handled every single application.

So you're saying I can deploy these live with a 1-hr max-age setting in order to limit my downtime to just 1 hour?

One other problem: Not all of my applications have their own domain. This means those applications now have to be handled as a batch, which greatly increases risk around downtime associated with HSTS.

@Ajedi32
Copy link

Ajedi32 commented Mar 21, 2017

@jaxidian Yes, Cloudflare supports the max-age header just fine. Also, if you're using Cloudflare I'm pretty sure you can just flip flexible SSL on right now and instantly start encrypting all connections between the client and Cloudflare (with or without the HSTS header). Later you can also set up encryption between your servers and Cloudflare's via the usual process of installing certs and setting up encryption on your servers, then migrating to "full SSL" on Cloudflare.

@kelunik
Copy link

kelunik commented Mar 21, 2017

@Ajedi32 No, you can't just flip flexible TLS on. It's just the same as switching it completely on without Cloudflare.

@Ajedi32
Copy link

Ajedi32 commented Mar 21, 2017

@kelunik Not exactly, since with Cloudflare:

  1. You don't have to manually configure anything on your servers to enable TLS; just flip the switch
  2. You don't have to obtain a cert yourself; Cloudflare will automatically get one for you
  3. You don't have to update your app to use HTTPS links or images; Cloudflare will transparently rewrite those URLs for you.

For all but the most complex setups I think that should make setting up basic, flexible SSL pretty much a turn-key operation for those using Cloudflare. Obviously you should ideally be using full SSL, not flexible SSL, but Cloudflare's solution seems like a good first step IMO.

@kelunik
Copy link

kelunik commented Mar 21, 2017

@Ajedi32 The points you mentioned are really the easiest issues one has to tackle. It's much harder to get rid of all existing mixed content (which might be hosted on different domains) and things like that.

@NickCraver
Copy link
Owner Author

NickCraver commented May 22, 2017

I didn't cover everything because it got a bit long at 12,000 words - but this is now live: https://nickcraver.com/blog/2017/05/22/https-on-stack-overflow/

@NickCraver
Copy link
Owner Author

Closing out to cleanup!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests