Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SPDY is deprecated. Switch to HTTP/2. #7452

Open
lavalamp opened this issue Apr 28, 2015 · 56 comments
Open

SPDY is deprecated. Switch to HTTP/2. #7452

lavalamp opened this issue Apr 28, 2015 · 56 comments
Assignees
Labels
kind/cleanup lifecycle/frozen priority/important-longterm sig/api-machinery sig/node

Comments

@lavalamp
Copy link
Member

lavalamp commented Apr 28, 2015

Filing for tracking purposes.

See #7392 where the spdy package we use doesn't even exist any more. (And btw it has bugs anyway, possibly security bugs-- see the trophies here: https://github.com/dvyukov/go-fuzz)

Blocked on github.com/docker/spdystream switching to HTTP/2.

(Note: HTTP/2 is based on SPDY, with just a few differences-- hopefully this transition will be easy.)

@thockin @ncdc

@lavalamp lavalamp added priority/backlog team/master sig/node labels Apr 28, 2015
@ncdc
Copy link
Member

ncdc commented Apr 28, 2015

Waiting on HTTP/2 support for Go that includes low level client/server ability to create and handle streams (https://github.com/bradfitz/http2).

moby/spdystream#53 pulls the upstream spdy code into spdystream.

@lavalamp
Copy link
Member Author

lavalamp commented Apr 28, 2015

Ah, after moby/spdystream#53 is merged, we can at least make godep happy again. :)

@davidopp davidopp added priority/awaiting-more-evidence and removed priority/backlog labels Apr 28, 2015
@ncdc
Copy link
Member

ncdc commented Apr 30, 2015

@lavalamp spdystream PR is merged, FYI

@ghost ghost removed the team/master label Aug 20, 2015
@ncdc
Copy link
Member

ncdc commented Dec 1, 2015

@thockin @lavalamp it looks like the Go HTTP/2 issue was just closed (golang/go#6891 (comment)). I glanced quickly through the code, but I don't see a way to hijack the connection and create our own streams, like we do with spdystream. Hopefully we can get this functionality in there at some point...

@ncdc
Copy link
Member

ncdc commented Dec 1, 2015

cc @smarterclayton

@ncdc
Copy link
Member

ncdc commented Dec 1, 2015

I created golang/go#13444 as an RFE for end-user stream support

@ddysher
Copy link
Contributor

ddysher commented Feb 23, 2016

What's the plan for the deprecation? We are using nginx in front of apiserver which drops support for SPDY since version 1.9.5. Our workaround is to use an older nginx, but would like to see http/2 support go upstream :)

@ncdc
Copy link
Member

ncdc commented Feb 23, 2016

I prototyped this with a prerelease version of go1.6. It worked somewhat, but there were some rough spots (probably stuff I was doing incorrectly). We can't move to http/2 until go1.6 at the earliest, and even then we'll have to do R&D to see if it's technically feasible. I had been hoping that the http/2 support in go1.6 would allow me to create streams just like we can do in spdystream, but that isn't exposed via the API. For my prototype, I ended up writing a simple multiplexer just like @smarterclayton wrote for the websocket support we have in Kubernetes now.

@ddysher
Copy link
Contributor

ddysher commented Feb 23, 2016

Thanks for the heads up. Looking at the linked comments, there is no intention to add the support in go1.6, so I guess it's probably still months away.

@bgrant0607 bgrant0607 added the sig/api-machinery label May 18, 2016
@JensRantil
Copy link

JensRantil commented Oct 29, 2016

We can't move to http/2 until go1.6 at the earliest

Golang 1.7 is out now. Does that mean we can make progress on this issue now? Anything else stopping us?

@smarterclayton
Copy link

smarterclayton commented Oct 29, 2016

A usable HTTP/2 framer in Go that allows us hijack control and the ability
to stand up the connection. Last I saw we were still blocked without
explicitly forking the Go implementation.

In the meantime, websockets continue to be an option (with portforward
being the last bit remaining).

On Oct 29, 2016, at 2:22 PM, Jens Rantil notifications@github.com wrote:

We can't move to http/2 until go1.6 at the earliest

Golang 1.7 is out now. Does that mean we can make progress on this issue
now? Anything else stopping us?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7452 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p-fe3BrfKe4VjAQyYLOgK7AjvhLkks5q447ngaJpZM4EK7sh
.

@ncdc
Copy link
Member

ncdc commented Oct 31, 2016

We should be able to use HTTP/2 without any changes to the Go implementation. But we would have to write our own code to do multiplexing of multiple data streams over a single request/response body pair, which we'd rather not do if we can avoid it.

@bradfitz
Copy link

bradfitz commented Nov 27, 2016

@ncdc, if it helps, https://godoc.org/golang.org/x/build/revdial is used by the Go build system to multiplex multiple net.Conns and a net.Listener all over a single net.Conn.

@fraenkel
Copy link
Contributor

fraenkel commented Nov 28, 2016

@bradfitz Thanks for the pointer, but it doesn't fix the underlying issue. While SPDY is being used, it is not following HTTP request/response semantics. A logical connection is created and then data is sent in either direction as needed. It would be equivalent to the client sending an HTTP request and the server doing a Push whenever.

@ncdc
Copy link
Member

ncdc commented Nov 28, 2016

@fraenkel I don't think there's any technical limitation in the facilities provided by the Go HTTP/2 implementation. As I wrote above, we would just need to write our own muxing/framing code, which we'd rather avoid (or find a library that can do it?).

@fraenkel
Copy link
Contributor

fraenkel commented Nov 28, 2016

@ncdc It might be easier to just build on top of gRPC which provides bi-directional streaming or copy what they have done.

@smarterclayton
Copy link

smarterclayton commented Nov 28, 2016

@luxas
Copy link
Member

luxas commented Jan 17, 2017

Ping, any update on this?

@wlan0
Copy link
Member

wlan0 commented Jan 20, 2017

We at Rancher Labs are also interested in this change.

@smarterclayton
Copy link

smarterclayton commented Jun 1, 2017

We can actually just move to web sockets in the near term. The server has supported it for a while and HOL blocking is not an issue

@elixxx
Copy link

elixxx commented Apr 7, 2020

Hey @brooksmtownsend i am sure switching to AWS NetworkLoadBalancer is the only solution at the moment.
I tried for days to get it running, ALB and ELB don't support SPDY and i can't imagine AWS will add support. So please safe time and use NLB.

@vespian
Copy link

vespian commented Jun 12, 2020

Is there any status update on this issue?

@librannk
Copy link

librannk commented Nov 29, 2020

any update on this ??

@matiasah
Copy link

matiasah commented Feb 3, 2021

Hi,

I have a setup where proxying kubectl exec is working, using openresty (docker image openresty/openresty:1.15.8.1-2-stretch to be precise).
It should also work with nginx as openresty is based on nginx.

The relevant config in the location block to have kubectl exec proxied correctly is to add:

              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection "Upgrade";

this is the listen line in the server block:

listen 443 http2 ssl

The proxied kube-apiserver is running 1.14.3-gke.11 (so, k8s 1.14 with gke modified version).

You have a proxy of apiserver over nginx? Is it possible to do the same with a nginx Ingress resource?

@landorg
Copy link

landorg commented Apr 9, 2021

You have a proxy of apiserver over nginx? Is it possible to do the same with a nginx Ingress resource?

... or kong? as it's based on openresty somehow.

@adisky
Copy link
Contributor

adisky commented Jun 24, 2021

/remove lifecycle-frozen
Some work is going on here #89163

@wgahnagl
Copy link
Contributor

wgahnagl commented Jun 24, 2021

still relevant! known tech debt but we don't have anyone working on fixing it :(

@mmiranda96
Copy link
Contributor

mmiranda96 commented Jun 25, 2021

/kind cleanup

@k8s-ci-robot k8s-ci-robot added the kind/cleanup label Jun 25, 2021
@arianitu
Copy link

arianitu commented Aug 10, 2021

Just want to mention that nginx does not support HTTP/2 when proxying. Are you guys sure you want to use HTTP/2?

Specifically:

client (http2) -> nginx -> (http2 does not work here) kube-api-server

We are running into essentially the same issue with our API (picking a good bidirectional streaming protocol) and we're trying to pick a good option that works well with proxying servers like nginx. We implemented HTTP/2 initially and then realized, oops, can't use it with nginx so now we are abandoning the implementation.

We are looking at replacing HTTP/2 with HTTP/1 + WebSockets which seem to work well with nginx.

@lavalamp
Copy link
Member Author

lavalamp commented Aug 10, 2021

Thanks -- at least some of our protocols are already available via websockets and that's a good reason to go all in on that.

Note that there is also an apiserver <-> kubelet / container runtime leg of some calls (exec, attach), and this also needs to not use SPDY any more, and it doesn't need to go through a proxy. So that needn't use websockets (although it could).

@sfxworks
Copy link

sfxworks commented Aug 11, 2021

I believe that would work out of the box with cloudflare

@slandelle
Copy link

slandelle commented Aug 11, 2021

some calls (exec, attach) ... and it doesn't need to go through a proxy

@lavalamp Why is that?

@danopia
Copy link

danopia commented Aug 11, 2021

I'm a user of Kubernetes so the below is my own viewpoint as an API consumer, not a Kubernetes contributor.

some calls (exec, attach) ... and it doesn't need to go through a proxy

Why is that?

When the API server makes HTTP calls to a particular kubelet, the connection is generally made without an HTTP proxy in the way; the kubelets each have their own TLS certificates and the apiserver checks this encryption, so installing an HTTP proxy in between would introduce MITM-like security concerns.

This connectivity of course may need to be relayed e.g. across the internet and this is done at the TCP level. For example GKE runs the apiserver->kubelet traffic through long-lived SSH tunnels, and the new system to replace those SSH tunnels, Konnectivity, is also a TCP-level proxy.


The client -> apiserver traffic is definitely more important for compatibility and this is where Websockets are already offered. Unfortunately the websocket interop isn't perfect:

  1. The new WebSocket() constructor in browsers don't allow attaching an Authorization header, so token auth isn't possible from an actual web client.
  2. Each WebSocket is historically its own TCP connection, so kubectl port-forward functionality would need to open a new TCP connection to the API Server for each new tunnel client if it were using WebSockets. RFC 8441 addresses this, but similarly, nginx seems to have no plans to implement websocket over HTTP2.

I've been working on a WebSocket client to Kubernetes and you can get things done with it. If the websocket subprotocol used by Kubernetes was extended to support functions like dynamically opening and closing port-forward streams, I could see it replacing SPDY. I just don't know if creating a more special-cased subprotocol in Websocket would be worth it vs. leveraging the http2 standard.

@lavalamp
Copy link
Member Author

lavalamp commented Aug 11, 2021

Thanks for the websocket feedback. I don't know if anyone is working on this right now, though. It may be a good idea to file a separate issue about that for visibility.

@pires
Copy link
Member

pires commented Aug 18, 2021

@danopia doesn't it make sense for the Authorization header go side-by-side with an Upgrade header in the same HTTP request to kube-apiserver and then, provided the requestor is authorized to proxy, upgrade to Websocket (not needing the Authorization header)? nevermind, it seems I misunderstood what you said.

@aeriksson
Copy link

aeriksson commented Sep 30, 2021

The new WebSocket() constructor in browsers don't allow attaching an Authorization header, so token auth isn't possible from an actual web client.

FYI it seems like there's a way to do this using the websocket subprotocol header: #47740

@danopia
Copy link

danopia commented Sep 30, 2021

FYI it seems like there's a way to do this using the websocket subprotocol header: #47740

That hack looks questionable at best, despicable or insecure at worst, and I'm sure not at all documented. (If the Kubernetes websocket subprotocols are documented somewhere I haven't seen it)

I've verified that the hack works for my usecase. Thanks for the link. 😄

@maradwan
Copy link

maradwan commented Jul 1, 2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/cleanup lifecycle/frozen priority/important-longterm sig/api-machinery sig/node
Projects
None yet
Development

No branches or pull requests