Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: SPDY error when accessing admin UI with chrome #5020

Closed
mberhault opened this issue Mar 9, 2016 · 13 comments
Closed

server: SPDY error when accessing admin UI with chrome #5020

mberhault opened this issue Mar 9, 2016 · 13 comments
Assignees
Milestone

Comments

@mberhault
Copy link
Contributor

mberhault commented Mar 9, 2016

between a3cff59 and 9136aef, hitting the admin page in secure mode fails in chrome with ERR_SPDY_PROTOCOL_ERROR.

I'll see if I can grab some network traffic. So far, nothing shows up in the debug window.

@tamird
Copy link
Contributor

tamird commented Mar 9, 2016

What do you mean "between"?

@petermattis
Copy link
Collaborator

petermattis commented Mar 9, 2016

Didn't Chrome remove support for SPDY? Perhaps you need to upgrade Chrome.

@mberhault
Copy link
Contributor Author

mberhault commented Mar 9, 2016

I mean between those two shas. Specifically, I'm guessing 3192a60 is the culprit.

@mberhault
Copy link
Contributor Author

mberhault commented Mar 9, 2016

I'm on 48.0.2564.116, but I'll restart, there is a new stable release of chrome. brb

@mberhault
Copy link
Contributor Author

mberhault commented Mar 9, 2016

No luck with latest stable: 49.0.2623.75

@tamird
Copy link
Contributor

tamird commented Mar 9, 2016

Seems likely.

Does this work correctly in insecure mode?

@mberhault
Copy link
Contributor Author

mberhault commented Mar 9, 2016

last event in the chrome net-trace:

5133: HTTP2_SESSION
localhost:26257 (DIRECT)
Start Time: 2016-03-09 13:18:20.509

t=1678 [st=0] +HTTP2_SESSION  [dt=1]
               --> host = "localhost:26257"
               --> proxy = "DIRECT"
t=1678 [st=0]    HTTP2_SESSION_INITIALIZED
                 --> protocol = "h2"
                 --> source_dependency = 5132 (SOCKET)
t=1678 [st=0]    HTTP2_SESSION_SEND_SETTINGS
                 --> settings = ["[id:3 flags:0 value:1000]","[id:4 flags:0 value:6291456]"]
t=1678 [st=0]    HTTP2_STREAM_UPDATE_RECV_WINDOW
                 --> delta = 15663105
                 --> window_size = 15728640
t=1678 [st=0]    HTTP2_SESSION_SENT_WINDOW_UPDATE_FRAME
                 --> delta = 15663105
                 --> stream_id = 0
t=1678 [st=0]    HTTP2_SESSION_SEND_HEADERS
                 --> fin = true
                 --> :method: GET
                     :authority: localhost:26257
                     :scheme: https
                     :path: /
                     pragma: no-cache
                     cache-control: no-cache
                     accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
                     upgrade-insecure-requests: 1
                     user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36
                     accept-encoding: gzip, deflate, sdch
                     accept-language: en-US,en;q=0.8,fr;q=0.6
                 --> priority = 0
                 --> stream_id = 1
                 --> unidirectional = false
t=1679 [st=1]    HTTP2_SESSION_CLOSE
                 --> description = "Framer error: 1 (INVALID_CONTROL_FRAME)."
                 --> net_error = -337 (ERR_SPDY_PROTOCOL_ERROR)
t=1679 [st=1]    HTTP2_SESSION_POOL_REMOVE_SESSION
                 --> source_dependency = 5133 (HTTP2_SESSION)
t=1679 [st=1] -HTTP2_SESSION

@mberhault
Copy link
Contributor Author

mberhault commented Mar 9, 2016

and yes, insecure mode is happy.

@tamird
Copy link
Contributor

tamird commented Mar 9, 2016

OK, I think I know what's going on here.

@tamird
Copy link
Contributor

tamird commented Mar 9, 2016

Another cmux bug.

@mberhault
Copy link
Contributor Author

mberhault commented Mar 9, 2016

interestingly, when listing HTTP2 sessions through chrome://net-internals, I see the failed ones (as listed above) in secure mode, but nothing shows up in insecure mode.

@tamird
Copy link
Contributor

tamird commented Mar 9, 2016

I'm guessing http1 is being used in secure insecure mode - chrome doesn't do h2c.

@tamird
Copy link
Contributor

tamird commented Mar 9, 2016

TL;DR: things are broken. Most http2 clients (chrome and go both included) behave as follows (after the preface):

  • send settings
  • send window update
  • wait for settings ack
  • send headers

grpc behaves differently: it doesn't wait for the settings ack. This means we can match grpc connections, but non-grpc H2 connections are broken - they will hang indefinitely while we wait for them to send us headers so we can figure out if they're grpc connections or not.

I'm still trying to work out a solution.

@petermattis petermattis modified the milestone: Beta Mar 9, 2016
tamird added a commit to tamird/cockroach that referenced this issue Mar 9, 2016
It turns out that accurately using cmux to sniff grpc connections is
hard. Most h2 clients behave as follows after connecting:

* S(Settings) -> S(WindowUpdate) -> R(SettingsAck) -> S(Headers)

Where S indicates sending and R indicates receiving.

grpc-go's h2 client behaves differently - it does not wait for the
SettingsAck. This allowed the old implementation to detect it based on
the headers it supplied, but broke all other h2 clients as connection
negotiation deadlocked.

This commit can be reverted wholesale when grpc.(*Server).ServeHTTP's
performance problems are addressed upstream.
See grpc/grpc-go#586.

Fixes cockroachdb#5020.
Fixes cockroachdb#5023.
tamird added a commit to tamird/cockroach that referenced this issue Mar 9, 2016
It turns out that accurately using cmux to sniff grpc connections is
hard. Most h2 clients behave as follows after connecting:

* S(Settings) -> S(WindowUpdate) -> R(SettingsAck) -> S(Headers)

Where S indicates sending and R indicates receiving.

grpc-go's h2 client behaves differently - it does not wait for the
SettingsAck. This allowed the old implementation to detect it based on
the headers it supplied, but broke all other h2 clients as connection
negotiation deadlocked.

This commit can be reverted wholesale when grpc.(*Server).ServeHTTP's
performance problems are addressed upstream.
See grpc/grpc-go#586.

Fixes cockroachdb#5020.
Fixes cockroachdb#5023.
tamird added a commit to tamird/cockroach that referenced this issue Mar 10, 2016
It turns out that accurately using cmux to sniff grpc connections is
hard. Most h2 clients behave as follows after connecting:

* S(Settings) -> S(WindowUpdate) -> R(SettingsAck) -> S(Headers)

Where S indicates sending and R indicates receiving.

grpc-go's h2 client behaves differently - it does not wait for the
SettingsAck. This allowed the old implementation to detect it based on
the headers it supplied, but broke all other h2 clients as connection
negotiation deadlocked.

This commit can be reverted wholesale when grpc.(*Server).ServeHTTP's
performance problems are addressed upstream.
See grpc/grpc-go#586.

Fixes cockroachdb#5020.
Fixes cockroachdb#5023.
tamird added a commit to tamird/cockroach that referenced this issue Mar 10, 2016
It turns out that accurately using cmux to sniff grpc connections is
hard. Most h2 clients behave as follows after connecting:

* S(Settings) -> S(WindowUpdate) -> R(SettingsAck) -> S(Headers)

Where S indicates sending and R indicates receiving.

grpc-go's h2 client behaves differently - it does not wait for the
SettingsAck. This allowed the old implementation to detect it based on
the headers it supplied, but broke all other h2 clients as connection
negotiation deadlocked.

This commit can be reverted wholesale when grpc.(*Server).ServeHTTP's
performance problems are addressed upstream.
See grpc/grpc-go#586.

Fixes cockroachdb#5020.
Fixes cockroachdb#5023.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants