Slow auth server can hang client connections in async recording mode #4695
Assignees
Comments
|
Is this resolved by restarting the auth server, or the node? I think I may have seen this issue on a node running 4.4.2 yesterday - new sessions would hang and not start with no obvious errors in the log, but this was resolved after restarting the node. |
|
@webvictim node |
|
Just had this happen again. Here's debugging info in case it's relevant. Version on auth/proxy: Version on node: Auth/proxy logs: The node is running at |
|
@webvictim do you use the |
|
@webvictim can you also set env vars on the node that hangs You should see a lot of GRPC state logs |
|
Hanging connections: grpc/grpc-go#3980 |
|
Check keepalives server vs client: |
|
@klizhentas Yes, I use IoT mode. I’ll set the variables and see if I can get extra debugging info. |
|
@klizhentas I set those variables in Teleport's environment (via |
klizhentas
added a commit
that referenced
this issue
Nov 6, 2020
This commit fixes #4695. Teleport in async recording mode sends all events to disk, and uploads them to the server later. It uploads some events synchronously to the audit log so they show up in the global event log right away. However if the auth server is slow, the fanout blocks the session. This commit makes the fanout of some events to be fast, but nonblocking and never fail so sessions will not hang unless the disk writes hang. It also adds ability to debug GRPC connection state when running in debug mode. To start sending GRPC connection state logs, set environment variables: GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=99 teleport start -d
|
@webvictim the variables were ignores, see my PR, I fixed them in debug buld |
klizhentas
added a commit
that referenced
this issue
Nov 9, 2020
This commit fixes #4695. Teleport in async recording mode sends all events to disk, and uploads them to the server later. It uploads some events synchronously to the audit log so they show up in the global event log right away. However if the auth server is slow, the fanout blocks the session. This commit makes the fanout of some events to be fast, but nonblocking and never fail so sessions will not hang unless the disk writes hang. It adds a backoff period and timeout after which some events will be lost, but session will continue without locking. It also adds ability to debug GRPC connection state when running in debug mode. To start sending GRPC connection state logs, set environment variables: GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=99 teleport start -d
klizhentas
added a commit
that referenced
this issue
Nov 12, 2020
This commit fixes #4695. Teleport in async recording mode sends all events to disk, and uploads them to the server later. It uploads some events synchronously to the audit log so they show up in the global event log right away. However if the auth server is slow, the fanout blocks the session. This commit makes the fanout of some events to be fast, but nonblocking and never fail so sessions will not hang unless the disk writes hang. It adds a backoff period and timeout after which some events will be lost, but session will continue without locking.
klizhentas
added a commit
that referenced
this issue
Nov 12, 2020
This commit fixes #4695. Teleport in async recording mode sends all events to disk, and uploads them to the server later. It uploads some events synchronously to the audit log so they show up in the global event log right away. However if the auth server is slow, the fanout blocks the session. This commit makes the fanout of some events to be fast, but nonblocking and never fail so sessions will not hang unless the disk writes hang. It adds a backoff period and timeout after which some events will be lost, but session will continue without locking.
klizhentas
added a commit
that referenced
this issue
Nov 14, 2020
This commit fixes #4695. Teleport in async recording mode sends all events to disk, and uploads them to the server later. It uploads some events synchronously to the audit log so they show up in the global event log right away. However if the auth server is slow, the fanout blocks the session. This commit makes the fanout of some events to be fast, but nonblocking and never fail so sessions will not hang unless the disk writes hang. It adds a backoff period and timeout after which some events will be lost, but session will continue without locking.
russjones
pushed a commit
that referenced
this issue
Nov 14, 2020
This commit fixes #4695. Teleport in async recording mode sends all events to disk, and uploads them to the server later. It uploads some events synchronously to the audit log so they show up in the global event log right away. However if the auth server is slow, the fanout blocks the session. This commit makes the fanout of some events to be fast, but nonblocking and never fail so sessions will not hang unless the disk writes hang. It adds a backoff period and timeout after which some events will be lost, but session will continue without locking.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Slow or unresponsive auth service can hang client connections in async recording mode.
What happens
Teleport async recording mode sends all events to disk, and uploads them to the server later.
It uploads some events synchronously to the audit log so they show up in the global event log right away.
However if the auth server is slow, the fanout blocks the session.
The text was updated successfully, but these errors were encountered: