Teleport UI login error #20963
Comments
|
If using systemd, the signal sent to the Teleport process on restart is
Ref: https://goteleport.com/docs/reference/signals/ If performing an upgrade of Teleport, while the Proxy is still in rotation and has not been fully drained of connections, the process may hang until all clients disconnect. If the load balancer attempts to send connections to this Proxy, the following error message will be seen in the Web UI upon authentication. The proper way to upgrade is to remove the Proxy from the load balancer, drain all connections, and then upgrade the Proxy instance. |
|
Talked with @zmb3 about this. We're thinking about handling this in two ways.
@travelton Because this is a nice to have, we're not scheduling this right now. We'll keep it in mind as a good starter issue for the future. |
russjones
added
the
good-starter-issue
|
@russjones @zmb3 customer checked/verified the systemd services and states all the processes/services are fine, they are not attempting to restart. |
|
A little more information. We originally upgraded from 9.x -> 10.x -> 11.x. The issue started happening in 10.x and 11.x. The proxy and auth servers are all new instances. The upgrade process is as followed:
The autoscaling group removes and terminates the old instance first before launching the new instance. |
rosstimothy
added a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
rosstimothy
added a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
github-actions bot
pushed a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
rosstimothy
added a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
rosstimothy
added a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
rosstimothy
added a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
rosstimothy
added a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
rosstimothy
added a commit
that referenced
this issue
Mar 28, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
rosstimothy
added a commit
that referenced
this issue
Mar 29, 2023
Closes #23533 and #20963. There was a race to create the `web.SessionContext` for a session when multiple Proxies are behind a load balancer. Only the Proxy that processes the login will have a `web.SessionContext` created for the session. Any subsequent requests to the other Proxies in the pool would create one if the request was authenticated. However, multiple requests within a short succession could cause a single Proxy to create multiple `web.SessionContext` for a single session. When that happens the most recently created `web.SessionContext` gets saved and the previous `web.SessonContext` gets closed. Closing causes the `auth.Client` to be closed, which causes any active requests for that client to return with a `grpc: client connection is closing` error. This manifests in a single request from the web UI to fail and depending on the request, for a banner to be displayed with the error. Refreshing the page or navigating to another page would resolve the problem because the most recent `web.SessionContext` would be used with the still open `auth.Client`. To prevent `web.Handler.AuthenticateRequest` from racing to create the `web.SessionContext` a `singleflight.Group` was added to the `web.sessionCache`. When multiple requests come in for the same session they now will all use the first `web.SessionContext` to be created instead of each creating their own.
|
Fixed in #23691. |
|
Is it possible to know what versions will include fix #23691? |
|
@kelcya the next v10, v11, and v12 release will contain the fix. That will be v12.1.3, v11.3.10, and v10.3.15 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected behavior:
Log in to your Teleport cluster thru the UI without any issues (Okta SAML).
Current behavior:
When you log in to the teleport cluster via WebUI (Okta SAML), it sometimes greets you with the error below:
"Internal error - rpc error: code = Canceled desc = grpc: the client connection is closing"The error goes away when you refresh the page.
Bug details:
***Note: Video of the issue observed:
Bug Video
Debug logs:
Auth Server logs:
Okta-SAML config:
Screenshot of developer tool error on a check:
We also have a HAR file upon request.
Extras:
The text was updated successfully, but these errors were encountered: