Can the Coder agent immediately realize it's disconnected? #5901
Comments
|
This is something I could look at before tackling #4677 to add shutdown lifecycle states. It should be fairly minor:
|
This change allows the agent to handle common shutdown signals like interrupt, hangup and terminate and initiate a graceful shutdown. As long as terraform providers initiate graceful shutdowns via the aforementioned signals, things like SSH connections will be closed immediately on shutdown instead of being left hanging/timing out due to the agent being abruptly killed. Refs: #4677, #5901
This change allows the agent to handle common shutdown signals like interrupt, hangup and terminate and initiate a graceful shutdown. As long as terraform providers initiate graceful shutdowns via the aforementioned signals, things like SSH connections will be closed immediately on shutdown instead of being left hanging/timing out due to the agent being abruptly killed. Refs: #4677, #5901
|
@mafredri is there another part to this? I noticed that the dashboard still reports the agent is online Screen.Recording.2023-02-10.at.1.49.00.PM.mov |
|
@bpmct if the agent is getting killed abruptly, then there’s not much we can do except wait for the timeout to detect the agent is disconnected. Recent changes should improve conditions to avoid being killed abruptly, but it’s still possible. Could you share how the agent is being run? That could belp discern how it is being stopped. Logs from the agent might also help. Ultimately, if the agent is receiving SIGINT or SIGTERM, then disconnection should work. You can try this via |
|
It's being started via user-data, same as our aws-linux example. What is the timeout for the agent at this point? How long would it take for the server to realize an agent looses connection after an agent is killed abruptly? |
|
I can evaluate the relevant timeouts on Monday, don’t know off the top of my head (I believe they may also be configurable). In the meantime, can you verify whether |
|
@bpmct I'm trying to reproduce this issue (not on AWS), but each time I try to kill the agent, gracefully or abruptly ( That is, it should be fixed in 2dbe00a. The thing is, I looked at your recording and it's showing your agent is based on commit 2dbe00a, and I have no idea why it's not working for you 🤔. |
|
So after @bpmct helped me reproduce this issue (🙏🏻), it seems that we have an agent inactivity reporting issue. As Ben observed, rebooting the server is an easy repro. This causes the agent to be considered disconnected only after like 20 minutes (Tailscale/WireGuard timeout I presume). Another way to reproduce this is to stop networking:
However, simply killing the agent (either gracefully or immediately) does not reproduce this issue:
Why the difference? Since networking is still up, in the last case, we can immediately detect that other end went away. In the reboot case, the networking is shut down before the agent because it's not running as a systemd service (manually stopping the network mimics this case). Typically an agent is considered disconnected after not talking to the server for 6 seconds, but for some reason, this is not happening here. We need to dig deeper into why this is and since we can't really reduce the Tailscale/WireGuard timeout, we need another path to discover that chatter has ceased. |
|
Nice! Screen.Recording.2023-03-13.at.6.34.12.AM.mov |
With #5292, the experience is quite nice when I type
sudo rebootor a pod gets rescheduled (e.g. on spot VMs). However, in the dashboard, the agent still thinks it's connected. Could this be instant and we display a message to the user saying the agent has disconnected? Eventually, this could work with #4680 or other messages in the terminal/IDEs to inform the user.The text was updated successfully, but these errors were encountered: