[observability] create an alert when file descriptors exhausted

[jenting]

Is your feature request related to a problem? Please describe

create an alert when file descriptors exhausted

Describe the behaviour you'd like

Having an alert when the number of file descriptors exhausted

Describe alternatives you've considered

None

Additional context

https://gitpod.slack.com/archives/C04245JPHKL/p1663083593170859

[jenting]

I tried different views of the metrics process_open_fds, to send the alerts as soon as possible.

However, view with the sum of process_open_fds, at 09/13 05:30 UTC, the value is still slow.

We could consider sending alerts if the sum of process_open_fd over 3,000,000, but it indicates that the alert will be sent out after 09/13 10:00 UTC. But I think it's too late to send the alert out.

Any suggestion to make the alerts send out as soon as possible? @kylos101

[kylos101]

@jenting can you share the query you found most promising?

[kylos101]

👋 hey @jenting , have you tried an alert like this? https://www.robustperception.io/alerting-on-approaching-open-file-limits/

[kylos101]

Hey @jenting , I recommend trying to write an alert that is node or workspace based, rather than cluster based.

[jenting]

For the node-based metric node_filefd_allocated, the grafana query.

If we write an alert that is node-based, and the alert criteria is the current file descriptors / total file descriptors, we can see that the current file descriptors is far far away to total file descriptors. The grafana query.

Therefore, we can't use the criteria node_filefd_allocated{cluster="<cluster-name>"}/node_filefd_maximum{cluster="<cluster-name>"} as the alert rule.

[kylos101]

@jenting I recommend handing this off to @utam0k , as he is on-call this week, to see if he can finish.

@utam0k perhaps you could look later this week?

[utam0k]

@jenting This incident was caused by ws-manager with PVC after all, right? So I don't think the alert really needs to be issued until just before the fd of the node is depleted, which is about 80% of the time. What do you think?

[jenting]

This incident was caused by ws-manager with PVC after all, right? So I don't think the alert really needs to be issued until just before the fd of the node is depleted, which is about 80% of the time. What do you think?

I agree with you. We could put the threshold 80% and I did check our overall fd usage, we are far away from 80%. (If I remember correctly, the fd usage is under 10%).

[utam0k]

The problem with the fd of the supervisor should have been a side issue and not the root cause.