karma hangs after X time with to many files again?

[jonaz]

Is might be related to #2888 but this time it took a while longer to stopp working. Still have 24 alertmanagers and the interval is now 30s with timeout 10s on each one.

# /usr/local/bin/karma --version
v0.81

Let me know if you need more info.
lsof is growing socks as last time:

# lsof | grep karma | wc -l
53415

and its growing. Will probably start responding with errors when it hits 65k.
Its the same symptoms as #2888

[prymitive]

I was able to reproduce a hang when working on #2888 but I with that fixed I can't anymore.
Do you scrape karma metrics and have a chart of process_open_fds and http_request_count_total over that 24h?

[jonaz]

Karma stops responding on /metrics when this happens. When silently leaks sock according to lsof until it reaches limit and spams to many files error in logs. From stop responding to spamming errors took a few days. We have 3 instances so we can afford to let a non working one be running to troubleshoot.

This is probably a more rare deadlock since it i
Only happened once since we updated to the fix in #2888 I'll update here when we run into it again.

[prymitive]

I don’t see any leakage so there’s a chance client connections are not being closed.
I’ve added HTTP server read and write timeouts, as I realised there are none at the moment. Try the latest master image to see if it helps.

[jonaz]

Is there a possible deadlock in https://github.com/prymitive/karma/blob/main/internal/alertmanager/models.go#L498 ?
Should not defer but unlock sooner?

[prymitive]

That seems likely, good catch

[prymitive]

Merged a fix for that, let me know if you still see any issues, thanks!