While ingestion is suspended, neither the status page is accessible, nor is a clean shutdown possible.

[grobie]

We just found one of our servers having problems to keep up with persistence up to the point that ingestion got suspended. The exact circumstances are still not clear and @beorn7 is investigating.

Unexpectedly (to me at least), the status page would not load during the whole time, while the query interface was still available. It seems the target manager got into an undefined state as well.

[beorn7]

And the server would not even shutdown properly.

It seems to me that some parts of the server were completely hosed, among them either signal handling or the parts that would shut down targets and query them for the status page... Very difficult to debug.

In general, I think we have to organize the throttling / suspension of sample ingestion more cleanly.

[beorn7]

I just saw the "no clean shutdown" issue again now after suspension of ingestion. Something breaks if that happens.

[beorn7]

Theory is now: As long as any target has suspended the ingestion, both shutdown and status page don't work.

We probably want one big red "suspension" switch that is flipped once we reach the limit of chunks to be persisted, and then flipped back once we are at 90% of that.

[beorn7]

This could go into (or needs to be coordinated with) #1064

[fabxc]

The status page issues probably come from the target manager being stuck while holding a lock, e.g. waiting for old scrapers to terminate, which in turn are stuck because their samples aren't ingested.

At which step does shutdown get stuck? Or does sending SIGTERM not even trigger anything?

[beorn7]

The theory is that SIGTERM triggers as usual, but the target manager can only shutdown once all targets are out of suspended ingestion (which might never happen). This theory is not completely proven, though.

But I think we should have a central switch "stop ingestion" anyway, which would make many things much cleaner. It can be gated by an atomic variable to not cause too much lock contention.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.