Dynflow collects stack trace across action suspends

[adamruzicka]

If an action suspends itself and on resume raises an exception, the printed stacktrace does not contain only trace from the resume to the place where the exception was raised, but it also includes the part leading to the suspend. Additionally, the traces accumulate over any number of suspends and resumes. If an action suspends, resumes, suspends and then raises an exception, the trace goes from start to suspend, from resume to suspend and finally from resume to raise. While this might be useful when developing dynflow internals, its usefulness in production deployments is debatable at best.

This is especially harmful with long running polling actions, which can accumulate huge amount of stack traces while they're active. And even if the stacktrace isn't printed, it is still collected leading to memory bloat.

For reproducer see gist

The formula for number of lines being emitted is lines = 53 * $SUSPEND_COUNT + 76, formula for output size is bytes = 4415 * SUSPEND_COUNT + 6470.

Number of suspends before raise	Stacktrace length [lines]	Stacktrace size [bytes]
1	129	10885
5	341	28545
10	606	50620
50	2726	227220
100	5376	447970
500	26576	2213970
1000	53076	4421470

I'm not sure how much of an issue is this in sidekiq-based deployments where events actually cross process borders.

BZ https://bugzilla.redhat.com/show_bug.cgi?id=1877917