Tokio stack overflow

[augustuswm]

This occurs a few times per day in the webhooky. Looks to be correlated to M-F work (as expected). Needs investigation.

2022-05-05 13:50:35.960 CDT thread 'tokio-runtime-worker' has overflowed its stack
2022-05-05 13:50:35.960 CDT fatal runtime error: stack overflow
2022-05-05 13:50:35.961 CDT Uncaught signal: 6, pid=1, tid=3, fault_addr=0.

[zephraph]

I believe @jessfraz was looking into something related to this a little while back.

[jessfraz]

Yeah we've talked about the stack overflows

[jessfraz]

The nightmare bugs

[augustuswm]

Looks to be highly correlated with pushes to RFD branches handled by handle_rfd_push. These are the main RFD branches (named 0\d\d\d) as opposed to something named 0003-new-edits (which are ignored).

Looking over the after the last two weeks all stack overflows are preceded by a log entry of the form:
{"level":"INFO", "msg":"push event -> file rfd/0219/README.adoc was modified on branch 0219", "ts":"2022-04-25T10:09:18.045736121Z"}
about 4-5 seconds before tokio crashes. The inverse also looks to be true (RFD push is followed by overflow).

This GCP query can be used to isolate the log entries: (("`push` event -> file" AND jsonPayload.msg =~ "branch 0\d\d\d$") OR "has overflowed its stack")

I need to look closer to determine what exactly is causing the overflow, but it looks to originate between lines 555 and 577 in webhooky/src/handlers_github.rs (those lines call a lot of other code though)

[augustuswm]

It seems to be failing on let r = github.repos().get(owner, repo).await?; in get_rfd_contents_from_repo I don't think the line itself it an issue, but possible the app is creating a deep enough tokio stack that it is actually running out of memory?

[jessfraz]

okay so this is where I have gotten as well. Then I dug deep into the generated client and couldn't find anything. And putting printfs random places didn't give me anything either. Somewhere there must be a recursive function right, but I could not for the life of me figure it out, like it must be in a lib of a lib or something, anyways I didn't get much further other than it was happening right before the auth function in the octorust generated lib

[augustuswm]

Running tokio with 4MB thread stacks instead of the default 2MB looks to provide enough memory for the RFD push events to be handled. After measuring more locally it looks like the large async trees are getting too large. I never hit an overflow locally, but got very close. My best guess at this point is we weren't hitting recursive calls, we were just creating too deeply nested of an async struct. I'll likely leave the increased thread stack size at 4MB for now, and will retest when we look at breaking down some of the functions.

[jessfraz]

OMG that makes so much more sense!

[jessfraz]

I was going crazy thinking something was recursive!!

[augustuswm]

Closing this for now as the immediate problem is resolved. We will revisit memory usage in the future to bring down the needed stack size.