Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Envoy Crash when connection state assert failure #9083
Title: Envoy Crash when connection state assert failure
Admin and Stats Output:
Well the good news is that this is a debug ASSERT, and there's actual error handling so with an opt build it shouldn't crash.
I think the problem is is here:
quick advice, switching to a debug build should fix the problem due to the error handling (and if I'm incorrect, PLEASE take it up with the envoy-security group, don't disclose crashes in public! :-) )
I've got a TODO to clean this area up, since it was obsoleted by #8352 . If switching to an opt build isn't an option I can do the quick fix of moving the read disable inside the connection checks. If removing that check or doing an opt build is sufficient to hold you over, I can probably get to the clean up some time in early December.
I can't see how the access log would create this problem (maybe it's making a race condition worse?) but having spent some time looking at the code I'm frankly baffled how you're getting here.
the underlying connection looks like it is closed in one of three places. onIdleTimeout and onConnectionDurationTimeout are never in the dispatch stack. the third is checkForDeferredClose() which I would think would be the culprit
checkForDeferredClose only calls close() if streams_.empty()
each dispatch() call for HTTP/1 we might create one stream, and then might checkForDeferredClose, then might read disable, but I can't see how we'd go from no streams where we checkForDeferredClose to having streams at read disable time because the parser doesn't act then.
I've played around with the network class to see if I can force a dispatch event in delay close and can't get that to fail either. I can do the quick fix but I'd really love to understand what's going on here - if you guys ever get another reproducible situation and can turn up a bit of logging ping me on slack and I'd be happy to try to debug with you