Fix re-entrant GetOrHandshake issues
#1044
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nbrownus
changed the title
GetOrHandshake issues
wadey
previously approved these changes
Dec 18, 2023
lighthouse.go
Outdated
Comment on lines
473
to
475
| if lh.l.Level >= logrus.DebugLevel { | ||
| lh.l.WithField("vpnIp", ip).Debug("Lighthouse query buffer was full, dropping request") | ||
| } |
There was a problem hiding this comment.
I wonder if this should be higher than debug, since without debug logs on it would be hard to tell this is happening and that you need to increase the buffer.
There was a problem hiding this comment.
I think it might be better to just make this a blocking write to a buffered channel
wadey
previously approved these changes
Dec 18, 2023
brad-defined
requested changes
Dec 19, 2023
brad-defined
approved these changes
Dec 19, 2023
wadey
approved these changes
Dec 19, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@wadey got a deadlock with v1.8.0 and was able to pull the stack trace.
HandshakeManager.GetOrHandshakecan be re-entrant viaHandshakeManager.StartHandshakethrough a call tohm.lightHouse.QueryServer(). Aside from the double read lock on the main hostmap not being great, theConnectionManagergo routine had fired between the 1st and 2nd calls toHandshakeManager.GetOrHandshakeand was waiting on a write lock for the main hostmap while blocking any future read locks.This is fixed by adding a channel and handling the actual lighthouse queries in a go routine. Should also speed up the hot path when many handshakes are occurring.
There is also a case when a tunnel is being tested and is using a relay for a double read lock in
ConnectionManager.This is fixed by turning the test packet into a traffic decision result and handling outside of the read lock.
My primary concern is in handling the
QueryServerwrites on a nonblocking buffered channel. I think we will want to block when full but I am leaving as nonblocking for now to review.