Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

node: don't allow obs reqs if the queue is full #1429

Merged
merged 1 commit into from
Aug 15, 2022

Conversation

bruce-riley
Copy link
Contributor

@bruce-riley bruce-riley commented Aug 13, 2022

Change-Id: Ifb0d038fa3adeddc6226e2289fe9dbfc8e39b4e7

Jeff noticed a while ago that too many reobservation requests can hang the guardian. This is because an attempt to write to the channel hangs if the channel has reached capacity. This PR addresses by making sure the channel is not full before writing to it.


This change is Reviewable

Change-Id: Ifb0d038fa3adeddc6226e2289fe9dbfc8e39b4e7
@netlify
Copy link

netlify bot commented Aug 13, 2022

Deploy Preview for tubular-dango-1656b2 canceled.

Name Link
🔨 Latest commit a309163
🔍 Latest deploy log https://app.netlify.com/sites/tubular-dango-1656b2/deploys/62f7f703e377ab000867a19a

@bruce-riley bruce-riley changed the title Don't allow obs reqs if the queue is full node: don't allow obs reqs if the queue is full Aug 13, 2022
Copy link
Collaborator

@SEJeff SEJeff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it! Do we want to consider bumping this channel size up a bit? Say maybe 100? During the worst events I've ever seen, we once had about 80 or so when the L1 was going bananas and nodes were dropping left and right.

Trying to rapid fire reobservations via some unspeakable shell one liners is how I found the "feature". This looks great!

@bruce-riley
Copy link
Contributor Author

We discussed increasing the queue size and decided to leave it as is for now.

@bruce-riley bruce-riley merged commit 99fb46d into dev.v2 Aug 15, 2022
@bruce-riley bruce-riley deleted the too_many_obs_reqs_hangs_guardian branch August 15, 2022 12:33
bruce-riley added a commit that referenced this pull request Aug 15, 2022
Change-Id: Ifb0d038fa3adeddc6226e2289fe9dbfc8e39b4e7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants