Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asynchronous notifier shouldn't block if queue full #47

Merged
merged 1 commit into from
Feb 23, 2016
Merged

Asynchronous notifier shouldn't block if queue full #47

merged 1 commit into from
Feb 23, 2016

Conversation

crunis
Copy link
Contributor

@crunis crunis commented Feb 22, 2016

SizedQueue blocks if the queue is full. I think this behaviour is completely unexpected in an asynchronous component.

I think it would be better to drop the error in this situation. I can picture 2 scenarios where this could happen and dropping the extra errors would make sense:

  1. The app is sending errors faster than Airbrake server can process them. It may be due to a bug in the app or because the app is being flooded by some sort of bogus request, we don't want the app to block for this reason, we would like, if possible, to keep providing service even if it's with degraded performance. In this situation, still many errors will make it to the Airbrake server so we won't miss that something is wrong.
  2. The Airbrake server has a problem and is not accepting errors. In this situation we also don't want our app to block and stop providing service. If the Airbrake server is not working we will notice at some point, the async workers will notify the timeout, and we can fix the problem without interrupting our application's service.

I have fixed the specs accordingly

@@ -54,7 +55,8 @@

it "prints the debug message the correct number of times" do
log = @stderr.string.split("\n")
expect(log.grep(/\*\*Airbrake: \{\}/).size).to eq(300)
expect(log.grep(/\*\*Airbrake: \{\}/).size).to \
be >= @sender.instance_variable_get(:@unsent).max
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: could you please break it into two lines like this instead:

expect(log.grep(/\*\*Airbrake: \{\}/).size).
  to be >= @sender.instance_variable_get(:@unsent).max

It is more consistent with the current style.

@kyrylo
Copy link
Contributor

kyrylo commented Feb 22, 2016

Thanks for the PR, it makes sense. Fixes #48.

  • do you think we should log something if we drop errors?
  • could you please squash?

@stevecrozz
Copy link

If it helps, here's a reference to an implementation I created a while back:
https://github.com/stevecrozz/hydraulic_brake/blob/master/lib/hydraulic_brake/async_sender.rb#L31

@unsent << notice
if @unsent.size >= @unsent.max
msg = "#{LOG_LABEL} dropping notice since sending queue is full"
@config.logger.error msg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was more of a question. I am not sure if we need to log, but maybe you would find it useful?
If we do want, I am not sure about the error severity. It's more like a warn to me, but I am open to other suggestions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea to log, so the information it's not lost and can be retrieved if necessary.

Having the queue full of errors and starting to not send them to the error log platform it's, for me, a critical situation. If, for some reason, I'm not aware of this situation and I'm looking at the logs I would like to see it quickly since it's an ERROR. What do you think ?

I'm almost done with the changes, I'm having some trouble with the specs but I hope to fix it soon.

@crunis
Copy link
Contributor Author

crunis commented Feb 22, 2016

Done! I've followed @stevecrozz style.

As I was saying in the previous comment, I think is interesting to log the notice (so the information is not lost) and I would consider it with Error severity. Like I've said: "Having the queue full of errors and starting to not send them to the error log platform it's, for me, a critical situation. If, for some reason, I'm not aware of this situation and I'm looking at the logs I would like to see it quickly since it's an ERROR".

What do you think ?


def will_not_deliver(notice)
backtrace = notice[:errors][0][:backtrace].map do |line|
"#{line[:file]}:#{line[:line]} in `#{line[:function]}`"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be ``#{line[:function]}'` (no backtick in the end) if you want to make it resemble a Ruby stack frame.

@kyrylo
Copy link
Contributor

kyrylo commented Feb 23, 2016

Makes sense to me, thanks for the explanation. I am happy to merge this with a couple minor fixes that I mentioned above.

@crunis
Copy link
Contributor Author

crunis commented Feb 23, 2016

Applied all your fixes and commented about the condition!

@kyrylo
Copy link
Contributor

kyrylo commented Feb 23, 2016

Looks great, thanks 👍

kyrylo added a commit that referenced this pull request Feb 23, 2016
Asynchronous notifier shouldn't block if queue full
@kyrylo kyrylo merged commit f33825b into airbrake:master Feb 23, 2016
kyrylo added a commit that referenced this pull request Feb 23, 2016
@crunis
Copy link
Contributor Author

crunis commented Feb 23, 2016

Thanks! When do you plan to release the next version of airbrake-ruby @kyrylo ?

@kyrylo
Copy link
Contributor

kyrylo commented Feb 23, 2016

I plan to release it sometime during this week. Waiting on #46 to be resolved and then we're ready to release :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants