Raven::Error - the server responded with status 403 Error in headers is: Event dropped due to filter

[hugoerg56]

I'm getting this error on a heroku app, Rails (4.2.6), raven-ruby (2.3.0)

2017-01-10T12:21:02.904714+00:00 app[web.1]: D, [2017-01-10T12:21:02.904649 #12] DEBUG -- sentry: ** [Raven] Sending event 7111d09234284641af0713bb0e83072b to Sentry
2017-01-10T12:21:03.051893+00:00 app[web.1]: E, [2017-01-10T12:21:03.051807 #12] ERROR -- sentry: ** [Raven] Unable to record event with remote Sentry server (Raven::Error - the server responded with status 403 Error in headers is: Event dropped due to filter)
2017-01-10T12:21:03.051958+00:00 app[web.1]: E, [2017-01-10T12:21:03.051922 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.2.0/lib/raven/transports/http.rb:33:in `rescue in send_event'
2017-01-10T12:21:03.051984+00:00 app[web.1]: E, [2017-01-10T12:21:03.051954 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.2.0/lib/raven/transports/http.rb:18:in `send_event'
2017-01-10T12:21:03.052014+00:00 app[web.1]: E, [2017-01-10T12:21:03.051982 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.2.0/lib/raven/client.rb:40:in `send_event'
2017-01-10T12:21:03.052037+00:00 app[web.1]: E, [2017-01-10T12:21:03.052008 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.2.0/lib/raven/instance.rb:81:in `send_event'
2017-01-10T12:21:03.052066+00:00 app[web.1]: E, [2017-01-10T12:21:03.052034 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.2.0/lib/raven/instance.rb:126:in `capture_type'
2017-01-10T12:21:03.052093+00:00 app[web.1]: E, [2017-01-10T12:21:03.052063 #12] ERROR -- sentry: ** [Raven] /app/vendor/ruby-2.3.1/lib/ruby/2.3.0/forwardable.rb:189:in `capture_type'
2017-01-10T12:21:03.052117+00:00 app[web.1]: E, [2017-01-10T12:21:03.052089 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.2.0/lib/raven/integrations/rack.rb:28:in `capture_type'
2017-01-10T12:21:03.052142+00:00 app[web.1]: E, [2017-01-10T12:21:03.052114 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/sentry-raven-2.2.0/lib/raven/integrations/rails/overrides/debug_exceptions_catcher.rb:8:in `render_exception'
2017-01-10T12:21:03.052172+00:00 app[web.1]: E, [2017-01-10T12:21:03.052140 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:27:in `rescue in call'
2017-01-10T12:21:03.052197+00:00 app[web.1]: E, [2017-01-10T12:21:03.052166 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
2017-01-10T12:21:03.052220+00:00 app[web.1]: E, [2017-01-10T12:21:03.052191 #12] ERROR -- sentry: ** [Raven] /app/vendor/bundle/ruby/2.3.0/gems/actionpack-4.2.6/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'

[nateberkopec]

403 Error in headers is: Event dropped due to filter

The Sentry server is dropping the event you're sending. If this is hosted Sentry, you'll probably have better luck asking support.

[marisawallace]

@nateberkopec Sentry has filters set up on the server side to filter out errors that it believes aren't relevant.

This begs the question: if Sentry is filtering these errors (i.e. not surfacing them on the dashboard), then why wouldn't we want to filter them out on the client-side as well? If it's an irrelevant error, we should at least have the option to exclude it from the logs.

There's a discussion of the same issue over in the Sentry-Elixir client repo: getsentry/sentry-elixir#89

---

I don't like jumping to solutions too quickly, but would you be open to a PR adding the option to set a log_transport_failure_callback to the config so end-users of this gem can provide their own (e.g. no-op) implementation of the failed_send logging?:

    def failed_send(e, event)
      @state.failure
      if e # exception was raised
        configuration.logger.error "Unable to record event with remote Sentry server (#{e.class} - #{e.message}):\n#{e.backtrace[0..10].join("\n")}"
      else

something like

    def failed_send(e, event)
      @state.failure
      if e # exception was raised
        if configuration.log_transport_failure_callback
          configuration.log_transport_failure_callback.call(e)
        else
          configuration.logger.error "Unable to record event with remote Sentry server (#{e.class} - #{e.message}):\n#{e.backtrace[0..10].join("\n")}"
          configuration.logger.error("Failed to submit event: #{get_log_message(event)}")
        end
      else
        configuration.logger.error "Not sending event due to previous failure(s)."
        configuration.logger.error("Failed to submit event: #{get_log_message(event)}")
      end
      configuration.transport_failure_callback.call(event) if configuration.transport_failure_callback
    end

^ just illustration code, happy to clean up the bloated conditional logic.

[nateberkopec]

@maxkwallace I think we can just add the exception object to the parameters provided to transport_failure_callback. It will break anyone's code using a lambda, so we could add it to the upcoming 3.0 release.

This begs the question: if Sentry is filtering these errors (i.e. not surfacing them on the dashboard), then why wouldn't we want to filter them out on the client-side as well? If it's an irrelevant error, we should at least have the option to exclude it from the logs.

I generally try to keep us on the same page as serverside Sentry filters, but we'll never be 100% the same.

[marisawallace]

@nateberkopec were you thinking of changing the way transport_failure_callback is currently used?

For my use case, I really just want a way to make the gem not log the events that get a 403 from the server, i.e. I want a way to stop

          configuration.logger.error "Unable to record event with remote Sentry server (#{e.class} - #{e.message}):\n#{e.backtrace[0..10].join("\n")}"

from running inside the gem. I don't think that's possible just from adding the exception object to the params passed to transport_failure_callback.

[nateberkopec]

I don't see the point of making that configurable. It's an error, it shouldn't happen regularly. We should fix whatever workload you're generating that causes so many failures.

[marisawallace]

@nateberkopec it's not a workload. The failures are intermittent and unexpected, stemming from normal usage-- nothing we'd consider erroneous or that we have control over.

Take a look at getsentry/sentry-elixir#89 -- in that case, the "errors" are coming from requests generated by web crawler.

[nateberkopec]

OK, looks like we can do the same thing they did too. I think we can probably log all 4XX at the warning level and 5XX statuses at the error level, does that make sense to you?

[marisawallace]

In general, that sounds reasonable.

For me personally (i.e. discount what I'm about to say), AFAIK that doesn't help at all. My app logs to Papertrail at the info level, and I don't know of a way to filter the logs to only show things at error or higher. So I can't commit to sending a PR, and I'd probably end up forking raven-ruby for my project anyway.

Given the discussion, I'm happy with either outcome, so please make the decision that is best for the community :)

[nateberkopec]

@maxkwallace After the proposed change, you would simply:

logger = Logger.new(STDOUT)
logger.level = Logger::ERROR

Raven.configure do |config|
  config.logger = logger
end

EDIT: I haven't used Papertrail in years but I think it's just slurping everything from STDOUT right? If not, change the first line to whatever source it needs to be.

[marisawallace]

Great, I'll give that a shot--thanks! I appreciate you pushing back and being conscientious about avoiding adding features to the gem.