Breadcrumbs feature release (#525)

* Breadcrumbs/Circular buffer: Added circular buffer class

* Breadcrumbs/Circular buffer: Added initial buffer unit tests

* Breadcrumbs/Circular buffer: Ensure test file is named correctly

* Breadcrumbs/Circular buffer: Code tidy up

* Breadcrumbs / Breadcrumb class: Add first pass breadcrumb class with unit tests

* Breadcrumbs/Circular buffer rubocop fixes

* Breadcrumbs / Breadcrumb class: Rubocop fixes

* Breadcrumbs/Cricular buffer: Improve test readability

* Breadcrumbs/Breadcrumb: Simplified timestamp, improved tests

* Breadcrumbs/Breadcrumb: Rubocop requirements

* Breadcrumbs/Breadcrumb: Added doc comments

* Breadcrumbs/Circular buffer: Fixed RSpec/Rspec typo

* Breadcrumbs/Breadcrumb: Using explicit RSpec.describe call

* Breadcrumbs/Validator: Added validator and tests

* Breadcrumbs/Validator: Rubocop fixes

* Breadcrumbs/Validator: Tidying up from initial review

* Breadcrumbs/Validator: Re-add renamed types file

* Breadcrumbs/Breadcrumb: Change message => name, add yard comments

* Breadcrumbs/Circular buffer: Added yard docs

* Breadcrumbs/Breadcrumb: Improve tests, convert to utc sooner

* Breadcrumbs/Configuration: Initial configuration changes

* Breadcrumbs/Report: Initial report changes to support breadcrumbs

* Breadcrumbs/Breadcrumb: Test timestamps match up

* Breadcrumbs/Validator: Change 'message' to 'name', use name and meta_data clones

* Breadcrumbs/Validator: Refactor toward instance_double & let

* Breadcrumbs/Configuration: Add in missing import statement

* Breadcrumbs/Circular buffer docs: Improve readability

* Breadcrumbs/Configuration: Add configuration tests

* Breadcrumbs/Configuration: Added yardoc comments

* Breadcrumbs/Configuration: Simplify before_breadcrumb_callbacks, allow automatic_breadcrumb_types editting

* Breadcrumbs/Configuration: Add automatic breadcrumbs types editablilty test

* Breadcrumbs/Configuration: Add types to Yardoc arrays

* Breadcrumbs/Validator: Use an instance_double to reflect Configuration better

* Breadcrumbs/Validator: Add Yardoc comments

* Breadcrumbs/Validator: Use select over each when filtering

* Breadcrumbs/Validator: Clean up tests

* Breadcrumbs/Validator: Use double for configuration

* Breadcrumbs/Validator: Improved Yardoc

* Breadcrumbs/Leave breadcrumb: Add implementation and tests

* Breadcrumbs/Middleware: Add breadcrumb middleware

* Breadcrumbs/Report: Rubify get_summary method name

* Breadcrumbs/Report: Ensure message is set correctly

* Breadcrumbs/Report: Change default breadcrumbs from Hash to Array

* Breadcrumbs/Report: Ensure breadcrumb meta_data is filtered correctly

* Breadcrumbs/Report: Fixed typo

* Bugsnag/Report: Added tests

* Breadcrumbs/Leave breadcrumb: Update rubocop to allow increased Bugsnag module length

* Breadcrumbs/middleware: Rubocop fixes

* Breadcrumbs/Leave-breadcrumb: Rubocop improvements/early return

* Breadcrumbs/Leave-breadcrumb: Better block styling

* Breadcrumbs/Report: Ensure report summary is consistent with dashboard

* Breadcrumbs/Report: Better Rubocop report exclusions

* Breadcrumbs/Report: Rspec improvements

* Breadcrumbs/Notify-breadcrumb: Initial implementation and tests

* Breadcrumbs/Notify breadcrumb: Add test against notifying a string

* Breadcrumbs/Report: Better checking against Java error

* Breadcrumbs/Report: Make summary more accurate

* Breadcrumbs/Report update: make summary guard more readable

* Breadcrumbs/Update automatic_breadcrumb_types to enabled_automatic_breadcrumb_types

* Breadcrumbs: Rails Breadcrumbs (#514)

* Breadcrumbs/Rails integration: Initial integration implemenetation

* Breadcrumbs/Rails: Minor fixes

* Breadcrumbs/Rails: Add initial maze tests

* Breadcrumbs/Rails breadcrumbs: Remove plural event data

* Breadcrumbs/Rails breadcrumbs: Ammend breadcrumb namespace

* Breadcrumbs/Rails breadcrumbs: Remove event_id from allowed keys, change name to event_name

* Breadcrumbs/Rails breadcrumbs: Require breadcrumb file when necessary to avoid namespace clashes

* Breadcrumbs/Rails breadcrumbs: Fix rubocop issues

* Breadcrumbs/Rails breadcrumbs: Clarify namespaces better

* Breadcrumbs/Rails breadcrumbs: Remove invalid data from captured breadcrumb definitions

* Breadcrumbs/Rails breadcrumbs: Added ActiveJob breadcrumb test

* Breadcrumbs/Rails breadcrumbs: Ammended active_job event id

* Breadcrumbs/Ruby breadcrumbs: Removed complex breadcrumb metadata

* Breadcrumbs/Ruby: Added cache breadcrumb

* Breadcrumbs/Ruby breadcrumbs: Fleshed out maze-test expectations where possible

* Breadcrumbs/Rails breadcrumbs: Fixed broken test fixture, made fixtures more rails-y

* Breadcrumbs: Add Mongo integration (#516)

* Breadcrumbs/Rails integration: Initial integration implemenetation

* Breadcrumbs/Rails: Minor fixes

* Breadcrumbs/Rails: Add initial maze tests

* Breadcrumbs/Mongo: Initial subscriber implementation

* Breadcrumbs/Rails breadcrumbs: Remove plural event data

* Breadcrumbs/Rails breadcrumbs: Ammend breadcrumb namespace

* Breadcrumbs/Rails breadcrumbs: Remove event_id from allowed keys, change name to event_name

* Breadcrumbs/Rails breadcrumbs: Require breadcrumb file when necessary to avoid namespace clashes

* Breadcrumbs/Mongo integration: Fix module name issue

* Breadcrumbs/Mongo integration: Maintain consistency with ActiveSupport arguments

* Breadcrumbs/Rails breadcrumbs: Fix rubocop issues

* Breadcrumbs/Rails breadcrumbs: Clarify namespaces better

* Breadcrumbs/Mongo integration: Add integration tests

* Breadcrumbs/Mongo integration: Remove example commits

* Breadcrumbs/Rails breadcrumbs: Remove invalid data from captured breadcrumb definitions

* Breadcrumbs/Mongo integration: Rubocop fixes

* Breadcrumbs/Mongo integration: Better doc comments

* Breadcrumbs/Mongo integration: Improve tests, add 'failure' test

* Breadcrumbs/Mongo integration: Re-add 'success' test

* Breadcrumbs/Rails breadcrumbs: Added ActiveJob breadcrumb test

* Breadcrumbs/Rails breadcrumbs: Ammended active_job event id

* Breadcrumbs/Ruby breadcrumbs: Removed complex breadcrumb metadata

* Breadcrumbs/Ruby: Added cache breadcrumb

* Breadcrumbs/Ruby breadcrumbs: Fleshed out maze-test expectations where possible

* Breadcrumbs/Mongo integration: Rubocop fixes

* Breadcrumbs/Rails breadcrumbs: Fixed broken test fixture, made fixtures more rails-y

* Breadcrumbs/Mongo integration: Maze fixes

* Breadcrumbs/Validator: Allow nil as an acceptable meta_data type (#520)

* Breadcrumbs/Mongo: Remove 'started' breadcrumb, add collection detail (#519)

* Breadcrumbs/Mongo integration: Remove 'started' event, add 'collection' data

* Breadcrumbs/Mongo: Add filter keys

* Breadcrumbs/Mongo integration: Made filters consistent with other breadcrumb implementations

* Breadcrumbs/Mongo integration: Rubocop fixes

* Breadcrumbs/Mongo integration: Address feedback

* Breadcrumbs: Additional Rails ActiveRecord data (#521)

* Breadcrumbs/Rails breadcrumbs: Capture event_id

* Breadcrumbs/Rails breadcrumbs: Capture 'sql' string and redacted bindings

* Breadcrumbs/Rails integration: Redact SQL string values

* Breadcrumbs/Rails integration: Remove SQL string acquisition

* Breadcrumbs/Rails improvements: Remove unnecessary require

* Breadcrumbs/Rails integration: Add integration tests for bindings

* Breadcrumbs/Ruby integration: lock nokogiri version

* Breadcrumbs/Ruby integration: Better meta_data naming, Ruby < 2.1 compatability

* Breadcrumbs/Mongo integration: Handle 'any_of' cases (#522)

* Breadcrumbs/Mongo integration: Handle 'any_of' cases

* Breadcrumbs/Mongo Integration: Improve filter sanitization process and add unit tests

* Breadcrumbs/Mongo integration: Ensure tests are notated correctly

* Breadcrumbs/Mongo filters: Add 1.9.3 & 2.0.0 compatability

* Breadcrumbs/Mongo integration: Appease rubocop

* Breadcrumbs/Mongo integration: Minor test and efficieny improvements

* Breadcrumbs/Mongo integration: Test improvements

* Breadcrumbs/Mongo integrations: Test spelling

* Breadcrumbs: Fix rubocop issues (excluding configuration from ClassLength until refactor)

.rubocop_todo.yml

@@ -289,6 +289,9 @@ Metrics/BlockNesting:
 # Configuration parameters: CountComments.
 Metrics/ClassLength:
   Max: 149
+  Exclude:
+    - 'lib/bugsnag/report.rb'
+    - 'lib/bugsnag/configuration.rb'
 
 # Offense count: 12
 Metrics/CyclomaticComplexity:

features/fixtures/docker-compose.yml

@@ -4,6 +4,9 @@ services:
   redis:
     image: redis
 
+  mongo:
+    image: mongo
+
   plain-ruby:
     build:
       context: plain
@@ -117,6 +120,7 @@ services:
       - BUGSNAG_SESSION_ENDPOINT
       - BUGSNAG_TIMEOUT
       - CALLBACK_INITIATOR
+      - SQL_ONLY_BREADCRUMBS
       - USE_DEFAULT_AUTO_CAPTURE_SESSIONS
     ports:
       - target: 3000
@@ -129,6 +133,8 @@ services:
       args:
         - RUBY_VERSION
         - APP_PATH
+    depends_on:
+      - mongo
     environment:
       - BUGSNAG_API_KEY
       - http_proxy
@@ -153,6 +159,7 @@ services:
       - BUGSNAG_SESSION_ENDPOINT
       - BUGSNAG_TIMEOUT
       - CALLBACK_INITIATOR
+      - SQL_ONLY_BREADCRUMBS
       - USE_DEFAULT_AUTO_CAPTURE_SESSIONS
     ports:
       - target: 3000
@@ -165,6 +172,8 @@ services:
       args:
         - RUBY_VERSION
         - APP_PATH
+    depends_on:
+      - mongo
     environment:
       - BUGSNAG_API_KEY
       - http_proxy
@@ -189,6 +198,7 @@ services:
       - BUGSNAG_SESSION_ENDPOINT
       - BUGSNAG_TIMEOUT
       - CALLBACK_INITIATOR
+      - SQL_ONLY_BREADCRUMBS
       - USE_DEFAULT_AUTO_CAPTURE_SESSIONS
     ports:
       - target: 3000

features/fixtures/rails3/app/app/controllers/breadcrumbs_controller.rb

@@ -0,0 +1,19 @@
+class BreadcrumbsController < ApplicationController
+  def handled
+    Bugsnag.notify("Request breadcrumb")
+    render json: {}
+  end
+
+  def sql_breadcrumb
+    User.where(:email => "foo").as_json
+    Bugsnag.notify("SQL breadcrumb")
+    render json: {}
+  end
+
+  def cache_read
+    Rails.cache.write('test', true)
+    Rails.cache.read('test')
+    Bugsnag.notify("Cache breadcrumb")
+    render json: {}
+  end
+end

features/fixtures/rails3/app/config/initializers/bugsnag.rb

@@ -11,4 +11,10 @@
   config.release_stage = ENV["BUGSNAG_RELEASE_STAGE"] if ENV.include? "BUGSNAG_RELEASE_STAGE"
   config.send_code = ENV["BUGSNAG_SEND_CODE"] != "false"
   config.send_environment = ENV["BUGSNAG_SEND_ENVIRONMENT"] == "true"
-end
+
+  if ENV["SQL_ONLY_BREADCRUMBS"] == "true"
+    config.before_breadcrumb_callbacks << Proc.new do |breadcrumb|
+      breadcrumb.ignore! unless breadcrumb.meta_data[:event_name] == "sql.active_record" && breadcrumb.meta_data[:name] == "User Load"
+    end
+  end
+end

features/fixtures/rails3/app/config/routes.rb

@@ -14,5 +14,6 @@
   get "/send_code/(:action)", controller: 'send_code'
   get "/send_environment/(:action)", controller: 'send_environment'
   get "/warden/(:action)", controller: 'warden'
+  get "/breadcrumbs/(:action)", controller: 'breadcrumbs'
   get "/(:action)", controller: 'application'
 end

features/fixtures/rails4/app/Gemfile

@@ -39,4 +39,8 @@ end
 # Added at 2018-03-26 15:02:53 +0100 by amoinet:
 gem 'bugsnag', path: '/bugsnag'
 
-gem 'devise'
+gem 'devise'
+
+gem "mongoid", '~> 5.4.0'
+
+gem "nokogiri", "1.6.8"

features/fixtures/rails4/app/app/controllers/breadcrumbs_controller.rb

@@ -0,0 +1,26 @@
+class BreadcrumbsController < ApplicationController
+  def handled
+    Bugsnag.notify("Request breadcrumb")
+    render json: {}
+  end
+
+  def sql_breadcrumb
+    User.find_by(email: "foo")
+    Bugsnag.notify("SQL breadcrumb")
+    render json: {}
+  end
+
+  def active_job
+    Thread.new { NotifyJob.perform_later }.join
+    render json: {}
+  end
+
+  def cache_read
+    Thread.new {
+      Rails.cache.write('test', true)
+      Rails.cache.read('test')
+      Bugsnag.notify("Cache breadcrumb")
+    }.join
+    render json: {}
+  end
+end

features/fixtures/rails4/app/app/controllers/mongo_controller.rb

@@ -0,0 +1,23 @@
+class MongoController < ApplicationController
+
+  def success_crash
+    doc = MongoModel.create(string_field: "String")
+    doc.save
+    "Statement".prepnd("Failing")
+  end
+
+  def get_crash
+    MongoModel.where(string_field: true).as_json
+    MongoModel.any_of({string_field: true}, {numeric_field: 123}).as_json
+    "Statement".prepnd("Failing")
+  end
+
+  def failure_crash
+    begin
+      Mongoid::Clients.default.database.command(:bogus => 1)
+    rescue
+    end
+
+    "Statement".prepnd("Failing")
+  end
+end

features/fixtures/rails4/app/app/jobs/application_job.rb

@@ -0,0 +1,2 @@
+class ApplicationJob < ActiveJob::Base
+end

features/fixtures/rails4/app/app/jobs/notify_job.rb

@@ -0,0 +1,5 @@
+class NotifyJob < ApplicationJob
+  def perform
+    Bugsnag.notify("Failed")
+  end
+end

features/fixtures/rails4/app/app/models/mongo_model.rb

@@ -0,0 +1,6 @@
+class MongoModel
+  include Mongoid::Document
+
+  field :string_field, type: String
+  field :numeric_field, type: Integer
+end

features/fixtures/rails4/app/config/initializers/bugsnag.rb

@@ -11,4 +11,10 @@
   config.release_stage = ENV["BUGSNAG_RELEASE_STAGE"] if ENV.include? "BUGSNAG_RELEASE_STAGE"
   config.send_code = ENV["BUGSNAG_SEND_CODE"] != "false"
   config.send_environment = ENV["BUGSNAG_SEND_ENVIRONMENT"] == "true"
+
+  if ENV["SQL_ONLY_BREADCRUMBS"] == "true"
+    config.before_breadcrumb_callbacks << Proc.new do |breadcrumb|
+      breadcrumb.ignore! unless breadcrumb.meta_data[:event_name] == "sql.active_record" && breadcrumb.meta_data[:name] == "User Load"
+    end
+  end
 end

features/fixtures/rails4/app/config/mongoid.yml

@@ -0,0 +1,22 @@
+development:
+  # Configure available database clients. (required)
+  clients:
+    # Defines the default client. (required)
+    default:
+      # Defines the name of the default database that Mongoid can connect to.
+      # (required).
+      database: rails4_development
+      # Provides the hosts the default client can connect to. Must be an array
+      # of host:port pairs. (required)
+      hosts:
+        - mongo:27017
+test:
+  clients:
+    default:
+      database: rails4_test
+      hosts:
+        - mongo:27017
+      options:
+        read:
+          mode: :primary
+        max_pool_size: 1

features/fixtures/rails4/app/config/routes.rb

@@ -16,4 +16,6 @@
   get "/send_code/(:action)", controller: 'send_code'
   get "/send_environment/(:action)", controller: 'send_environment'
   get "/devise/(:action)", controller: 'devise'
+  get "/breadcrumbs/(:action)", controller: 'breadcrumbs'
+  get "/mongo/(:action)", controller: 'mongo'
 end

features/fixtures/rails5/app/Gemfile

@@ -45,3 +45,7 @@ gem 'bugsnag', path: '/bugsnag'
 
 # Added at 2018-04-26 10:20:10 +0100 by amoinet:
 gem "clearance", "~> 1.16"
+
+gem "mongoid"
+
+gem "nokogiri", "1.6.8"

features/fixtures/rails5/app/app/controllers/breadcrumbs_controller.rb

@@ -0,0 +1,24 @@
+class BreadcrumbsController < ApplicationController
+  def handled
+    Bugsnag.notify("Request breadcrumb")
+    render json: {}
+  end
+
+  def sql_breadcrumb
+    User.find_by(email: "foo")
+    Bugsnag.notify("SQL breadcrumb")
+    render json: {}
+  end
+
+  def active_job
+    NotifyJob.perform_later
+    render json: {}
+  end
+
+  def cache_read
+    Rails.cache.write('test', true)
+    Rails.cache.read('test')
+    Bugsnag.notify("Cache breadcrumb")
+    render json: {}
+  end
+end

features/fixtures/rails5/app/app/controllers/mongo_controller.rb

@@ -0,0 +1,22 @@
+class MongoController < ApplicationController
+  def success_crash
+    doc = MongoModel.create(string_field: "String")
+    doc.save
+    "Statement".prepnd("Failing")
+  end
+
+  def get_crash
+    MongoModel.where(string_field: true).as_json
+    MongoModel.any_of({string_field: true}, {numeric_field: 123}).as_json
+    "Statement".prepnd("Failing")
+  end
+
+  def failure_crash
+    begin
+      Mongoid::Clients.default.database.command(:bogus => 1)
+    rescue
+    end
+
+    "Statement".prepnd("Failing")
+  end
+end

features/fixtures/rails5/app/app/jobs/notify_job.rb

@@ -0,0 +1,5 @@
+class NotifyJob < ApplicationJob
+  def perform
+    Bugsnag.notify("Failed")
+  end
+end

features/fixtures/rails5/app/app/models/mongo_model.rb

@@ -0,0 +1,6 @@
+class MongoModel
+  include Mongoid::Document
+
+  field :string_field, type: String
+  field :numeric_field, type: Integer
+end

features/fixtures/rails5/app/config/initializers/bugsnag.rb

@@ -11,4 +11,10 @@
   config.release_stage = ENV["BUGSNAG_RELEASE_STAGE"] if ENV.include? "BUGSNAG_RELEASE_STAGE"
   config.send_code = ENV["BUGSNAG_SEND_CODE"] != "false"
   config.send_environment = ENV["BUGSNAG_SEND_ENVIRONMENT"] == "true"
+
+  if ENV["SQL_ONLY_BREADCRUMBS"] == "true"
+    config.before_breadcrumb_callbacks << Proc.new do |breadcrumb|
+      breadcrumb.ignore! unless breadcrumb.meta_data[:event_name] == "sql.active_record" && breadcrumb.meta_data[:name] == "User Load"
+    end
+  end
 end

features/fixtures/rails5/app/config/mongoid.yml

@@ -0,0 +1,23 @@
+development:
+  # Configure available database clients. (required)
+  clients:
+    # Defines the default client. (required)
+    default:
+      # Defines the name of the default database that Mongoid can connect to.
+      # (required).
+      database: rails5_development
+      # Provides the hosts the default client can connect to. Must be an array
+      # of host:port pairs. (required)
+      hosts:
+        - mongo:27017
+
+test:
+  clients:
+    default:
+      database: rails5_test
+      hosts:
+        - mongo:27017
+      options:
+        read:
+          mode: :primary
+        max_pool_size: 1

features/fixtures/rails5/app/config/routes.rb

@@ -50,4 +50,13 @@
   get 'clearance/create', to: 'clearance#create'
   get 'clearance/unhandled', to: 'clearance#unhandled'
   get 'clearance/handled', to: 'clearance#handled'
+
+  get 'breadcrumbs/handled', to: 'breadcrumbs#handled'
+  get 'breadcrumbs/sql_breadcrumb', to: 'breadcrumbs#sql_breadcrumb'
+  get 'breadcrumbs/active_job', to: 'breadcrumbs#active_job'
+  get 'breadcrumbs/cache_read', to: 'breadcrumbs#cache_read'
+
+  get 'mongo/success_crash', to: 'mongo#success_crash'
+  get 'mongo/get_crash', to: 'mongo#get_crash'
+  get 'mongo/failure_crash', to: 'mongo#failure_crash'
 end