SensorCacheJob issue

[matherej]

Hi,
I have a problem with SensorCacheJob. When this job should start it is going down and I have to turn on it manually in administration menu. So, I dont have actual dashboard. Please help.
Thanks

[rvineyard]

This has been an ongoing issue for me. I worked with mephux a bit on IRC awhile back and it sounded like the issue might be that I have a large number of alerts in my database (millions). I looked at the stack traces and tracked the problem to the fetch_src_ip_metrics and fetch_dst_ip_metrics functions in lib/snorby/jobs/cache_helper.rb. For me, the issue is on the second-to-last line of these two functions:

    @events.group_by { |x| x.ip.ip_src.to_s }.collect { |x,y| @src_ips.merge!({x => y.size}) }

It seems that there are cases where x.ip can be nil, so I get errors about the ip_src method not existing on a NilClass object. I didn't dig too deeply to find out the cause, but I implemented the following to fix this (and my Ruby/Rails skills are very rusty, so I'm sure this is not an optimal fix; also by the way this is against git version 2.2.5):

In lib/snorby/jobs/cache_helper.rb, I changed the above-mentioned line to the following sequence (and also to the corresponding line in the fetch_dst_ip_metrics function):

    @events.group_by { |x|
      if x.ip.nil?
      then
        nil
      else
        x.ip.ip_dst.to_s
      end
    }.collect { |x,y| @src_ips.merge!({x => y.size}) }

This may be redundant, but I also commented out some lines in the load function in lib/snorby/model/types/numeric_ip_addr.rb in the interest of making absolutely sure that the NumericIPAddr variables inside of the IP objects belonging to Event objects are never instantiated to nil:

    def load(ip)
      #case ip
      #when nil, 0
      #  nil
      #else
        ::IPAddr.new(ip,Socket::AF_INET)
      #end
    end

Just for good measure, I also setup cronjobs to run every minute and restart any of the workers that may have crashed, and email me whenever this happens. Since implementing the two fixes above, I have not seen a crash, however the worker processes do use up a lot of CPU and RAM now (usually 100% of one CPU for each one, and I've seen sustained memory usage regularly spike above 8gb - for the time being I just moved my snorby install over to its own VM with a ton of RAM and four virtual CPUs and things seem to be all good now).

On the plus side, my dashboard is now stable and I've got pretty graphs to show to management. Our CISO loves the dashboard by the way, and one of his first feature requests was of course to make the graphs clickable so that you can interactively drill down into the event data (already open as an earlier feature request) :-)

[rvineyard]

I may have spoken too soon... after some further rule tuning I decided to clear out the cache using "Snorby::Jobs.clear_cache(true)" and let things rebuild. Unfortunately this seems to have wiped out my events too (at least in Snorby... they still seem to exist in my Snort database). [EDIT: The events are back now after the workers ran for a bit, the rest is still rebuilding] Since then, the worker jobs have crashed a few more times leaving my dashboard nearly empty; this time with a different error that I'm not sure how to fix - I've increased the size of the relevant fields in mysql from TEXT to MEDIUMTEXT to no avail:

/var/rails/snorby/vendor/cache/ruby/1.9.1/gems/railties-3.0.3/lib/rails/commands/runner.rb:50:in `eval': Data too long for column 'dst_ips' at row 1 (DataObjects::DataError)
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-do-adapter-1.0.2/lib/dm-do-adapter/adapter.rb:63:in `block in execute'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-do-adapter-1.0.2/lib/dm-do-adapter/adapter.rb:260:in `with_connection'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-do-adapter-1.0.2/lib/dm-do-adapter/adapter.rb:61:in `execute'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-do-adapter-1.0.2/lib/dm-do-adapter/adapter.rb:185:in `update'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/repository.rb:180:in `update'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource/state/dirty.rb:54:in `update_resource'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource/state/dirty.rb:22:in `commit'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:968:in `_persist'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:999:in `block in update_with_hooks'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:996:in `catch'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:996:in `update_with_hooks'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:1033:in `save_self'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-validations-1.0.2/lib/dm-validations.rb:111:in `save_self'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:1018:in `block in _save'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:1234:in `run_once'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:1017:in `_save'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:417:in `save'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-validations-1.0.2/lib/dm-validations.rb:98:in `block in save'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-validations-1.0.2/lib/dm-validations/support/context.rb:30:in `validation_context'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-validations-1.0.2/lib/dm-validations.rb:98:in `save'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/resource.rb:391:in `update'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-validations-1.0.2/lib/dm-validations.rb:104:in `block in update'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-validations-1.0.2/lib/dm-validations/support/context.rb:30:in `validation_context'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-validations-1.0.2/lib/dm-validations.rb:104:in `update'
    from /var/rails/snorby/lib/snorby/jobs/daily_cache_job.rb:139:in `create_cache_record'
    from /var/rails/snorby/lib/snorby/jobs/daily_cache_job.rb:121:in `build_cache'
    from /var/rails/snorby/lib/snorby/jobs/daily_cache_job.rb:65:in `block in perform'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/collection.rb:511:in `block in each'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/support/lazy_array.rb:413:in `block in each'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/support/lazy_array.rb:413:in `each'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/support/lazy_array.rb:413:in `each'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/dm-core-1.0.2/lib/dm-core/collection.rb:508:in `each'
    from /var/rails/snorby/lib/snorby/jobs/daily_cache_job.rb:48:in `perform'
    from (eval):1:in `<top (required)>'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/railties-3.0.3/lib/rails/commands/runner.rb:50:in `eval'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/railties-3.0.3/lib/rails/commands/runner.rb:50:in `<top (required)>'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/railties-3.0.3/lib/rails/commands.rb:39:in `require'
    from /var/rails/snorby/vendor/cache/ruby/1.9.1/gems/railties-3.0.3/lib/rails/commands.rb:39:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'

[djcas9]

This issue has been fixed in Snorby 2.3.1