PeriodicPoller::JVM load average on linux when an exception occur reading the file doesn't raise an exception

[colinsurprenant]

Got this intermittent failure on my PR Travis tests.
https://travis-ci.org/elastic/logstash/jobs/192806190

 1) LogStash::Instrument::PeriodicPoller::JVM load average on linux when an exception occur reading the file doesn't raise an exception
     Failure/Error: expect(::File).to receive(:read).with("/proc/loadavg").and_raise("Didnt work out so well")
       (<File (class)>).read("/proc/loadavg")
           expected: 1 time with arguments: ("/proc/loadavg")
           received: 2 times with arguments: ("/proc/loadavg")
     # ./logstash-core/spec/logstash/instrument/periodic_poller/jvm_spec.rb:47:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rspec-wait-0.0.9/lib/rspec/wait.rb:46:in `(root)'
     # ./rakelib/test.rake:49:in `(root)'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/task.rb:194:in `invoke_with_call_chain'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:108:in `top_level'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:108:in `top_level'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:80:in `run'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
     # ./vendor/bundle/jruby/1.9/gems/rake-12.0.0/exe/rake:27:in `(root)'
Finished in 3 minutes 25.3 seconds (files took 6.15 seconds to load)
2148 examples, 1 failure
Failed examples:
rspec ./logstash-core/spec/logstash/instrument/periodic_poller/jvm_spec.rb:50 # LogStash::Instrument::PeriodicPoller::JVM load average on linux when an exception occur reading the file doesn't raise an exception
Randomized with seed 31247

[ph]

My theories

• either we have a background rogue agent running that make the mock fails
• Or some linux only test case are conflicting.

The same seed doesn't raise the problem on mac, trying on linux.

[ph]

Ran this suite with the same seed multiple time on linux / mac without reproducing it, so this lead me to believe that we effectively have a rogue agent/pipeline running.

[ph]

I took some time to investigate it, this is a symptom of another problem, I am pretty sure we have a periodic poller in the background started by the agent and doing some work and because of timing it can make this mock fails.

Refactoring the agent with #6455 will make sure the poller/webserver are only started when really needed. but we also need to make sure that correctly shutdown the classes.

[jsvd]

this is a dup of #6354