Abrupt spec abort with no errors reported with a combination of :isolated_environment and cli spec behavior

[pirj]

Some of the examples seem to be silently swallowed by RSpec in some circumstances.
I've reduced it to:

# spec/rubocop/config_loader_spec.rb
RSpec.describe RuboCop::ConfigLoader do
  include FileHelper

  include_context 'cli spec behavior'

  before do
    described_class.debug = true
    # Force reload of default configuration
    described_class.default_configuration = nil
  end

  after { described_class.debug = false }

  describe '.configuration_from_file', :isolated_environment do
    subject(:configuration_from_file) do
      described_class.configuration_from_file(file_path)
    end

    context 'when a new cop is introduced' do
      def cop_enabled?(cop_class)
        configuration_from_file.for_cop(cop_class).fetch('Enabled')
      end

      let(:file_path) { '.rubocop.yml' }
      let(:cop_class) { RuboCop::Cop::Metrics::MethodLength }

      before do
        create_file('~/.rubocop.yml',
                    <<~YAML)
                      Metrics/MethodLength:
                        Enabled: complete_nonsense
                    YAML
        create_file(file_path, config)
      end

      context 'no errors' do
        let(:config) { '' }
        it('nothing to fail') {}
      end

      context 'when not configured explicitly' do
        let(:config) { ['inherit_from: ~/.rubocop.yml'] }

        it 'is disabled' do
          expect(cop_enabled?(cop_class)).to eq 'pending'
        end
      end
    end
  end
end

The culprit is this line (mistake in the config file):

Enabled: complete_nonsense

Expected behavior

The underlying failure is reported.

Actual behavior

$ rspec --format documentation spec/rubocop/config_loader_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Randomized with seed 36379

RuboCop::ConfigLoader
  .configuration_from_file
    when a new cop is introduced
      no errors
        nothing to fail
      when not configured explicitly

Finished in 0.04127 seconds (files took 4.97 seconds to load)
2 examples, 0 failures

Randomized with seed 36379

$ rspec --format documentation spec/rubocop/config_loader_spec.rb
Run options: include {:focus=>true}

All examples were filtered out; ignoring {:focus=>true}

Randomized with seed 48762

RuboCop::ConfigLoader
  .configuration_from_file
    when a new cop is introduced
      when not configured explicitly

Finished in 0.01265 seconds (files took 1.77 seconds to load)
1 example, 0 failures

Randomized with seed 48762

Steps to reproduce the problem

Replace spec/rubocop/config_loader_spec.rb with the contents above.
Run the following command several times. Notice different number of examples executed being reported.

 rspec --format documentation spec/rubocop/config_loader_spec.rb

RuboCop version

$ [bundle exec] rubocop -V
0.78.0 (using Parser 2.6.5.0, running on ruby 2.5.5 x86_64-darwin18)

[buehmann]

The different number of examples that get run is due to random order of examples. I can trigger the two orders on my system by adding --order rand:0 or --order rand:1 to the rspec call.

Making the console output visible like this

diff --git a/spec/support/cli_spec_behavior.rb b/spec/support/cli_spec_behavior.rb
index 59dc649a8..354bf0433 100644
--- a/spec/support/cli_spec_behavior.rb
+++ b/spec/support/cli_spec_behavior.rb
@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 RSpec.shared_context 'cli spec behavior' do
-  include_context 'mock console output'
+  # include_context 'mock console output'
 
   include FileHelper

reveals

Randomized with seed 55055

RuboCop::ConfigLoader
  .configuration_from_file
    when a new cop is introduced
      when not configured explicitly
configuration from /tmp/d20191224-19344-qc68ry/work/.rubocop.yml
Inheriting configuration from /tmp/d20191224-19344-qc68ry/home/.rubocop.yml
Property Enabled of cop Metrics/MethodLength is supposed to be a boolean and complete_nonsense is not.

That error message leads to
https://github.com/rubocop-hq/rubocop/blob/b0c6efdc5f542e4901e3b75a104edd95d40c55e8/lib/rubocop/config_loader.rb#L254-L257

which "terminates execution immediately". That explains why RSpec stops running further examples. This should probably be replaced by raising some exception.