Difference in handling apply_manifest(pp, :expect_failures => true)

[electrical]

Im applying a manifest and i expect it to fail.
With the help of @hunner i've set the run to this:

apply_manifest(pp, :expect_failures => true)

On puppet 2.7 and 3.1 i get the following:

debian-70rc1-x64 $ env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes /tmp/apply_manifest.pp.ZbMeaB
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
info: Applying configuration version '1391813873'
info: FileBucket adding {md5}da672da1bfe0db7baac2c2287b35b00f
info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Filebucketed /etc/elasticsearch/templates_import/elasticsearch-template-foo.json to puppet with sum da672da1bfe0db7baac2c2287b35b00f
notice: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]/content: content changed '{md5}da672da1bfe0db7baac2c2287b35b00f' to '{md5}6c6e2f263dbb3bc2657b380309602dd7'
info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Scheduling refresh of Exec[delete_template_foo]
notice: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Triggered 'refresh' from 1 events
info: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Scheduling refresh of Exec[insert_template_foo]
err: /Stage[main]//Elasticsearch::Template[foo]/Exec[insert_template_foo]: Failed to call refresh: curl -sL -w "%{http_code}\n" -XPUT http://localhost:9200/_template/foo -d @/etc/elasticsearch/templates_import/elasticsearch-template-foo.json -o /dev/null | egrep "(200|201)" > /dev/null returned 1 instead of one of [0] at /etc/puppet/modules/elasticsearch/manifests/template.pp:110
notice: Finished catalog run in 30.97 seconds
err: /File[/var/lib/puppet/rrd]: Could not evaluate: Could not find group puppet
err: Could not send report: Got 1 failure(s) while initializing: Could not evaluate: Could not find group puppet

debian-70rc1-x64 executed in 32.93 seconds
Exited: 2
    run should fail (FAILED - 1)



Failures:

  1) elasticsearch template define: Insert a template with bad json content run should fail
     Failure/Error: apply_manifest(pp, :expect_failures => true)
     Beaker::Host::CommandFailure:
       Host 'debian-70rc1-x64' exited with 2 running:
        env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes /tmp/apply_manifest.pp.ZbMeaB
       Last 10 lines of output were:
        info: FileBucket adding {md5}da672da1bfe0db7baac2c2287b35b00f
        info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Filebucketed /etc/elasticsearch/templates_import/elasticsearch-template-foo.json to puppet with sum da672da1bfe0db7baac2c2287b35b00f
        notice: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]/content: content changed '{md5}da672da1bfe0db7baac2c2287b35b00f' to '{md5}6c6e2f263dbb3bc2657b380309602dd7'
        info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Scheduling refresh of Exec[delete_template_foo]
        notice: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Triggered 'refresh' from 1 events
        info: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Scheduling refresh of Exec[insert_template_foo]
        err: /Stage[main]//Elasticsearch::Template[foo]/Exec[insert_template_foo]: Failed to call refresh: curl -sL -w "%{http_code}\n" -XPUT http://localhost:9200/_template/foo -d @/etc/elasticsearch/templates_import/elasticsearch-template-foo.json -o /dev/null | egrep "(200|201)" > /dev/null returned 1 instead of one of [0] at /etc/puppet/modules/elasticsearch/manifests/template.pp:110
        notice: Finished catalog run in 30.97 seconds
        err: /File[/var/lib/puppet/rrd]: Could not evaluate: Could not find group puppet
        err: Could not send report: Got 1 failure(s) while initializing: Could not evaluate: Could not find group puppet
     # ./spec/acceptance/003_template_spec.rb:117:in `block (3 levels) in <top (required)>'

On Puppet 3.2/3.3/3.4 it works as expected and flags the test as passed ( as I'm expecting it to fail )

Can give more details via IRC.

[electrical]

did some checking together with 'ashp' on IRC.
Puppet 2.7 and 3.1 are returning exit code 2. Puppet 3.2 and up returning exit code 6.
Any idea's on resolving this?

[electrical]

Solved it for now by limiting tests based on the puppetversion fact.
Im guessing that's the only way for now?

[anodelman]

From the apply_manifest code:
if opts[:catch_changes]
args << '--detailed-exitcodes'

      # We're after idempotency so allow exit code 0 only.
      on_options[:acceptable_exit_codes] |= [0]
    elsif opts[:catch_failures]
      args << '--detailed-exitcodes'

      # We're after only complete success so allow exit codes 0 and 2 only.
      on_options[:acceptable_exit_codes] |= [0, 2]
    elsif opts[:expect_failures]
      args << '--detailed-exitcodes'

      # We're after failures specifically so allow exit codes 1, 4, and 6 only.
      on_options[:acceptable_exit_codes] |= [1, 4, 6]
    elsif opts[:expect_changes]
      args << '--detailed-exitcodes'

      # We're after changes specifically so allow exit code 2 only.
      on_options[:acceptable_exit_codes] |= [2]
    else
      # Either use the provided acceptable_exit_codes or default to [0]
      on_options[:acceptable_exit_codes] |= [0]
    end

From our code, a return value on failure is 1,4,6 while a return value of 2 indicates a change was made to the machine configuration.