testing for warnings not possible

[vStone]

I've been breaking my head about this, I would love to be able to not only check for raised errors but also for warnings.

[rodjek]

You mean, test for warnings generated with warning() in a manifest? Yeah, that's not going to be easy. The options are parsing the output log messages looking for warnings or monkeypatching Puppet::Util::Warning, neither of which are really appealing.

[vStone]

I've tried to test it the way it's being tested in the puppet sources but Puppet.expect(:warning) does not seem to do the trick :/

[daenney]

The Puppet sources test Ruby code, you're testing Puppet manifests (I'm guessing). Wnat rspec-puppet allows you to do is test the resources within a manifests, warning isn't a resource so that won't show up.

notify is a resource however so maybe you can do something with that.

[vStone]

Maybe the log output can be tested instead. Looks like rspec-puppet-augeas does something like that.

@daenney although from a end-user point of view that might be correct, but I would like to know that what my manifests produce is what I want them to produce. Including spewing warnings/errors.

[hunner]

rspec-puppet's strength is in verifying properties of catalogs, and warnings are a side-effect of catalog compilation.

rspec-system is a lot slower and more heavyweight than rspec-puppet, but you could use rspec-system to parse the output of a given puppet run and capture the warnings printed there, such as in https://github.com/puppetlabs/puppetlabs-apache/blob/master/spec/system/basic_spec.rb#L6

[vStone]

rspec-puppet-augeas seems to deal with parsing log files in its own way, this could maybe be backported or merged.

[razorsedge]

+1 for being able to catch deprecation warnings at some point in the unit testing. It is nice to know when dependency module foo is deprecating something that my module depends upon and I can get to updating my code before the functionality disappears.

[jhoblitt]

I've had have mixed feelings about this issue since I first ran into it several years ago. On one hand, log messages are essentially a side effect of catalog "compilation" and don't affect agent behavior. On the other, they are an important signaling mechanism between module author(s) and end users.

As puppet modules are evolving towards the point where they can often be used as a blackbox without having to do careful code inspection, and I've felt the need to test the stderr in acceptance tests, I feel this is becoming a more important concern for unit tests.

The solution in rspec-puppet-augeas for collecting logs appears to be fairly elegant. The log stream is captured into an array using what appears to be core functionality (no monkey patching).

https://github.com/domcleal/rspec-puppet-augeas/blob/2d8c874d9cd90e15b3c513e7119daddf3e0e55ef/lib/rspec-puppet-augeas/fixtures.rb#L40-L41
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/log.rb#L115

[jhoblitt]

I suppose #load_catalogue could return an array of log output gathered using the rspec-puppet-augeas method in addition to the catalog, but that would need to be manually carried around.

rspec-puppet/lib/rspec-puppet/support.rb

Line 10 in e163926

def load_catalogue(type)

Would we be better off adding a new class that carries both the catalog and log output?

[LeoCavaille]

👍 on the ability to ensure a log is output to the user, essential for deprecation warnings.

[DavidS]

Nowadays you can add expectations to the scope (https://github.com/rodjek/rspec-puppet#accessing-the-parser-scope-where-the-function-is-running), e.g, scope.expects(:warning).never

[jhoblitt]

👍 !

[jhoblitt]

@DavidS The scope object is only available for :puppet_function examples so it doesn't actually resolve this issue as it isn't possible to test for a warning() call in spec tests for a class.

undefined local variable or method `scope' 

[DavidS]

I've created a clean new feature issue for this.